Learned Reporting Preferences in RLVR Can Conflict with the Current Request
Researchers have identified that models trained using reinforcement learning with verifiable rewards often prioritize formatting patterns preferred during training over the actual requirements of new prompts. This behavioral conflict can cause reasoning models to fail when a specific request deviates from the established reporting style learned during the reward process.
Covered by 1 source
- AarXiv CS.AI↗Yupeng Chang, Wenxuan Zhang, Yuan WuSep 30