When Can LLMs Learn to Reason with Weak Supervision?

Large language models' reasoning capabilities have improved substantially through reinforcement learning with verifiable rewards, but as these models become more powerful, creating high-quality reward signals becomes increasingly challenging. Researchers have investigated the conditions under which reinforcement learning with verifiable rewards can succeed with weaker forms of supervision, conducting a systematic empirical study across various model families. The study aims to understand the limitations and potential of reinforcement learning with weaker supervision, which is crucial for developing more robust and reliable large language models. Specifically, the research focuses on the interplay between model architecture, training data, and reward signals, exploring how these factors impact the models' ability to reason effectively¹. This research has significant implications for the development of large language models, as it can help mitigate potential risks and improve their overall performance, ultimately affecting the security landscape of natural language processing applications.