Steer, Don't Tell: Breaking Reasoning Plateaus with Reachability-Aligned Self-Distillation
Post training for large reasoning models remains insufficient on challenging problems, whose correct trajectories are hard to sample. Though prior works address this difficulty by integrating…
- 用途
- 原典で確認
- 難易度の目安
- Hard
- 実装コストの目安
- High
概要
Post training for large reasoning models remains insufficient on challenging problems, whose correct trajectories are hard to sample. Though prior works address this difficulty by integrating supervised fine tuning on expert trajectories with reinforcement learning, such external experts are not always available. Existing self improvement methods instead use reference solutions to create a privileged self-teacher. Yet privileged teacher correctness does not imply student reachability. Solution conditioning shifts the teacher distribution toward actions unlikely under the unprivileged student, producing seemingly "correct" trajectories that are unreachable or even mal-formed learning targets. Accordingly, we identify reachability alignment as the key to self constructed supervision and propose Self-Steered Reasoner (SSR). SSR realizes this principle through student support projection, which transforms the privileged teacher distribution into a directional signal only over reasoning actions locally reachable under the unguided student. Verification then retains correct trajectories sampled from this reachable space. To further boost the sampling efficiency, an iterative refinement framework is adopted, using diagnostic guidance based on current trajectories. On problems with pass@64 = 0 even after extensive RL, SSR recovers correct and learnable trajectories for 74.6% without parameter updates. Fine tuning on these trajectories followed by resumed reinforcement learning improves aggregate accuracy by 4.9 percentage points on AIME25/26 and HMMT26. These results show that privileged guidance becomes effective self-improvement data when aligned with the reasoning actions the student can reach.
参考リンク
リンク先のライセンス・再現性・データ条件をご確認ください。自動分類や実装の目安は保証ではありません。
実装を始めるプロンプト
参考リンクを確認したうえで、次の内容をコピーして利用できます。
# 論文・記事 Steer, Don't Tell: Breaking Reasoning Plateaus with Reachability-Aligned Self-Distillation # 概要 Post training for large reasoning models remains insufficient on challenging problems, whose correct trajectories are hard to sample. Though prior works address this difficulty by integrating… # 原典 https://arxiv.org/abs/2611.00796 上記の手法を実装してください。実行前に依存関係とライセンスを確認してください。