Abstract
Multi-agent discussion produces reasoning collaboratively, while deployment still relies on a smaller student that must answer alone. Full-graph distillation transfers such discussions by treating each correct solution as a separate training target, but what teachers provide need not match what the student learns. This mismatch appears in two ways: several correct solutions may amount to the same lesson in the student's representation (over-teaching), while some training questions remain missed after learning (under-learning). We quantify both with rates that are high on discussion graphs. To address them, we introduce FAR (Fold--Attempt--Revisit), a learner-centered distillation procedure that organizes supervision around the student's own state. Before training, FAR folds solutions within each discussion state into representative lessons. After training, the student's own attempts identify missed questions, while poorly represented clusters indicate where folding may have lost useful variation; FAR then writes targeted lessons from retained discussion contexts. Across two students and four reasoning tasks, FAR outperforms full-graph distillation with far fewer training targets and lower under-learning. Further analyses examine the behavior of FAR and the sources of its gains. Overall, the results show that organizing discussion supervision around the student can make distillation both more effective and more efficient.
Abstract (translated)
URL
https://arxiv.org/abs/2610.16543
