NeoHorse-1 turns a deployed model router into a training signal: routing tiers organize agent trajectories into a difficulty-graded curriculum for supervised fine-tuning and on-policy distillation, so what the fleet observes about a model’s weak spots directly shapes its next training mixture.
Suppose you run a production agent system where user requests get dispatched to different backend models based on how hard they look: cheap model for a quick lookup, big model for multi-step debugging. That dispatcher (a routing harness) is already logging, per turn, what it thought the request needed, which model actually ran, what tools got called, and whether the task finished. Today that log mostly serves billing and monitoring.
The paper’s claim is that this log is also the missing ingredient for Recursive Self-Improvement (RSI). Prior agentic training work (FireAct, AgentTuning, and Llama 3’s tool-use recipe are the ones the paper cites) mostly treats interaction traces as static supervision: collect trajectories, do SFT, ship. What’s missing is a way to let the system’s own observed weaknesses steer the next round of data collection and training. The routing harness supplies that because it labels every turn with an estimated capability demand, which the authors reuse as a difficulty signal.
The baseline being improved on is Qwen3.5 at 4B and 9B scales; NeoHorse-1 is a post-trained variant of those base models.
The pipeline has three parts that feed each other.
Turning harness logs into training examples. Each recorded interaction is sliced into user turns: one user request plus every assistant message, tool call, and tool result up to the next user request. The turn keeps its full history as context but only the assistant spans in the current turn get prediction loss. Earlier reasoning traces are dropped (following a convention from DeepSeek-V3.2), current-turn reasoning is kept. Each turn goes through structural checks (are tool calls closed? do IDs match?) and a six-dimension semantic judge (goal attainment, instruction adherence, tool use, evidence consistency, error recovery, termination) that can mark each dimension PASS/WARN/FAIL/NOT_EVALUATED rather than collapsing to a single score.
Routing scores as a difficulty proxy. The harness router assigns each turn one of four tiers, C0 through C3, roughly “trivial” to “needs the strongest model.” Crucially, the authors do NOT use the tier that actually served the request (that reflects user overrides and availability). They re-run the router on just the request plus prior context to get a clean capability-demand estimate, either as a hard tier index or a soft weighted average across tiers.
Curriculum SFT, then on-policy distillation. Training runs in three stages that progressively add higher-scored (harder) examples while keeping some easy ones around so late training isn’t all high-demand. The same three-stage schedule is then reused for On-Policy Distillation: the student generates responses from recorded starting contexts, and a fixed teacher provides token-level distributions on the student’s own outputs. Both models are compared over the same top-K candidate tokens plus a bin for the leftover mass, and the student minimizes reverse KL against the teacher.
for stage in [1, 2, 3]:
batch = sample_contexts(pool, routing_score_stage=stage)
for ctx in batch:
student_response = student.generate(ctx) # on-policy rollout
for t in positions(student_response):
p = student.topk_plus_bin(ctx, prefix_up_to(t))
q = teacher.topk_plus_bin(ctx, prefix_up_to(t))
loss += kl(p, q)
update(student, loss) # teacher frozen
Closing the loop. After training, the new checkpoint is evaluated on a stratified suite, and per-attribute deficits shift the next training mixture toward weak regions. The updated model goes back into the harness pool, generates new trajectories, and the cycle repeats. The paper reports one pass of this loop, not many.
Evaluation covers ten benchmarks in three buckets: agentic (including BFCL V4, tau2-bench, plus four harness-run agent suites), coding (HumanEval, LiveCodeBench v6), and instruction following (IFEval, IFBench).
•
Macro-average lifts: the 4B model goes from 58.94 to 64.87, and the 9B model from 65.60 to 69.04. The paper frames this as broad gains rather than a single-benchmark spike, and NeoHorse-1-4B matches or beats the Qwen3.5-9B base on several benchmarks.
•
Where gains concentrate: harness-based agent tasks and execution-heavy coding move the most. Instruction-following scores stay roughly flat at 9B, with one metric slightly down.
•
Data source ablation: with the same routing-guided curriculum, optimizer, seed, and budget, training on the authors’ routing-harness trajectories beats training on Toucan by +6.26 points on a five-benchmark average, with the biggest deltas on HumanEval (+8.54) and $\tau^2$-Bench (+11.31). This is the cleanest evidence that the harness-sourced data itself, not just the curriculum, is doing work.
•
Scaling supervision: adding more unique routing-harness tokens (nested subsets, everything else held fixed) moves the five-benchmark average from 69.31 to 71.45. Consistent but modest.
•
Trajectory case studies (illustrative, not quantitative for the main claim): the post-trained 4B model retrieves evidence the base model skips; the 9B model is better at edit-test-repair loops and at abandoning a blocked strategy (switching from an unavailable pandas dependency to the stdlib csv module).
A caution the paper itself flags: benchmark gains show the recipe works for one pass. They do NOT show the RSI loop compounds across generations, because only one iteration was run.
•
If you already operate a multi-model router, the concrete takeaway is that the router’s per-turn tier prediction is usable as a curriculum signal without any new annotation. The paper is careful to re-score requests offline rather than trust the served tier, because served-tier reflects policy and availability. Worth replicating that separation if you try this.
•
If you’re choosing between public agent-trajectory datasets and your own harness logs, the Toucan comparison suggests logs from your actual deployment transfer better under matched training, at least for the task mix the authors evaluated. Worth testing on your own eval before committing.
•
If you’re thinking about on-policy distillation, the top-K-plus-bin trick (both models score the same K candidates plus one bin for residual mass) is a practical way to keep teacher logits cheap to store and align. The teacher stays frozen; the rollout checkpoint refreshes periodically so later batches reflect current student behavior.
•
What this evidence does not support: claiming that harness-mediated RSI compounds. The paper ran one evaluation-selection-update cycle. Whether iterating produces continued gains is called out as future work, not demonstrated.
Artifacts are released on Hugging Face and GitHub.
•
Several baselines and one harness component in the paper have names that don’t match well-known public systems (OpenSquilla, Spark-X2.5, Nanbeige-4.2-3B, Ornith-1.5-9B, Muse-Glimmer-30B, DeepSeek-V4-Flash, Qwen3.5). The paper doesn’t clarify whether these are internal renamings or genuinely distinct models, so cross-reading against other leaderboards is difficult.
•
The RSI framing is aspirational for this report. One loop iteration was run; the compounding claim is a design argument, not a measured result.
•
Gains at 9B are noticeably smaller than at 4B, and instruction-following barely moves at 9B. The recipe looks most useful when the base model has clear headroom on execution-heavy tasks.
•
The routing tier scheme (C0-C3) is defined by the authors’ own policy. Reusing this recipe requires either their router or an equivalent capability-demand estimator; the paper doesn’t release a standalone router.
•
Case studies are hand-picked trajectories, useful as intuition but not evidence of general behavior change.