RSI-top
htophtop is the live monitor you open to diagnose a machine: what is running, and what is stuck. for autoresearch
Escaping exploration collapse in long-horizon autoresearch
Metric dashboardExploration collapseDiversity control
htophtop is the live monitor you open to diagnose a machine: what is running, and what is stuck. for autoresearch
Escaping exploration collapse in long-horizon autoresearch
Main contributions
htop gives you an at-a-glance view of process activity and resource usage, helping you decide where to look before digging into logs. RSI-top builds the same view for an autoresearch run: it summarizes each trajectory using task-specific metrics defined before the run and computed in the background.
An autoresearch run is a sequence of experiments organized by parent–child relationships. In each iteration, the agent picks an earlier candidate as the parent, proposes a change, implements it, and ends with a new candidate, for example a kernel or a checkpoint. An evaluator scores the candidate, and the result joins the archive that later iterations build on. We call the full record of one iteration a trajectory: the agent's reasoning trace, the actions and code changes it made, and the execution and evaluation logs. A run makes hundreds of these trajectories over many hours. The best score found so far is what we call the frontier.
Think of the search as gradient descent on a landscape of ideas, where height is the loss: lower is better. Early on, progress comes easily. Then the agent settles into one valley and keeps running experiments in the same narrow direction, so the frontier stops moving. Its proposals collapse onto a narrow family of approaches. Only when the frontier stalls does RSI-top step in: it launches a diverse cohort to explore alternative directions, adds the resulting candidates to the archive, and lets the native search continue.
We ran OpenEvolve and ShinkaEvolve on Anthropic kernel optimization, each with DeepSeek v4 Flash and Gemini 3.7 Flash. All four runs follow the same pattern: after early gains, the frontier is stalled for 84% to 92% of the run.1 We use frontier stalls to trigger interventions; a stall alone does not establish that exploration has collapsed.1A stall is any stretch of at least 20 minutes, after the first hour of the run, in which the best score improves by fewer than 2 cycles.
Why does the agent get stuck? One likely contributor is context inertia: the model commits to a direction in its context and keeps pursuing it after the evidence says to change course. Every attempt in that direction adds to the context that pulls the next attempt along.
Yet the agent cannot simply ignore its history. To leave a valley, it needs to know what was tried, what helped, and which branches still look promising, and after hundreds of trajectories that history no longer fits in context anyway. The usual ways to manage the context either keep too much or throw away exactly the information an escape needs. The challenge is to retain enough information to choose what to inspect without repeatedly loading the full history. RSI-top keeps a dashboard as a comparable index of every trajectory and retrieves raw trajectories only on demand.
Key idea
Nobody reads every log line to see what a machine is doing. You use htop to inspect processes and resource usage, or nvidia-smi for GPUs,2 then dig into the logs that matter. RSI-top gives autoresearch the same thing: a dashboard that compresses each trajectory into a compact set of task-specific metrics. When the search stalls, agents read the dashboard, unfold the few trajectories worth a closer look, and plan an escape. We call retrieving and inspecting a full trajectory “unfolding.” Diversity control encourages different intervention plans.2htop is an interactive process viewer for Unix-like systems; nvidia-smi reports the state of NVIDIA GPUs.
The metric definitions are fixed before each run and remain unchanged throughout it. Each trajectory gets one row of numbers. That row is a compressed record of the trajectory that agents can scan, sort, and compare without loading the trajectory itself.
| # | Metric | Definition |
|---|---|---|
| Unarythe trajectory on its own | ||
| 1 | cycles | Authoritative performance of the current valid candidate. |
| 2 | valid_attempt | Whether the attempt reached an authoritative valid evaluation. |
| 3 | recorded_change_count | Number of distinct changes recorded for the current attempt. |
| Binarythe trajectory against one reference: its parent, the frontier, or its nearest earlier state | ||
| 4 | parent_gain_cycles | Measured improvement relative to the actual parent. |
| 5 | frontier_relative_gap | Candidate gap to the frontier immediately before the attempt. |
| 6 | parent_code_distance | Normalized implementation distance from the actual parent. |
| 7 | frontier_code_distance | Normalized implementation distance from the frozen frontier reference. |
| 8 | nearest_history_similarity | Similarity to the closest strictly earlier implementation state. |
| 9 | meaningful_relative_gain | Gain relative to the causal progress anchor. |
| 10 | gap_in_recent_step_units | Authoritative frontier gap measured in recent meaningful frontier-step units. |
| Regiona recent window of trajectories | ||
| 11 | frontier_relative_spread_F | Implementation spread of the latest complete eligible-state window relative to the frozen frontier code. |
| 12 | root_relative_spread_S | Implementation spread of the latest complete eligible-state window relative to the fixed root code. |
| 13 | parent_selection_concentration | Concentration of immediate-parent selection in the latest complete window. |
| 14 | window_frontier_gain_cycles | Total measured frontier improvement across the latest complete window. |
| 15 | window_redundancy_rate | Fraction of the latest complete terminal window that is an exact duplicate or no-op. |
| 16 | window_valid_rate | Fraction of the latest complete terminal window with an authoritative eligible evaluation. |
| 17 | window_frontier_conversion_rate | Fraction of eligible attempts in the latest complete terminal window that made a meaningful frontier improvement. |
| 18 | within_one_recent_step_rate | Fraction of valid candidates in the latest window of 12 trajectories that are within one recent verified frontier step. |
| 19 | dominant_failure_signature_mass | Largest failure-signature share among failed attempts in the latest window of 12 trajectories. |
| 20 | repeated_failure_rate | Fraction of failed attempts in the latest window of 12 trajectories whose failure signature appeared strictly earlier. |
| 21 | quality_conditioned_frontier_diversity | Implementation spread among valid candidates in the latest window of 12 trajectories that are within one recent meaningful frontier step. |
| Globalthe whole run | ||
| 22 | seconds_since_meaningful_improvement | Wall-clock time since the last meaningful frontier improvement. |
| 23 | frontier_epoch_age_steps | Terminal steps elapsed in the current frontier epoch. |
| 24 | best_descendant_cycles | Best authoritative cycles reached by descendants of this row by the frozen audit cutoff. |
| 25 | descendant_frontier_move_count | Number of verified frontier moves produced by this row's descendants by the frozen audit cutoff. |
A shared dashboard alone does not ensure diverse proposals. RSI-top runs the intervention as a cohort of multiple agents and pushes them apart in three ways.4 Click each one for details.4In the default configuration, each agent sees a random 15 of the 25 metrics.
Like an ensemble whose members see different features, each agent in the cohort receives a different random subset of the dashboard's metrics. Agents that watch different signals rank the history differently, so they unfold different trajectories and reach different conclusions about where the search went wrong.
In contrast, giving every agent the same view can lead to overlapping retrieval choices and similar proposals.
Asked for one idea eight times, an LLM can return closely related variations of its most typical answer. Following Verbalized Sampling, each agent instead proposes eight directions as one portfolio, each with a verbalized probability. Each probability estimates how likely an ordinary planner would be to propose that direction independently, so a low value marks a less obvious idea. The values need not sum to 1, and they are not a calibrated chance of success. Asking for a distribution brings out credible low-probability ideas.
Generate the directions as one joint portfolio sampled across the distribution of credible interventions, not as variations of the single most typical answer. Include credible lower-probability modes as well as common modes. Add a numeric `verbalized_probability` between 0 and 1 to every direction. [...] Directions are substantively distinct only when they test materially different bottleneck -> mechanism -> expected-effect hypotheses; changing only a target, parameter, or wording is insufficient.
Evolutionary frameworks pick a parent from the archive and build on it. Linking every trajectory to its parent turns the archive into a DAG. Each lineage is a different branch of the search history. All agents share this history; the prompt asks each agent to find distinct causal subgraphs in it and ground its directions in different ones, so the portfolio covers genuinely different opportunities.
Use selective unfolding to find as many distinct, relevant causal subgraphs as the evidence supports. Ground directions in different subgraphs when possible so the portfolio covers genuinely different opportunities.
We evaluate on two benchmarks where every trajectory builds on the ones before it. Anthropic kernel optimization asks the agent to optimize a kernel for a simulated machine, scored in cycles (lower is better). MLE-bench asks it to compete in Kaggle competitions, scored on a validation set during the search and then tested on the hidden test set. On Anthropic kernel optimization, we plug RSI-top into OpenEvolve and ShinkaEvolve: it adds interventions around the native search loop while leaving its internal selection and mutation rules unchanged. On MLE-bench, we plug RSI-top into a simple ReAct loop.
All four framework and model pairs follow the same pattern: the OpenEvolve or ShinkaEvolve run stalls, and RSI-top climbs out of the stall. As a reference, we also run each framework on its own, without RSI-top, with GPT-5.6 Sol and with Claude Opus 5 until a $300 API budget is spent (dashed lines). RSI-top with a Flash-class model reaches nearly the same score at a fraction of that cost.
RSI-top runs with Gemini 3.7 Flash on one A100 for 12 hours. Across 75 competitions, it achieves a mean medal rate of 63.1%, matching the highest reported mean among the listed methods. It produces valid submissions on all competitions and achieves the highest reported mean above-median rate (77.3%) and gold rate (43.1%). Most listed methods use a 24-hour time limit.55MLE-bench has 75 Kaggle competitions grouped by complexity into Low, Medium, and High; MLE-bench Lite is the 22-competition Low split.
MLE-bench · full benchmark, 75 competitions (Low, Medium, High)
| Agent | Budget | Low | Medium | High | All | Valid | Med+ | Gold |
|---|---|---|---|---|---|---|---|---|
| Proprietary methods | ||||||||
| FM-AgentGemini-2.5-Pro | 24 h1× A800 (80GB) | 62.1 ±1.5 | 36.8 ±1.5 | 33.3 ±0.0 | 43.6 ±0.9 | 96.9 ±1.2 | 51.6 ±1.2 | 22.7 ±0.8 |
| MLE-STAR-Pro-1.5Gemini-2.5-Pro | 24 h2× A100 (40GB) | 68.2 ±2.6 | 34.2 ±1.5 | 33.3 ±0.0 | 44.0 ±1.3 | 93.8 ±0.4 | 52.9 ±1.6 | 19.1 ±1.8 |
| MARSGemini-3-Pro-preview | 24 h1× A100 (40GB) | 74.2 ±1.5 | 52.6 ±3.0 | 37.8 ±2.2 | 56.0 ±1.5 | 98.7 ±0.0 | 65.8 ±1.6 | 31.1 ±0.4 |
| MARS+Gemini-3-Pro-preview | 24 h2× H100 (80GB) | 78.8 ±1.5 | 60.5 ±1.5 | 44.4 ±2.2 | 62.7 ±0.8 | 100.0 ±0.0 | 74.2 ±0.9 | 33.8 ±0.4 |
| AIBuildAIClaude-Opus-4.6 | 24 h1× A100 (80GB) | 77.3 ±0.0 | 61.4 ±0.9 | 46.7 ±0.0 | 63.1 ±0.4 | 100.0 ±0.0 | 71.1 ±1.2 | 25.8 ±0.4 |
| Open-source methods | ||||||||
| AIDEo1-preview | 24 h1× A10 (24GB) | 35.9 ±1.9 | 8.5 ±0.4 | 11.7 ±1.3 | 17.1 ±0.6 | 82.8 ±1.1 | 29.4 ±1.3 | 9.4 ±0.8 |
| AIRA-Dojoo3 | 24 h1× H200 (141GB) | 55.0 ±1.5 | 22.0 ±1.2 | 21.7 ±1.1 | 31.6 ±0.8 | 97.5 ±0.3 | 45.5 ±0.8 | 17.3 ±0.4 |
| LeerooGemini-3-Pro-preview | 24 h1× H100 (80GB) | 68.2 ±2.6 | 44.7 ±1.5 | 40.0 ±0.0 | 50.7 ±1.3 | 50.7 ±1.3 | 50.7 ±1.3 | 21.3 ±2.0 |
| ML-MasterDeepSeek-R1 | 12 h1× A100 (80GB) | 48.5 ±1.5 | 20.2 ±2.3 | 24.4 ±2.2 | 29.3 ±0.8 | 93.3 ±1.3 | 44.9 ±1.2 | 17.3 ±0.8 |
| ML-Master 2.0DeepSeek-V3.2-Speciale | 24 h2× RTX 4090 (24GB) | 75.8 ±1.5 | 50.9 ±3.5 | 42.2 ±2.2 | 56.4 ±2.5 | 95.6 ±1.2 | 63.1 ±1.2 | 19.6 ±0.9 |
| R&D-Agentgpt-5 | 12 h1× V100 (32GB) | 68.2 ±2.6 | 21.1 ±1.5 | 22.2 ±2.2 | 35.1 ±0.4 | 53.3 ±0.0 | 40.4 ±0.9 | 16.4 ±0.9 |
| MLEvolveGemini-3-Pro-preview | 12 h1× H200 (141GB) | 80.3 ±1.5 | 57.9 ±1.5 | 42.2 ±2.2 | 61.3 ±1.3 | 98.2 ±0.4 | 72.9 ±1.2 | 32.0 ±2.8 |
| Ours | ||||||||
| RSI-topGemini-3.7-Flash | 12 h1× A100 (80GB) | 80.3 ±2.6 | 60.5 ±9.5 | 44.4 ±3.8 | 63.1 ±6.3 | 100.0 ±0.0 | 77.3 ±6.7 | 43.1 ±4.7 |
On MLE-bench Lite, we rerun the open-source baselines with Gemini 3.7 Flash to compare different autoresearch methods using the same base model.
MLE-bench Lite · 22 low-complexity competitions
Autoresearch agents are starting to run for days, and the bottleneck is shifting from how much compute an agent gets to how it sees its own history. RSI-top is a first step: give the agent a dashboard of its own history, let it retrieve only the trajectories worth reading, and push the search in more than one direction when it stalls. We are excited about where this goes next.
Concurrent work. Several concurrent systems see long autoresearch runs plateau and fix it the same way: discard the conversation and keep the work. Fork-and-Flush [3] calls the effect "idea basins"; it forks an agent into copies that share the workspace but start with an empty context, runs them for a fixed horizon, and continues from the best copy. Scaling Discovery Requires Forgetting [4] periodically discards every agent's session and the candidate history, then restarts from the best candidate so far. Anthropic's Memento harness [5] stops each agent at a fixed timeout and hands a fresh agent the codebase and a short note. These approaches reset the conversational context and carry forward only the workspace or the best candidate. RSI-top instead keeps a structured index of prior experiments to guide selective retrieval and intervention planning, and diversifies the agents it launches.
References.