CURIO: Curiosity-Driven Test-Time Learning for Open-Ended Discovery

Tao Feng¹ Fangxu Yu² Zijie Lei¹ Jiaru Zou³ Changjiang Jiang¹ Yi Yan¹ Jiaxuan You¹ Pan Lu³

¹University of Illinois Urbana-Champaign ²University of Maryland, College Park ³Stanford University

arXiv (Coming Soon) Code

Discovery Tasks

Six math problems and single-cell denoising.

8B–235B

Qwen3 Backbones

Gains over task-only RL at every tested scale.

+18.3%

Hadamard (235B)

Relative gain over TTT-Discover.

+10.8%

Denoising Score

Relative gain on Tabula (8B).

Abstract

Learning from Feedback, Without Losing the Urge to Explore

Open-ended discovery requires learning from repeated attempts while continuing to explore directions whose value is not yet apparent. Search with a frozen large language model (LLM) can reuse previous solutions in context, but cannot update the model from its successes and failures on the test problem. Reinforcement learning (RL) enables such adaptation; however, strongly favoring high-reward trajectories may suppress low-reward yet potentially promising directions too early.

We introduce CURIO, a curiosity-driven test-time learning framework that complements task feedback with an Intrinsic Curiosity World Model (ICWM). The ICWM learns transitions in the policy's hidden-state representation and supplies prediction-error bonuses at sampled tokens outside the policy's top-$k$ choices. Epoch normalization and an annealed weight regulate their contribution to the policy update.

On six mathematical discovery tasks and single-cell denoising with Qwen3 backbones from 8B to 235B, three-run means improve over a matched task-only RL control on five mathematical objectives, match the best reported performance on Circle Packing, and improve denoising Score and mean squared error (MSE) on both held-out corpora at every tested scale. Relative gains reach 18.3% on Hadamard and 10.8% on denoising Score. Code-diversity measurements show greater structural variation among generated programs, supporting curiosity as a complementary exploration signal for learning in open-ended discovery.

How It Works

A Curiosity Signal Inside the Discovery Loop

CURIO follows the test-time RL discovery loop: a policy expands parents from an archive into rollouts, an evaluator scores them, the archive keeps the better solutions, and the policy is updated on the same problem. On top of the task reward, CURIO learns where the policy is surprising itself and rewards exploration exactly there.

CURIO overview: the test-time discovery loop on the left; on the right, (i) tokens outside the policy's top-k are selected as branch points, (ii) an Intrinsic Curiosity World Model learns to predict the next hidden state, and (iii) its prediction error becomes a curiosity bonus added to the task coefficient. — **Overview of CURIO.** A policy expands archive parents into rollouts. The ICWM learns token-level transitions in encoded policy hidden states, and its prediction residual informs the curiosity bonus. The signal is retained at sampled tokens outside the policy's top-$k$ choices and combined with the task coefficient, $\tilde A_t = \mathrm{sg}[A_t + \eta_\tau b_t]$.

(i) Find Branch Points

During generation, CURIO records which sampled tokens fall outside the policy's top-$k$ choices. These departures from the most likely continuation are the only places where the curiosity bonus is applied.

(ii) Learn an Intrinsic Curiosity World Model

An encoder and forward predictor learn $s_t, a_t \rightarrow s_{t+1}$ over the current actor's last-layer hidden states and token embeddings. Its prediction error marks transitions the policy has not yet learned to anticipate.

(iii) Reward Novelty, Then Anneal

Residuals are normalized per epoch and added to the task coefficient with weight $\eta_\tau$, cosine-annealed to a floor. Setting $\eta_0 = 0$ recovers the task-only TTT-Discover update exactly.

Main Results

Better Discoveries at Every Scale

We compare against frozen-LLM search (AlphaEvolve via OpenEvolve, ShinkaEvolve, Best-of-25600, SimpleTES) and trainable test-time RL (ThetaEvolve, TTT-Discover). TTT-Discover is the matched task-only control: the same procedure as CURIO with $\eta_0 = 0$. All results are means over three runs.

Mathematical Discovery

Method	Erdős ↓	AC1 ↓	AC2 ↑	AC3 ↓	Circle Packing ↑	Hadamard ↑
Qwen3-8B
AlphaEvolve	0.381002	1.50561	0.8947	1.5062	2.63585	0.5589
ThetaEvolve	0.380999	1.50649	0.9449	1.4942	2.63597	0.5748
ShinkaEvolve	0.381026	1.51493	0.8501	1.5074	2.63597	0.5661
Best-of-25600	0.381091	1.51032	0.9318	1.5081	2.63570	0.5554
SimpleTES	0.381006	1.50628	0.9458	1.4951	2.63597	0.5776
TTT-Discover	0.380995	1.50546	0.9463	1.4935	2.63598	0.5809
CURIO	0.380932	1.50513	0.9481	1.4846	2.63598	0.6048
Qwen3-14B
AlphaEvolve	0.381068	1.50971	0.9158	1.5109	2.63482	0.5762
ThetaEvolve	0.383710	1.51488	0.9351	1.6428	2.62491	0.5867
ShinkaEvolve	0.381041	1.51052	0.9048	1.4996	2.63509	0.5826
Best-of-25600	0.381119	1.50997	0.9271	1.5051	2.63439	0.5708
SimpleTES	0.381009	1.50819	0.9374	1.4879	2.63546	0.5902
TTT-Discover	0.380979	1.50608	0.9391	1.4592	2.63598	0.5864
CURIO	0.380919	1.50591	0.9418	1.4578	2.63598	0.6061
Qwen3-235B
TTT-Discover	0.380918	1.50376	0.9562	1.4561	2.63598	0.7846
CURIO	0.380880	1.50315	0.9588	1.4549	2.63598	0.9284

Best feasible objective found per run, averaged over three runs. ↓ lower is better, ↑ higher is better. Within each scale, bold is best and underline is second-best; ties share the marking.

Single-Cell Denoising

Method	PBMC			Tabula
Method	Score ↑	MSE ↓	Poisson ↓	Score ↑	MSE ↓	Poisson ↓
Qwen3-8B
AlphaEvolve	0.6738	0.1817	0.0554	0.6785	0.1716	0.0323
ThetaEvolve	0.6981	0.1588	0.0494	0.7207	0.1441	0.0302
ShinkaEvolve	0.6787	0.1764	0.0535	0.6881	0.1669	0.0312
Best-of-25600	0.6274	0.2071	0.0607	0.6382	0.1968	0.0354
SimpleTES	0.6958	0.1641	0.0503	0.7118	0.1514	0.0305
TTT-Discover	0.6842	0.2087	0.0818	0.6468	0.1963	0.0363
CURIO	0.7009	0.1552	0.0493	0.7165	0.1485	0.0301
Qwen3-14B
AlphaEvolve	0.6684	0.1821	0.0504	0.6586	0.1824	0.0302
ThetaEvolve	0.6814	0.1678	0.0503	0.6892	0.1586	0.0302
ShinkaEvolve	0.6782	0.1715	0.0502	0.6783	0.1618	0.0301
Best-of-25600	0.6171	0.2023	0.0506	0.6576	0.1822	0.0303
SimpleTES	0.6873	0.1651	0.0499	0.6966	0.1558	0.0302
TTT-Discover	0.6901	0.1615	0.0492	0.7012	0.1526	0.0301
CURIO	0.7052	0.1555	0.0491	0.7218	0.1423	0.0292
Qwen3-235B
TTT-Discover	0.7011	0.1562	0.0491	0.7254	0.1401	0.0291
CURIO	0.7156	0.1501	0.0490	0.7408	0.1334	0.0281

The epoch is selected by held-in MSE; neither held-out corpus is used for selection. Means over three runs.

CURIO improves on TTT-Discover for Erdős, AC1–AC3, and Hadamard at every tested scale, matches the best reported performance on Circle Packing, and improves denoising Score and MSE on both held-out corpora. The largest gain is on Hadamard at 235B (+18.3%).

Analysis

More Structurally Diverse Programs

We measure code-level structural variation among generated programs with OpenEvolve's island code-diversity metric, which compares pairs of programs by differences in code length, line count, and character sets. CURIO generally maintains higher diversity than TTT-Discover over training. This is a code-level diagnostic: it does not by itself establish semantic diversity of the underlying constructions.

Heatmap of code diversity by training stage for Erdős and Circle Packing; CURIO is higher than TTT-Discover in the early, middle, late, and final stages. — **Code diversity by training stage.** On Erdős and Circle Packing, CURIO has higher code diversity than task-only RL (TTT-Discover) in every stage: early (epochs 0–12), middle (13–25), late (26–38), and final (39–49).

Code diversity over epochs on Erdős — Erdős

Code diversity over epochs on Circle Packing — Circle Packing

Code diversity over epochs on HadamardMatrix — HadamardMatrix

Ablations

What Makes the Curiosity Signal Work

On Qwen3-14B across six tasks, we vary three design choices of the ICWM. Each panel fixes its reference configuration at 50 on every task, so values above 50 indicate improvement and values below 50 indicate degradation.

Radar chart comparing curiosity weights 0, 0.3, 0.5 and 0.8 across six tasks — (a) Curiosity weight $\eta_0$

Radar chart comparing MLP, GRU and Transformer forward predictors across six tasks — (b) ICWM predictor architecture

Radar chart comparing coupled, frozen-actor and frozen external-LLM representations across six tasks — (c) Curiosity representation

Curiosity strength matters

$\eta_0 = 0.3$ is best on the autocorrelation tasks and $\eta_0 = 0.5$ on Erdős, Hadamard, and denoising. The largest weight, $0.8$, underperforms the task-only control on some tasks.

A simple MLP predictor is enough

The default MLP forward predictor is competitive with GRU and Transformer alternatives; the best architecture depends on the task.

Coupling to the evolving policy helps

Reading states from the current actor leads on most tasks, compared with a frozen initial actor or a frozen external LLM.

Get Started

Run CURIO on Your Own GPUs

CURIO runs fully locally: generation on Ray Data LLM (vLLM) and LoRA training on DeepSpeed, with resumable per-stage checkpoints and launchers for single nodes and SLURM clusters. See the repository for setup, the required Ray patch, and all configuration options.

git clone https://github.com/ft2023/CURIO.git && cd CURIO
source .venvs/curio-runtime/bin/activate

export CURIO_ERDOS_CONFIG=qwen3_8b_2xH100
export CURIO_EVAL_PYTHON=$PWD/.venvs/curio-eval-erdos/bin/python
export CURIO_LOG_ROOT=/path/to/your/log/root

bash tasks/erdos/launchers/single_node/run_all.sh

BibTeX

@article{feng2026curio,
  title   = {CURIO: Curiosity-Driven Test-Time Learning for Open-Ended Discovery},
  author  = {Feng, Tao and Yu, Fangxu and Lei, Zijie and Zou, Jiaru and Jiang, Changjiang and Yan, Yi and You, Jiaxuan and Lu, Pan},
  journal = {arXiv preprint},
  year    = {2026}
}