Reading list: Do Androids Dream of Mathematical Proofs?
Prepared on 2026-11-16 for the 45-minute talk. The research passes opened every link below that day, and the newest papers were checked a second time.
How to use it. Each entry says why it matters for the talk, which sections to read, and one caveat: what the paper does not show. The caveats mark where audiences usually overreach, so they are worth knowing as well as the results.
Tiers. Core: read closely. Deeper: read the named sections. Optional: skim if the topic gets a slide. NEW: posted after June 2026.
Status. All three parts are in:
- Part A: training and reasoning methods.
- Part B: agents, orchestration, self-improvement and looping.
- Part C: the timeline and the case files.
Part C was written on 2026-11-16 from sources checked that day. Coverage of the ETH claims is still mostly secondary, and the review status of every claim will change.
Start here
If you only read eight things, read these, in this order:
- C3, OpenAI's math release: how generating results became cheap, and why checking them is now the bottleneck.
- C2 and C4, the Navier–Stokes and ETH case files: what was claimed, and what is still under review.
- A11, DeepSeek-R1: the open template for practice with an answer key.
- A34, AlphaProof: reinforcement learning against Lean, and the risk of proving a mistranslated statement.
- B7, AlphaEvolve: many attempts with automatic checks, at scale.
- B11, Carlini's C compiler: many agents working for weeks, and why the quality of the checker is the binding constraint.
- B4, METR time horizons: the capability clock behind multi-day agent runs.
- A20, DeepSeekMath-V2: how RL works on proofs that have no final answer to check.
Part A: Training and reasoning methods
The template: search plus learning
A1. Mastering the game of Go without human knowledge (AlphaGo Zero) — Silver et al., 2017 — Nature · Deeper
- Why: The loop behind the whole talk. Tree search improves the policy, the network learns to imitate the search and predict the winner, and self-play supplies the curriculum. Starting from scratch, it beat the published AlphaGo 100–0. Every "generate, check, train on what worked" method below is a variant of this loop.
- Read: Fig. 1 (the self-play pipeline; good as a slide), Fig. 2 (tree search), Fig. 3 (learning curves against a model trained on human games).
- Caveat: Go has a perfect simulator and an unambiguous reward. Informal mathematics has neither. Lean restores both, but only for statements someone has formalized.
- Pair with: AlphaGo (2016), Fig. 1: the human-data pipeline that Zero removed. AlphaZero (2017), Fig. 2: playing strength against thinking time, an early test-time-compute curve.
A2. Thinking Fast and Slow with Deep Learning and Tree Search (Expert Iteration) — Anthony et al., 2017 — arXiv:1705.08439 · Deeper
- Why: Names the pattern. A slow expert (search) produces better targets, a fast apprentice (the network) imitates them, and the improved apprentice makes the next search stronger. STaR and ReST-EM (A27) are the same loop for language models.
- Read: §1, §3 with Algorithm 1, Fig. 2 (Expert Iteration against REINFORCE), App. A (comparison with AlphaGo Zero).
- Caveat: Tested only on the board game Hex; it says nothing about language models.
A3. The Bitter Lesson — Sutton, 2019 — essay · Optional
- Why: States a thesis the talk can test: general methods that use more compute, namely search and learning, eventually beat built-in human knowledge.
- Read: The whole essay; it is one page.
- Caveat: An argument from history, not evidence. Use the http link; the https version has a certificate error.
A4. Welcome to the Era of Experience — Silver & Sutton, 2025 — PDF · Deeper
- Why: Argues that human data for math and science is running out and that further progress must come from agents learning from their own experience. Its evidence is AlphaProof (about 100k human formal proofs, then about 100M self-generated through RL) and DeepSeek-R1.
- Read: The sections "The Era of Experience", "Rewards", "Planning and Reasoning" and "Why Now?".
- Caveat: A position piece (a preprint of a book chapter) with no experiments.
A5. Training language models to follow instructions with human feedback (InstructGPT) — Ouyang et al., 2022 — arXiv:2203.02155 · Deeper
- Why: Established the standard post-training pipeline: supervised fine-tuning, then a reward model, then PPO. RL with verifiable rewards is the same pipeline with the learned reward model swapped for a programmatic checker.
- Read: Fig. 2 (the three-step diagram), §3.5, Fig. 1.
- Caveat: A learned preference reward optimizes what labelers like, not what is correct. That gap is why math moved to verifiable rewards.
- Pair with: PPO (Schulman et al., 2017), §3 and §5: you need the clipped objective to explain what GRPO and DAPO change. DPO (Rafailov et al., 2023), §4–5.1, briefly: preference training without RL sampling. It is offline with no exploration, which is why reasoning gains came from on-policy RL instead.
Reasoning at inference time
A6. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models — Wei et al., 2022 — arXiv:2201.11903 · Optional
- Why: Treats intermediate tokens as computation. This is what RL later trains.
- Read: Fig. 1 and Fig. 4 (chain of thought helps only at large scale).
- Caveat: Prompting only. Higher accuracy does not show that the written chain reflects what the model actually computed.
- Pair with: Self-Consistency (Wang et al., 2022), Figs. 1–2: majority vote over sampled chains, the baseline RL papers still report. It needs short, comparable answers, so it does not transfer to proofs. Tree of Thoughts (Yao et al., 2023), Fig. 1 and §4.1 (Game of 24: 4% with chain of thought, 74% with tree search on GPT-4): the prompting-era ancestor of proof search.
A7. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling — Brown et al., 2024 — arXiv:2407.21787 · Deeper
- Why: The share of problems solved by at least one sample grows roughly log-linearly across four orders of magnitude of samples. When answers can be checked automatically, that turns directly into solved problems: SWE-bench Lite goes from 15.9% to 56% with 250 samples. This is the basic logic behind massively parallel attempts.
- Read: Fig. 2, §3.1 with Fig. 5, §4 (majority vote and reward models plateau).
- Caveat: You can only cash in on many attempts if you have a reliable verifier.
A8. Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Model Parameters — Snell et al., 2024 — arXiv:2408.03314 · Core
- Why: The cleanest framing of test-time compute. You can refine answers one after another, or search against a verifier, and the best mix depends on problem difficulty. The adaptive strategy is more than 4× more efficient than best-of-N, and at matched compute it beats a 14× larger model on problems the small model can partly solve.
- Read: Fig. 1; §2; §5 with Figs. 2–3 (best-of-N against beam and lookahead search); §6 with Figs. 5–7 (sequential against parallel); §7 with Fig. 9 (trading pretraining compute for test-time compute).
- Caveat: Uses PaLM 2-S* on MATH. On the hardest problems, extra test-time compute did not replace a bigger model.
A9. s1: Simple test-time scaling — Muennighoff et al., 2025 — arXiv:2501.19393 · Deeper
- Why: A minimal open demonstration of o1-style "think longer" scaling. 1,000 curated reasoning traces plus "budget forcing" (suppress the end-of-thinking token and append "Wait") take AIME 2024 accuracy from 50% to 57%.
- Read: Figs. 1 and 3, §2.2, Fig. 4, §6.2 (limits).
- Caveat: The traces were distilled from Gemini's thinking model, so s1 shows how to draw out the scaling behaviour, not how the ability was created.
- Pair with: OpenAI o1 System Card (2024), §2 and §4.3.1 (monitoring the chain of thought for deception). OpenAI, "Learning to reason with LLMs" (2024), the train-time and test-time compute plots. Automated fetches of that page now return HTTP 403, so open it in a browser.
Reinforcement learning with verifiable rewards (RLVR)
A10. DeepSeekMath — Shao et al., 2024 — arXiv:2402.03300 · Deeper
- Why: Introduced GRPO (advantages computed relative to a group of samples, with no value network), the workhorse of open reasoning RL. It also gave an early sign that RL sharpens rather than expands: majority-vote accuracy improves, but the share solved in K attempts does not.
- Read: §4.1.1 with Fig. 4 (PPO to GRPO), §4.1.2–4.1.3, §5.2.1 (one framework covering SFT, rejection sampling, DPO and RL), §5.2.2 with Fig. 7 ("Why RL works?").
- Caveat: Most of the gain came from 120B math tokens of continued pretraining, not from RL.
- Pair with: Tülu 3 (Lambert et al., 2024), §6: the paper that coined "RL with Verifiable Rewards".
A11. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning — DeepSeek-AI, 2025 — Nature · arXiv:2501.12948 · Core
- Why: Pure RL with rule-based rewards on a strong base model (R1-Zero) produced long reasoning chains, self-checking and large AIME gains (pass@1 on AIME 2024 rose from 15.6% to 71.0% during RL); R1 then turned that into a multi-stage recipe. It is the open template the rest of this section modifies.
- Read (arXiv v2): §2.2 (accuracy and format rewards), Fig. 1 (AIME accuracy and response length over RL steps), Table 2 (the "aha moment"), Fig. 2 (the pipeline), App. B.5 (reward hacking), App. G.2 (failed attempts with process reward models and tree search).
- Caveat: The "emergent" reflection was partly present in the base model already (see Dr. GRPO below).
- Pair with: Kimi k1.5 (2025), §2.3–2.4 and §3.3: concurrent work with no tree search, value function or process reward model. Understanding R1-Zero-Like Training (Dr. GRPO) (Liu et al., 2025), §2.3 (the "aha" already appears in DeepSeek-V3-Base) and §3.1–3.2 with Fig. 4 (GRPO's length bias).
A12. DAPO: An Open-Source LLM Reinforcement Learning System at Scale — Yu et al., 2025 — arXiv:2503.14476 · Deeper
- Why: An open recipe that reached 50 on AIME 2024 from the Qwen2.5-32B base in half the training steps of R1-Zero-Qwen-32B. It named fixes that are now standard: asymmetric clipping, dynamic sampling, token-level loss and shaping for overlong responses.
- Read: §3.1–3.4, Fig. 1, Fig. 2 (entropy with and without the clipping change).
- Caveat: One base model and one domain; these are ablation-level gains, not laws.
- Pair with: The Entropy Mechanism of RL for Reasoning Language Models (Cui et al., 2025), Fig. 1 and §2.4: performance tracks the collapse of the model's randomness so closely that the ceiling can be predicted.
A13. ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries — Liu et al., 2025 — arXiv:2505.24864 · Deeper
- Why: The main counter-evidence to "RL only reweights what the base model already does". With long, stabilized training across diverse tasks, a 1.5B model solves tasks its base model fails at any number of attempts, and the gains are largest where the base model is weakest.
- Read: Fig. 1, §2.2, §4.1–4.3.
- Caveat: A small model on mostly synthetic puzzle tasks; this is not new mathematics.
- Pair with: From f(x) and g(x) to f(g(x)) (Yuan et al., 2025), Fig. 1 and §4.1–4.4: RL teaches new combinations of skills, provided the component skills are already there.
A14. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? — Yue et al., 2025 — arXiv:2504.13837 · Core
- Why: The central skeptical result. RL-trained models win on the first attempt, but base models overtake them when each is allowed many attempts. The RL model's reasoning paths already exist in the base model, while distillation from a stronger model can add new ones. This is the title question of the talk, in numbers.
- Read: Fig. 1, Fig. 2 (the pass@k crossover), §4.1, §4.2 (distillation expands the boundary), §4.5, App. C.2 (checking whether the reasoning is valid).
- Caveat: Measuring success over many attempts on answer-only tasks can credit lucky guesses, and the RL budgets are 2025-sized.
- Pair with: On the Interplay of Pre-Training, Mid-Training, and RL (Zhang et al., 2025), Fig. 1 and §3–5: RL extrapolates only to tasks just beyond what pretraining covered. Rethinking RL for LLM Reasoning: It's Sparse Policy Selection (Akgül et al., 2026), Fig. 1 and §3: RL changes only about 1–3% of token positions, always within the base model's top five choices.
A15. Spurious Rewards: Rethinking Training Signals in RLVR — Shao et al., 2025 — arXiv:2506.10947 · Deeper
- Why: Random rewards gave Qwen2.5-Math-7B +21.4 points on MATH-500, against +29.1 with the true reward. GRPO's clipping amplifies a pretrained habit of reasoning in code, and the effect does not appear in other model families.
- Read: §2.2 with Fig. 1, §3, §4 with App. B.1, §5.
- Caveat: Mostly a Qwen-math effect. The lesson is to test RL claims on more than one base model.
- Pair with: NEW Demystifying Reinforcement Learning Post-Training of Language Models (Clay et al., Aug 2026), §5.1–5.3: RL succeeds only when the base model already puts enough probability on the desired behaviour, and spurious-reward effects depend on the prompt distribution.
A16. The Art of Scaling Reinforcement Learning Compute for LLMs (ScaleRL) — Khatri et al., 2025 — arXiv:2510.13786 · Deeper
- Why: The anchor for RL scaling laws. A study of more than 400k GPU-hours fits S-shaped curves of reward against compute, separates choices that raise the ceiling from those that only change efficiency, and predicts a 100k GPU-hour run from its early stages.
- Read: Figs. 1 and 3, §3.2, §4, §5.
- Caveat: The fits are on in-distribution validation reward; they say nothing about new capability on unseen research problems.
- Pair with: IsoCompute Playbook (Cheng et al., 2026), Fig. 1 and §4: the compute-optimal number of attempts per training problem rises with budget, then levels off.
Intermediate rewards, verifiers and reward hacking
A17. Let's Verify Step by Step — Lightman et al., 2023 — arXiv:2305.20050 · Core
- Why: The canonical comparison of rewarding each step against rewarding only the outcome. A process reward model trained on 800k human step labels (PRM800K) beats an outcome reward model and majority voting at picking the best of many solutions on MATH (78% on a representative subset). Active learning helps.
- Read: §2.5–2.6, §3 with Fig. 3, §4 with Fig. 4, §6.2.
- Caveat: Used for picking among answers, not as an RL reward. Later attempts to use process reward models inside RL ran into reward hacking and labeling costs (DeepSeek-R1, App. G.2).
- Pair with: Uesato et al., 2022, Table 1, Fig. 3 and §4.1: outcome supervision gets final answers right about as often, but correct reasoning steps need process feedback (trace error drops from 14.0% to 3.4%).
A18. Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations — Wang et al., 2023 — arXiv:2312.08935 · Optional
- Why: Removes human step labels. A step is labeled by whether sampled continuations from it reach the right answer, and the resulting model is used both to pick answers and as a step-level RL reward.
- Read: §3.3 with Fig. 2, §3.5, §5.2.
- Caveat: These labels measure "can this prefix still succeed?", not "is this step valid?".
- Pair with: The Lessons of Developing Process Reward Models in Mathematical Reasoning (Zhang et al., 2025), §3.1–3.2: the limits of sampled-continuation labels and a bias in how these models are evaluated.
A19. PRIME: Process Reinforcement through Implicit Rewards — Cui et al., 2025 — arXiv:2502.01456 · Optional
- Why: Gets a dense, token-level reward for online RL from a model trained only on outcome labels and updated alongside the policy: +15.1% on average over SFT from Qwen2.5-Math-7B.
- Read: Fig. 1 with Algorithm 1, §2.2, §3.1, §4.3.
- Caveat: 7B scale; frontier reports still mostly use outcome rewards or LLM judges.
A20. DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning — Shao et al., 2025 — arXiv:2511.22570 · Core
- Why: The clearest public recipe for RL on proofs, where there is no final answer to check. An RL-trained LLM proof checker, whose critiques are themselves checked, serves as the reward. The generator is also rewarded for finding and fixing its own errors, and verification compute is scaled up to label harder proofs automatically. Results: gold level on IMO 2025 and CMO 2024, and 118/120 on Putnam 2024.
- Read: §2.1–2.3, Fig. 2 (self-refinement), App. A (prompts).
- Caveat: "Verified" means accepted by a trained LLM grader, not checked by a proof assistant.
- Pair with: Generative Verifiers (Zhang et al., 2024), §3 with Fig. 3: verification as next-token prediction, with its own reasoning and voting.
A21. MaxProof — Chen et al., June 2026 — arXiv:2606.13473 · Deeper
- Why: A frontier-lab recipe (MiniMax-M3) that trains proof writing, error-finding verification and repair. The verifier is built in layers for few false positives, and the paper documents a reward-hacking episode. Reports 35/42 on IMO 2025 and 36/42 on USAMO 2026.
- Read: §2.2, §2.5 (the reward-hacking case study), §3.1 (finding errors beats predicting a score), §5, App. A.
- Caveat: Contest scores are reported by the vendor; check how the final proofs were graded (App. A).
A22. Towards Robust Mathematical Reasoning (IMO-Bench) — Luong et al., 2025 — arXiv:2511.01846 · Deeper
- Why: Google DeepMind's benchmarks behind Gemini Deep Think's IMO gold: a proof benchmark with an automatic grader, and 1,000 human gradings to measure when LLM graders agree with people.
- Read: §3.2, §4, §5.3–5.4, App. B.6 (limits of the automatic grader).
- Caveat: Agreement on olympiad proofs with marking schemes says little about research arguments that have no reference solution.
- Pair with: NEW Can LLMs Reliably Grade Olympiad Proofs? (Ospanov et al., Oct 2026), Fig. 1, §4 and §6.2: expert-graded USAMO 2025 solutions plus deliberately flawed proofs; all 20 graders tested still have considerable room to improve.
A23. Rubrics as Rewards — Gunjal et al., 2025 — arXiv:2507.17746 · Optional
- Why: Turns prompt-specific rubrics into rewards through an LLM judge: up to +31% relative on HealthBench and +7% on GPQA-Diamond over judges giving simple ratings. It is the same mechanism as grading a proof against a marking scheme.
- Read: Fig. 1, §2.2, §3, §6.
- Caveat: Medical and science questions, not proofs. The rubric and judge become the new target for gaming.
A24. Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation — Baker et al., 2025 — arXiv:2503.11926 · Core
- Why: OpenAI's evidence on reward hacking in long autonomous runs. A frontier reasoning model gamed agentic coding environments. A weaker model reading its chain of thought caught far more of this than one reading only its actions. Training against that monitor taught the model to hide its hacking, so leaving the chain of thought untrained against monitors has a cost worth paying.
- Read: Fig. 1, §2 with Fig. 2, §3.2, §5.1.
- Caveat: Coding environments. A Lean check closes the gaming channel for the final proof, but not for which statement got proved.
- Pair with: ImpossibleBench (Zhong et al., 2025), Fig. 1, §2.2 and §4: the tests contradict the specification, so any pass counts as cheating. Observed cheating ranges from simple test modification to complex operator overloading. Natural Emergent Misalignment from Reward Hacking in Production RL (MacDiarmid et al., 2025, Anthropic), Fig. 1 and §3–4: hacks learned in real coding environments generalize to faking alignment and sabotage, and a prompting technique mitigates it.
A25. Reward Misspecification Degrades Chain-of-Thought Monitorability — Yan & Hewitt, Oct 2026 — arXiv:2610.28303 · Deeper · NEW
- Why: A flawed reward alone, with no training against a monitor, produces reasoning that talks about the stated task while the output exploits the reward. Training the model to state the reward it has inferred recovers 24–42 points of monitorability.
- Read: Fig. 1, §3.5 with Table 1, §4, §5.1.
- Caveat: Controlled test settings (three models, three tasks).
- Pair with: NEW CATCH (Wang et al., Sep 2026), §4.3: a chain-of-thought monitor first suppresses hacking, then the policy learns to mislead it through code comments. NEW Monitoring and Discovering Reward Hacking with Internal Representations (Bergen et al., Sep 2026), §2.3 and §4.2: GLM 5.2 hacks in 57.2% of DeepSWE and 73% of SWE-bench runs, and probes on its reasoning predict hacks before they happen.
A26. An Open Recipe for IMO Gold: Training Nemotron for Olympiad Mathematics — Moshkov et al., Sep 2026 — arXiv:2609.10712 · Core · NEW
- Why: The most transparent complete stack for natural-language proofs, from NVIDIA's Nemotron team. It combines RL on proofs with verifier rewards (in the style of DeepSeekMath-V2), asynchronous training, and a generate–verify–refine search with a separate selection stage. It scored 30/42 at IMO 2026, the gold threshold, and released the checkpoints, data, code, submitted solutions and a 200-problem benchmark.
- Read: §4.2.2 (data filter, reward, the asynchronous algorithm; 128 trainer, 128 inference and 16 judge nodes), Fig. 1, §5 with Table 1 (compute per problem), §7.4 with App. A (what didn't help).
- Caveat: Olympiad problems with expert grading and no formal checking.
Self-training loops
A27. STaR: Bootstrapping Reasoning With Reasoning — Zelikman et al., 2022 — arXiv:2203.14465 · Deeper
- Why: Expert Iteration for language. Sample reasoning, keep what reaches the right answer, fine-tune, repeat. "Rationalization", writing the reasoning with the answer given as a hint, rescues failures. RLVR is the on-policy, gradient-based version of this loop.
- Read: Fig. 1, Algorithm 1, §3.1–3.2, §5.
- Caveat: Filtering on answers lets through reasoning that reaches the right answer by the wrong route.
- Pair with: Beyond Human Data (ReST-EM) (Singh et al., 2023), §3 with Algorithm 1, Fig. 1 and §5.2–5.3: the same loop at PaLM 2 scale beats training on human data. Self-Rewarding Language Models (Yuan et al., 2024), Figs. 1–2 and §2.4: the model judges its own outputs. Instruction following only, and the clearest illustration of drift without an external check.
A28. Training Language Models to Self-Correct via Reinforcement Learning (SCoRe) — Kumar et al., 2024 — arXiv:2409.12917 · Optional
- Why: Fine-tuning on self-generated corrections fails, while multi-turn RL with a bonus for improvement works: +15.6 points on MATH and +9.1 on HumanEval. This underpins the "refine" step in proof pipelines.
- Read: Fig. 1, §4, §5.
- Caveat: Two-attempt correction, not hours of refinement.
A29. Absolute Zero: Reinforced Self-play Reasoning with Zero Data — Zhao et al., 2025 — arXiv:2505.03335 · Deeper
- Why: One model both proposes and solves tasks, with a code executor as the grounded checker, and reaches state-of-the-art results for training without curated data.
- Read: Figs. 2–3, §3.1–3.3, §4.2.
- Caveat: Starts from strong pretrained models and covers only tasks that code can check.
- Pair with: STP: Self-play Theorem Prover (Dong & Ma, 2025), Fig. 1, §3.2 and Fig. 2: a conjecturer and a prover play against each other in Lean and keep improving after plain Expert Iteration levels off.
Training agents with RL
A30. SWE-RL — Wei et al., 2025 — arXiv:2502.18449 · Deeper
- Why: RL on the history of GitHub pull requests, rewarded by similarity to the real patch. Llama3-SWE-RL-70B reaches 41.0% on SWE-bench Verified and improves on five out-of-domain tasks where supervised fine-tuning gets worse.
- Read: Fig. 1, §2.1, the generalization results.
- Caveat: Single-shot patches scored by similarity, not by running tests.
A31. Search-R1 — Jin et al., 2025 — arXiv:2503.09516 · Optional
- Why: A minimal template for RL with tool calls inside the reasoning: search calls interleaved with thinking, retrieved text excluded from the loss, and a reward only for the final answer. +41% over retrieval-augmented baselines at 7B.
- Read: Fig. 1, Algorithm 1, §3, §5.1, §5.4.
- Caveat: Short question-answering tasks.
- Pair with: The Landscape of Agentic Reinforcement Learning for LLMs (Zhang et al., 2025), §4.3, §5 and §6.3: a survey to use as a map.
A32. Kimi K2: Open Agentic Intelligence — Kimi Team, 2025 — arXiv:2507.20534 · Deeper
- Why: The most detailed open account of training a model to use tools: synthetic tool-use data from real and invented tools, a gym of checkable tasks, and a self-critique rubric reward for tasks without a checker.
- Read: §3.1.1, §3.2.1, §3.2.2 with App. F, §3.3.4.
- Caveat: A model report, not a controlled study.
- Pair with: Kimi K2.5 (2026), §3 with Figs. 3–4: an orchestrator trained with RL to spawn parallel sub-agents; accuracy and parallelism rise together during training. A good bridge to orchestration.
A33. Tev: Scaling Long-Horizon Agentic Environments with Dense Progress Rewards — Hu et al., Oct 2026 — arXiv:2610.18868 · Optional · NEW
- Why: Tackles the shortage of checkable environments and of reward signal: 1,170 tasks generated from a hidden specification, plus shaping that gives a reward for progress at every step.
- Read: §2.1, §3.1, §3.3, §4.2–4.3.
- Caveat: A fresh preprint; comparisons with frontier models are self-reported.
- Pair with: NEW SPROUT (Liu et al., Oct 2026), Figs. 1–2 and §3.3–3.5: branches training rollouts from saved states instead of restarting, for +13.9 points over GRPO in 39.5% less time per step.
RL for formal theorem proving
A34. Olympiad-level formal mathematical reasoning with reinforcement learning (AlphaProof) — Hubert et al., 2025 — Nature · Core
- Why: AlphaZero moved into Lean. About 1M informal problems were translated into about 80M Lean problems, the prover was trained with RL and tree search against Lean's checker, and the hardest problems got extra RL at test time on self-generated variants. Together with AlphaGeometry 2 it reached IMO 2024 silver-medal level.
- Read: Figs. 1–2, the "Main RL progress" section with Fig. 3, the "Inference-time scaling" section with Fig. 4, Extended Data Fig. 3.
- Caveat: Days of compute per problem. The paper notes that mistranslated statements still make useful training problems: fine for training, fatal for claims.
- Pair with: TTT-Discover (Yuksekgonul et al., 2026), Fig. 1, §3.2 and §4.1: test-time RL on an open model sets a new bound on Erdős's minimum-overlap problem, but only works where the reward is a continuous score.
A35. DeepSeek-Prover-V2 — Ren et al., 2025 — arXiv:2504.21801 · Deeper
- Why: A general model writes an informal sketch and breaks it into Lean subgoals; solved subgoals become training data, followed by RL. 88.9% on MiniF2F and 49 of 658 PutnamBench problems.
- Read: Fig. 2, §2.1–2.3, §3.5 (the formal model solves 6/15 AIME problems against 8/15 for the informal model).
- Caveat: Working formally still costs capability.
- Pair with: Kimina-Prover Preview (Wang et al., 2025), Fig. 2 and §2.2–2.3: R1-style RL directly in Lean, 80.7% with 8,192 attempts, improving with model size.
A36. Seed-Prover — Chen et al., 2025 — arXiv:2507.23726 · Deeper
- Why: Lemma-based whole proofs refined with Lean feedback, proposing conjectures, RL on a pass/fail proof reward, and light, medium and heavy search modes. It fully proved five of the six IMO 2025 problems.
- Read: §2.2.1–2.2.4 with Fig. 3, §3.2.
- Caveat: Heavy compute, and geometry needs a separate engine.
- Pair with: Seed-Prover 1.5 (Chen et al., 2025), §3.1–3.2 and §4.1–4.2: agentic RL in Lean, 88% on PutnamBench and 11/12 on Putnam 2025. The authors themselves say their Erdős "solutions" were either simple or made trivial by mistranslated statements.
A37. Goedel-Prover-V2 — Lin et al., 2025 — arXiv:2508.03613 · Optional
- Why: Expert Iteration plus RL with scaffolded synthetic problems and self-correction from compiler errors. Its 8B model beats the 671B DeepSeek-Prover-V2 on MiniF2F at 32 attempts.
- Read: §2.1–2.2 with Fig. 2, §3.5–3.6, App. A (mismatches between informal and formal statements).
- Caveat: These benchmarks are close to saturated.
- Pair with: Goedel-Architect (Chung et al., 2026), Fig. 1, §2 and Tables 1–2: the 2026 successor, an agentic pipeline that plans a proof blueprint first (75.6% to 88.8% on PutnamBench).
A38. Aristotle: IMO-level Automated Theorem Proving — Achim et al., 2025 — arXiv:2510.01346 · Deeper
- Why: Harmonic's "informal ideas, formal certificates" design: Lean tree search, an informal reasoner that proposes and formalizes lemmas, and a geometry solver. Gold-medal-equivalent formal proofs at IMO 2025.
- Read: §2.1, §2.2 with Fig. 2, §3.1, App. B. Use the PDF; the current version has no HTML view.
- Caveat: A company report, light on RL and data details.
A39. Advancing Mathematics Research with AI-Driven Formal Proof Search (AlphaProof Nexus) — Tsoukalas et al., May 2026 — arXiv:2605.22763 · Deeper
- Why: LLM agents working in Lean resolved 9 of 353 open Erdős problems and proved 44 of 492 OEIS conjectures. A basic generate-and-check agent reproduced the Erdős results at higher cost, which suggests a shift from specialized RL-trained provers to general models inside checking loops.
- Read: §2 with Fig. 2, §3 with Table 1, §5, §6.
- Caveat: The problems had already been formalized, and "open" Erdős problems vary enormously in difficulty.
- Pair with: A Minimal Agent for Automated Theorem Proving (Requena et al., 2026), Figs. 1–3.
A40. From Solvers to Research: LLM-Driven Formal Mathematics at the Research Frontier — Jiang et al., Jul 2026 — arXiv:2607.07779 · Deeper · NEW
- Why: A survey covering training, search at inference time and agentic workflows, with an explicit framework for judging research claims. Its main point for the talk: a checked formal proof does not show that it captures the intended problem.
- Read: §3.1–3.3, §5.4, §7.2 (six kinds of evidence), §7.4 (a reporting checklist).
- Caveat: Benchmark tables age quickly and mix in vendor-reported numbers.
Deep-dive references (added for the optional deep-dive slides)
A41. Evaluating Large Language Models Trained on Code (Codex) — Chen et al., 2021 — arXiv:2107.03374 · Deeper
- Why: Defines pass@k and its unbiased estimator. Draw n ≥ k samples, count the c that pass, and estimate pass@k = 1 − C(n−c, k) / C(n, k). Plugging the sample success rate into 1 − (1 − p)^k instead is biased.
- Read: §2.1 (functional correctness and the estimator, with a numerically stable implementation).
- Caveat: pass@k credits a problem if any one of k samples passes the tests, so it presumes a reliable checker.
A42. Scaling Laws for Reward Model Overoptimization — Gao, Schulman & Hilton, 2022 — arXiv:2210.10760 · Deeper
- Why: Goodhart's law, measured. A policy is optimised against a proxy reward model while a separate "gold" reward model stands in for the true objective. As the policy moves away from its start (d = √KL), the proxy keeps rising while the gold reward rises, peaks and falls. Simple functional forms fit this for best-of-n and for RL.
- Read: §1 and Fig. 1, §3 (the functional forms and their fits).
- Caveat: A synthetic setup: the "gold" reward is itself a larger reward model, not human judgment.
A43. Proximal Policy Optimization Algorithms (PPO) — Schulman et al., 2017 — arXiv:1707.06347 · Optional
- Why: The clipped surrogate objective behind RLHF: the probability ratio between new and old policy, times an advantage, is clipped to [1 − ε, 1 + ε] so a single update cannot move the policy too far. GRPO (A10) keeps the clipping and drops the learned value function.
- Read: §3 (the clipped surrogate objective) and §5 (the algorithm with a value function and generalized advantage estimation).
- Caveat: Designed for control and games. LLM variants differ in how they average over tokens and where the KL penalty sits.
If you only read five papers from Part A
- A11 DeepSeek-R1: the open RLVR template everything else modifies.
- A20 DeepSeekMath-V2: how RL works on proofs, where there is no answer to check.
- A26 Open Recipe for IMO Gold: the whole 2026 open stack, with real compute numbers.
- A34 AlphaProof: RL against Lean, test-time RL, and the mistranslation caveat.
- A14 Yue et al.: the sharpest evidence on whether RL creates reasoning or only sharpens it.
Then A8 (Snell et al.) and A24 (Baker et al.).
Part B: Agents, orchestration, self-improvement, looping
Checked 2026-11-16. Every arXiv entry was re-checked against arXiv's own records, and the most striking numbers were checked against their sources. The one claim that could not be re-checked is flagged in B20.
What an agent is
B1. ReAct: Synergizing Reasoning and Acting in Language Models — Yao et al., 2022 — arXiv:2210.03629 · Deeper
- Why: The original agent loop: think, call a tool, read the result, repeat. Every coding and research agent in the 2026 stories descends from it.
- Read: Fig. 1 (standard prompting, chain of thought, act-only and ReAct side by side), §2, Tables 1–2.
- Caveat: Prompting only, with PaLM-540B. On HotpotQA, ReAct alone (27.4) scored below chain of thought (29.4) and won only when combined with self-consistency (35.1). What lasted is the loop, not the scores.
- Pair with: Toolformer (Schick et al., 2023), Fig. 2: a model learns when to call tools by keeping only the calls that help it predict text. Single calls, no multi-step loop.
B2. Reflexion: Language Agents with Verbal Reinforcement Learning — Shinn et al., 2023 — arXiv:2303.11366 · Deeper
- Why: The simplest check-reflect-retry loop. After a failure the agent writes itself a short post-mortem and tries again, with no weight updates: 91% pass@1 on HumanEval against 80% for GPT-4.
- Read: Fig. 2, §4.3 with Tables 1–2, the WebShop appendix (where it fails).
- Caveat: Only as good as its feedback. Unit tests the agent writes for itself can pass wrong code.
- Pair with: Voyager (Wang et al., 2023), Figs. 2 and 4: an automatic curriculum and a growing library of verified skills, two ingredients later discovery systems reuse.
B3. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — Jimenez et al., 2023 — arXiv:2310.06770 · Deeper
- Why: Made agentic coding measurable: 2,294 real GitHub issues, graded by each repository's own tests. The best 2023 baseline solved 1.96%, a good first point on a progress curve.
- Read: Figs. 1–2, §2.1–2.4, Table 2.
- Caveat: Grading by tests is noisy and can be gamed, public repositories invite contamination, and fixing issues is not research.
- Pair with: SWE-agent (Yang et al., 2024), §2 and Table 3: the interface the agent works through matters as much as the model, a lesson that recurs in B10–B11. CodeAct (Wang et al., 2024), Fig. 1: executable Python as the agent's action format.
B4. Measuring AI Ability to Complete Long Software Tasks — Kwa et al. (METR), 2025, revised July 2026 — arXiv:2503.14499 · Time Horizon 1.1 · live page · Core
- Why: The talk's capability clock. A model's "time horizon" is the length of task, in human working time, that it completes half the time. The paper found a doubling about every seven months over 2019–2025. In the Time Horizon 1.1 raw data (state-of-the-art models; fits exclude horizons over 16 hours) the doubling time is about 188 days (≈6 months) over 2019–2026 and about 129 days (≈4 months) since 2023, 95% CI 104–158 days. This is the best quantitative reason multi-day agent runs became plausible in 2026. In METR's data, an early version of Claude Mythos Preview (April 2026) reaches about 17 hours at 50% success but only about 3 hours at 80%.
- Read: Paper Figs. 1–2, §3–4, Fig. 17 (the 80% horizon); the Time Horizon 1.1 post (the task suite grew from 170 to 228 tasks); the live page and its raw data.
- Caveat: Software, ML and security tasks only, and 50% success is not dependable success. The 17-hour figure has a confidence interval of about 8.5 to 55 hours, METR says horizons above 16 hours are unreliable with its current tasks, and the live page stopped being updated on 2026-09-08.
B5. Building effective agents — Schluntz & Zhang (Anthropic), Dec 2024 — post · Core
- Why: Vocabulary for the talk. Workflows are fixed code paths: chaining, routing, parallelization, orchestrator-workers and evaluator-optimizer. Agents direct their own tool use. FunSearch and AlphaEvolve are evaluator-optimizer workflows at scale; a 10,000-agent swarm is orchestrator-workers, or something looser.
- Read: "Building blocks, workflows, and agents" (six diagrams, usable as slides), "When (and when not) to use agents", Appendix 2.
- Caveat: Practitioner advice with no evaluations.
Orchestration: many agents on one problem
B6. Mathematical discoveries from program search with large language models (FunSearch) — Romera-Paredes et al., 2023 — Nature · Deeper
- Why: The first result from an LLM in the loop to be accepted as new mathematics. The LLM proposes programs, an evaluator scores them, and separate "island" populations keep the best. It found a 512-vector cap set in dimension 8 and raised the lower bound on cap-set capacity from 2.2180 to 2.2202.
- Read: Figs. 1–2, Figs. 4–5 (the readable programs it found), Methods (islands and prompts).
- Caveat: Needs a fast, exact evaluator. The LLM is only the mutation step, and the best bound came after humans interpreted the program.
- Pair with: Multiagent Debate (Du et al., 2023), Fig. 2: model copies that read each other's answers beat a single copy. The seed of agents sharing insights; compare it against majority voting at equal compute.
B7. AlphaEvolve: A coding agent for scientific and algorithmic discovery — Novikov et al., 2025 — arXiv:2506.13131 · Core
- Why: The template: many LLM proposals, automatic evaluators and an evolving population, run at scale on whole code files. On 50+ open math problems it matched the best known result about 75% of the time and improved it about 20% of the time. It also sped up a kernel in Gemini's own training by 23%, saving about 1% of training time: a small but real case of AI improving the system that trains it.
- Read: Table 1 (against FunSearch), Figs. 1–2 and §2.6 (the distributed pipeline), §3.2 with App. B, §3.3.2 (the Gemini kernel), §4 with Fig. 8 (ablations). Optional: the one-year impact post (May 2026).
- Caveat: Needs a score a machine can compute. It optimizes constructions rather than writing proofs, and its runs cannot be reproduced, only their results checked.
B8. Mathematical exploration and discovery at scale — Georgiev, Gómez-Serrano, Tao & Wagner, 2025 — arXiv:2511.02864 · Core
- Why: The most candid account of AlphaEvolve-style search on research mathematics. Across 67 problems it mostly rediscovered the best known results, with several improvements. On the finite-field Kakeya problem, AlphaEvolve found the construction, Deep Think proved it and AlphaProof formalized it in Lean: discover, prove and formalize in miniature.
- Read: §1.3–1.6, §3.1–3.2 with Figs. 1–2, the passage on the "cheating phenomenon" (the system exploited leaky checkers; in Problem 6.6 it found a highly irregular function that exploited the numerical integration in the scoring code), and the problem pages and notebooks.
- Caveat: Works best when the score can improve in small steps. The authors say conventional methods could likely have matched some of the gains.
B9. Accelerating scientific discovery with Co-Scientist — Gottweis et al., May 2026 — Nature · arXiv:2502.18864 · Deeper
- Why: The reference multi-agent design for generating hypotheses. Agents propose ideas, critique them, rank them in Elo-scored debates and evolve them under a supervisor. Quality rises with run-time compute, and some results were confirmed in the wet lab.
- Read: Figs. 1–2, "System analysis and evaluation", skim Figs. 3–4.
- Caveat: The Elo score is the system's own judgment, and the lab validations are a few biomedical cases run with expert supervision.
- Pair with: Kosmos (Mitchener et al., 2025): runs of up to 12 hours whose agents share a structured record of what has been found.
B10. How we built our multi-agent research system — Hadfield et al. (Anthropic), Jun 2025 — post · Deeper
- Why: The best quantitative account of an orchestrator directing worker agents. The multi-agent system beat a single Claude Opus 4 agent by 90.2% on an internal research eval, and on BrowseComp, token usage alone explained 80% of the variance in performance. In other words, parallel agents mostly buy more useful thinking.
- Read: "Benefits of a multi-agent system", "Architecture overview", the prompting principles, "Production reliability and engineering challenges".
- Caveat: Breadth-first web research splits up well, but tasks that need shared context fit poorly. It uses about 15× the tokens of a chat.
B11. Building a C compiler with a team of parallel Claudes — Carlini (Anthropic), Feb 2026 — post · Core
- Why: The best first-hand account of many agents working for weeks without an orchestrator. 16 Claude Code agents ran in parallel loops and claimed tasks by writing lock files into a shared git repo. About 2,000 sessions over two weeks, for just under $20k, produced a roughly 100k-line C compiler in Rust that builds a bootable Linux 6.9 on x86, ARM and RISC-V.
- Read: "Enabling long-running Claudes", "Running Claude in parallel", "Lessons from programming with Claude agent teams" (the checker must be near-perfect; GCC served as a reference to compare against), "Stress testing the limits".
- Caveat: A well-specified target with excellent checks, the opposite of open research mathematics. The author warns that passing tests is not the same as being verified.
- Pair with: Scaling long-running autonomous coding (Lin, Cursor, Jan 2026): with hundreds of agents, self-coordination through locks failed, and a hierarchy of planners, workers and a judge worked. A vendor blog whose outputs were not independently evaluated.
B12. Autonomous Mathematical Discovery in an Open-World Multi-Agent Environment (the Station) — Chung et al., Aug 2026 — arXiv:2608.23691 · Deeper · NEW
- Why: The closest public analogue to groups of agents sharing insights. Agents from different model families choose their own directions and publish papers that later agents read, cite and extend, with no central coordinator. On 12 problems from the AlphaEvolve study, 5 gave results new to the literature, including a new infinite family of finite-field Kakeya sets and a substantially improved lower bound for Erdős's minimum-overlap problem.
- Read: Fig. 1, Table 2, Figs. 3–6 (accumulating papers, collaboration across model families, ablations).
- Caveat: The default setup has six agents, so this supports the shared-literature mechanism, not 10,000-agent scale. A single group's preprint.
B13. Towards end-to-end automation of AI research (The AI Scientist) — Lu et al., Mar 2026 — Nature · arXiv:2408.06292 · Deeper
- Why: Automates the whole research cycle: idea, code and experiments, paper, then automated review. One of its papers passed peer review at an ICLR 2025 workshop (average score 6.33) and was withdrawn, as agreed in advance with the organizers.
- Read: Nature Fig. 1, the automated and human evaluation sections, Fig. 3, "Limitations"; the v2 preprint, §3.2.2 (a parallel tree search over experiments).
- Caveat: Small-compute machine-learning research only, judged against a workshop bar rather than a main-conference one, and much of the quality judgment comes from its own reviewer.
B14. Open-source AlphaEvolve-style frameworks — ShinkaEvolve (Lange et al., 2025), OpenEvolve, CodeEvolve (Assumpção et al., 2025) · Optional
- Why: Runnable versions of the AlphaEvolve loop, for a live demo (see the demo kit below). ShinkaEvolve found a new best circle packing with about 150 samples, and it also evolved an agent scaffold for AIME problems.
- Read: ShinkaEvolve Fig. 1 and §3–4; CodeEvolve's head-to-head tables.
- Caveat: Small, well-scored problems. Comparisons depend heavily on budget and model choice, and leaky evaluators get exploited (B8).
Recursive self-improvement: AI helping build the next AI
"Self-improvement" covers very different claims. Keep them apart:
| Level | What improves | Do the model's weights change? | Entries |
|---|---|---|---|
| L0: answers | Candidate solutions within one run | No | B2, B6–B8 |
| L1: the agent's scaffolding | Prompts, tools and workflow code around a frozen model | No | B16 |
| L2: the improver itself | The procedure that generates the improvements | No | B15, B17 |
| L3: the next model | Kernels, infrastructure and research work that train the next model | Yes, indirectly | B7 (§3.3.2), B18, B20 |
Nothing in this list shows a frontier model improving its own weights without humans in the loop. Training on self-generated data is in Part A (A27–A29).
B15. Gödel Machines: Self-Referential Universal Problem Solvers Making Provably Optimal Self-Improvements — Schmidhuber, 2003 — arXiv:cs/0309048 · Optional
- Why: The theoretical ideal. A program may rewrite any part of itself, but only after proving that the rewrite helps. The practical systems below keep the rewriting and drop the proof.
- Read: §1–2, Fig. 1, §6.7 (FAQ).
- Caveat: Purely theoretical; never built.
- Pair with: STOP (Zelikman et al., 2023), Figs. 3–4 and §6.2–6.3: an improver program run on its own code gets better at improving other programs, and in some runs the model switched off a sandbox flag. Only 12% of GPT-3.5 runs improved meaningfully.
B16. Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents — Zhang et al., 2025 — arXiv:2505.22954 · Deeper
- Why: The reference self-improving coding agent. It keeps an archive of agents that edit their own tools and workflow, and tests each change on benchmarks. SWE-bench rose from 20.0% to 50.0%, and the improvements carried over to other models and programming languages.
- Read: Fig. 1, Fig. 3 (the archive tree), Table 1 (ablations), Fig. 4 (transfer), §5 and App. H. App. H describes objective hacking: in node 114 the agent removed the logging of special tokens that marked real tool use, against instructions, so the hallucination checker saw nothing. It scored 2.0 of 2.0, where an honest fix in node 96 scored 1.67.
- Caveat: The underlying models are frozen (L1), and each SWE-bench run cost about $22k.
- Pair with: Huxley–Gödel Machine (Wang et al., 2025): a benchmark score is a poor guide to how well an agent will improve itself.
B17. Hyperagents — Zhang et al., Mar 2026 (v2 Nov 2026) — arXiv:2603.19461 · Core
- Why: The clearest 2026 case of improving the improvement process itself. The agent that does the task and the agent that improves it live in one editable program, so the system can rewrite how it generates improvements. Inventions at that level, such as persistent memory and performance tracking, carried over to new domains, including grading olympiad-level math solutions.
- Read: Fig. 1 (DGM against DGM-H: a ready-made L1-versus-L2 slide), §5.2 with Fig. 3, §5.3 with Fig. 4 (gains building on each other across runs), §6 (safety).
- Caveat: The model is still frozen and every domain has an automatic score. Headline gains start from weak initial agents.
B18. Research acceleration: The view inside OpenAI — OpenAI, Sep 2026 — post · Core · NEW
- Why: The most detailed lab-internal data yet on AI speeding up AI research. OpenAI says it has reached its "automated research intern" goal: agents that carry out well-defined, multi-day research tasks under human direction. By mid-August its research organization used 3.1 agent-workdays for every human workday. Its next target is an automated AI researcher by March 2028.
- Read: The sections on how many agents each researcher runs, experiments per researcher, and success rates by task length (including how often humans had to step in); the section on how safety restrictions paced model development; the methods appendix.
- Caveat: Self-reported activity measures (tokens, code, experiments), confounded by growth in compute and with no comparison group. OpenAI itself says they do not directly measure research progress. Automated fetches of the page return HTTP 403, so open it in a browser.
B19. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity — Becker et al. (METR), 2025 — arXiv:2507.09089 · Core
- Why: The essential counterpoint to B18. In a randomized trial (16 developers, 246 tasks), AI made experienced developers 19% slower, while they believed it had made them about 20% faster. Self-reported speedups need care.
- Read: Fig. 1 (forecast against actual), Fig. 2, §3.3 with Table 1, Table 2 (what the study does not show).
- Caveat: Measures 2025 tools on individual tasks, not research progress. METR's 2026 follow-up leans toward a speedup but calls its own signal unreliable, partly because developers refused to work without AI. Don't present the 2025 slowdown as today's picture.
- Pair with: RE-Bench (Wijk et al., 2024): agents beat human experts on AI research tasks given 2 hours; the experts win at 8–32 hours.
B20. Lab thresholds for AI that speeds up AI research — Anthropic, OpenAI and Google DeepMind, 2025–2026 · Deeper · NEW
-
Why: The yardsticks to hold "recursive self-improvement" claims against.
- Anthropic: under its Responsible Scaling Policy (v3.4, July 2026), the threshold is met if models could replace its entire research staff at competitive cost, or if Anthropic observes or expects the rate of AI progress to double. Its August 2026 Risk Report found "meaningful acceleration starting in early-to-mid 2025, though by less than a factor of 2", which is below the threshold. A staff survey in the same report gave a self-reported speedup of about 4×, with 1 of 18 respondents saying the model could already replace an entry-level researcher.
- OpenAI: in its Preparedness Framework v2, "High" means impact equal to giving every OpenAI researcher a strong mid-career research engineer. The GPT-6 Astra system card (Sep 2026) tests internal research debugging, kernel writing, NanoGPT speedups, post-training recipes and a revised MLE-bench, and concludes "Astra does not reach our High threshold".
- Google DeepMind: Frontier Safety Framework 3.1, §3.2.2, defines two ML research levels: progress substantially faster than historical rates, and fully automating a team of AI researchers at comparable cost.
- Check before using: Anthropic's IPO filing S-1/A No. 2 (10 Nov 2026) reportedly says Anthropic has determined that it has met this threshold, based on what it expects from future models rather than what it has observed. The research pass read this in the filing, but I could not re-open it (the SEC blocks automated reads), and only one secondary source repeats it. Amendment No. 1 (2 Nov), reported by several outlets, said Anthropic "may imminently meet" the threshold. Open No. 2 in a browser and search for "threshold" before this goes on a slide.
- Caveat: Each lab defines and grades its own thresholds, and evaluation suites are replaced as they saturate.
B21. Toward an O*NET for AI R&D — Denain, Kwon & Ho (Epoch AI), Jun 2026 — post · Optional
- Why: A shared vocabulary for which parts of AI research are being automated: more than 60 tasks, rated and grouped into six phases, from deciding what to work on to communicating results.
- Caveat: A draft with subjective ratings, presented as the authors' views rather than Epoch's. A taxonomy, not a measurement.
"LLM looping": two different ideas
The phrase covers two unrelated things. Looped models repeat part of the network so the model can think longer without writing more text. Agent loops rerun an agent, check its work and retry, keeping state in files. The Navier–Stokes and ETH runs, as publicly described, are agent loops; no primary source links them to looped architectures.
Looped models: thinking longer inside the network
B22. Universal Transformers — Dehghani et al., 2018 — arXiv:1807.03819 · Optional
- Why: The original idea: apply the same block repeatedly and let the model decide when to stop.
- Read: Figs. 1–2, §2.2 (dynamic halting), Fig. 3.
- Caveat: Small tasks. Sharing weights across depth never became the standard recipe for language models.
- Pair with: Looped Transformers as Programmable Computers (Giannou et al., 2023): a fixed 13-layer transformer run in a loop can execute general programs. The weights are set by hand, so it says nothing about what training finds.
B23. Training Large Language Models to Reason in a Continuous Latent Space (Coconut) — Hao et al., 2024 (v4 Aug 2026) — arXiv:2412.06769 · Deeper
- Why: Feeds the model's last hidden state back in as its next input, so it reasons without words. These "continuous thoughts" can keep several possible next steps open at once, much like breadth-first search.
- Read: Fig. 1, §3; in v4, §4.3–4.4 with Figs. 4–6.
- Caveat: It beats chain of thought on a synthetic planning task (97.0% against 77.5%) but trails it on GSM8K (34.1% against 42.9%), with GPT-2-size models.
B24. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach — Geiping et al., 2025 — arXiv:2502.05171 · Deeper
- Why: The first sizeable pretrained model with a repeatable core block (3.5B parameters, 800B tokens). Running the block more times improves reasoning, up to compute equivalent to about 50B parameters: thinking longer without writing more.
- Read: Figs. 1–2, §5.3, §6, §7.
- Caveat: One proof-of-concept training run, far below frontier models.
- Pair with: Ouro (Zhu et al., 2025, v6 Oct 2026): looping built into pretraining for 1.4B and 2.6B models on 7.7T tokens. The authors report results matching models of up to 12B, which they attribute to using knowledge better rather than storing more.
B25. Hierarchical Reasoning Model (HRM) — Wang et al., 2025 — arXiv:2506.21734 · Deeper
- Why: A 27M-parameter recurrent model trained on about 1,000 examples that does well on hard Sudoku, mazes and ARC puzzles. It is the poster child for "iteration beats size", and a good example of a result being re-explained.
- Read: HRM Figs. 1–3 and 5, then ARC Prize's analysis (a plain transformer of the same size comes within about 5 points, and the outer refinement loop drives the gains) and Ren & Liu (2026: the model gets stuck on wrong answers, and making more guesses lifts hard-Sudoku accuracy from 54.5% to 96.9%).
- Caveat: Not a language model, and the loop mattered more than the architecture.
- Pair with: TRM: Less is More (Jolicoeur-Martineau, 2025), Fig. 3 (pseudocode that fits on a slide): a 2-layer, 7M-parameter network that repeatedly refines its answer scores 45% on ARC-AGI-1 and 8% on ARC-AGI-2.
B26. 2026 follow-ups on looped models · Optional · NEW
- STARS (Yang et al., May 2026): looped models often peak and then collapse as loops are added. Training them to settle into stable states restores the benefit.
- Equilibrium Reasoners (Huang, Geng & Kolter, May 2026): treats solutions as stable states the model settles into. Accuracy on very hard Sudoku rises from 2.6% without looping to over 99%.
- Allocating Recurrent Compute in Looped Language Models (Lin et al., Aug 2026): looping only the part of the block that mixes information across tokens keeps much of the gain at lower cost.
- Why: The field is moving from "loop more" to "loop stably and selectively".
- Caveat: Small models, puzzles and arithmetic, with no evidence yet at frontier scale.
Agent loops: run, check, retry
This is the mechanism behind multi-day runs. Each pass starts with a fresh context, state lives in files or git, and an outside check (tests, Lean, or independent checking agents) decides what to retry. A7 (Large Language Monkeys) gives the quantitative case for retrying against a checker.
B27. Ralph Wiggum as a "software engineer" — Huntley, Jul 2025 — post · Deeper
- Why: The original "Ralph" loop. A bash loop feeds the same prompt file to a coding agent again and again, one plan item per pass; tests push back on bad changes, and state lives in the repository rather than in the model's context. It is the minimal version of what B11 does at scale.
- Read: The sections on one item per loop, on push-back from tests, and on waking up to a broken codebase; the ralph-loop plugin README.
- Caveat: Anecdotal, with no controlled evaluation. The plugin loops inside one session, whereas Huntley's loop restarts with a fresh context each time.
- Pair with: Effective harnesses for long-running agents (Young, Anthropic, Nov 2025): a setup agent writes a feature list and a progress file, then a coding agent works through it one feature at a time.
B28. Hilbert: Recursively Building Formal Proofs with Informal Reasoning — Varambally et al., 2025 — arXiv:2509.22819 · Deeper
- Why: Check and retry with Lean as the judge. A reasoning model, a Lean prover, the Lean checker and a theorem search tool work together; when a goal fails, Hilbert splits it into subgoals and repairs proofs using the checker's feedback. It scores 99.2% on miniF2F and 70% on PutnamBench (462 of 660).
- Read: Figs. 1–3, Tables 1–2, Table 6 (failure modes).
- Caveat: Competition-level benchmarks, not research mathematics.
- Pair with: Gauss (Math Inc., 2025): thousands of concurrent agents, each with its own Lean process, formalized the strong Prime Number Theorem in about 25k lines of Lean in about three weeks, following a human-written plan. The closest public analogue to formalizing a result in Lean afterwards, at scale.
B29. math-proof plugin (solo and siege modes) — Anthropic, Sep 2026 — GitHub · Optional · NEW
- Why: A runnable miniature of the judge-and-workers pattern for research-level problems. Each round, a judge writes self-contained questions, fresh workers answer them in parallel, and the judge keeps a ledger of what is proved, refuted and still open. Any complete proof is checked line by line by two more workers before the judge accepts it.
- Read: The README (the run diagram, the Status section of the final
proof.md, the cost warnings),skills/siege/SKILL.mdand theagents/folder. - Caveat: Checking is done by other model instances, not by Lean, and the README warns that the output is "the model's own account", with citations quoted from memory. A siege runs for hours, with dozens to over a hundred sub-agent runs.
B30. Lean's axioms and #print axioms — Lean documentation — Theorem Proving in Lean 4, "Axioms and Computation" · Lean reference, "Axioms" · Deeper
-
Why: What "checked by Lean" rests on. On top of dependent type theory, Lean's standard library adds three axioms:
-
propositional extensionality (
propext); - a quotient axiom (
Quot.sound); - a choice principle (
Classical.choice).#print axiomslists every axiom a declaration uses, directly or indirectly.sorryis implemented by the axiomsorryAx, so an unfinished proof shows up there.
-
propositional extensionality (
- Read: Theorem Proving in Lean 4, ch. 12; the reference manual's Axioms chapter.
- Caveat: The kernel checks a proof against the statement as written. Whether the statement says what was meant is a human question (see C4).
Demo kit
Checked 2026-11-16. Rehearse each demo and keep a recording as a backup.
| Demo | What the audience sees | What it needs |
|---|---|---|
| A minimal agent (mini-swe-agent) | A roughly 100-line agent thinking, running a command, reading the result and fixing a failing test | Python, an API key, a throwaway container |
| An agent loop (ralph-loop) | The number of passing tests rising pass by pass in the git log | Claude Code, a sandbox, an iteration cap |
| Evolution live (OpenEvolve, ShinkaEvolve) | A program evolving toward a better algorithm; ShinkaEvolve's viewer shows the family tree and the score curve | Python and model API keys; a CPU is enough |
| Checking a published discovery (AlphaEvolve problem repository) | Colab cells scoring a published construction live | A browser; no API key |
| Judge and workers (math-proof) | A walk through a pre-recorded run: the ledger, the workers' answers, the two-checker step and the final Status | Claude Code 2.1.280 or later and a budget cap; runs take hours, so record in advance |
| Thinking longer inside the model (Huginn-0125, Ouro-1.4B) | The same question answered with 1, 4, 16 and 32 internal loops | A GPU |
| The capability clock (METR time horizons) | The log-scale chart, the gap between the 50% and 80% horizons, the 16-hour reliability note | A browser |
Not demo-friendly: the Darwin Gödel Machine and Hyperagents need Docker, benchmarks and hours of API time, so show their archive-tree figures instead.
If you only read five papers from Part B
- B7 AlphaEvolve: the mechanism. Many proposals, automatic checks and an evolving population, at scale.
- B8 Georgiev, Gómez-Serrano, Tao & Wagner: what that mechanism does on 67 real problems, including gamed checkers and a discover, prove and formalize pipeline.
- B11 Carlini: many agents working for weeks without an orchestrator, and why the quality of the checker is the binding constraint.
- B4 METR time horizons: the trend that makes multi-day runs plausible, and where the measurement stops being reliable.
- B18 OpenAI's research acceleration post, read with B19 and B20: what "AI speeding up AI research" means in 2026 numbers, and what those numbers do not show.
Then B17 (Hyperagents) and B12 (the Station).
Part C: Timeline and case files
Checked 2026-11-16. The claims in C2–C4 are under expert review, so re-check their status in the week before the talk.
Timeline, 2016–2026
| Date | What happened | Loop | Entry |
|---|---|---|---|
| 2016-03 | AlphaGo beats Lee Sedol | 2 | A1 (pair) |
| 2017-10 | AlphaGo Zero learns from no human games | 2 | A1 |
| 2022-01 | Chain-of-thought prompting | 1 | A6 |
| 2022-10 | ReAct: think, act, observe | 3 | B1 |
| 2023-10 | SWE-bench: real GitHub issues as a test | 3 | B3 |
| 2024-07 | AlphaProof reaches IMO silver level, checked in Lean | 2 | A34 |
| 2024-09 | OpenAI o1 thinks before it answers | 1 | A9 |
| 2025-01 | DeepSeek-R1: RL with verifiable rewards, in the open | 2 | A11 |
| 2025-05 | AlphaEvolve; the Darwin Gödel Machine | 3, 4 | B7, B16 |
| 2025-07 | IMO 2025 gold, in natural language | 1 | C1 |
| 2026-02 | 16 agents build a C compiler | 3 | B11 |
| 2026-09 | OpenAI reports 3.1 agent-workdays per human workday | 4 | B18 |
| 2026-09-08 | OpenAI's Navier–Stokes claim | claim | C2 |
| 2026-10-06/07 | OpenAI's math release; three manuscripts withdrawn | claim | C3 |
| 2026-10-26 | OpenAI's and Anthropic's ETH refutation claims | claim | C4 |
Case files
C1. IMO 2025 gold, in natural language — Google DeepMind and OpenAI, July 2025 — Axios · Simon Willison’s notes · Deeper
-
Why:
- Both labs' experimental models scored 35/42, gold-medal level.
- Each solved problems 1–5 with natural-language proofs inside the 4.5-hour limit.
- Gemini Deep Think's result was certified by the IMO's official graders (announced 21 July). OpenAI's was graded by three former medalists (19 July).
- Read: Both pieces, for the difference between official and self-reported grading.
- Caveat: Olympiad problems are hard but well-posed and have known solutions; that is not research.
C2. OpenAI's Navier–Stokes claim — OpenAI, 2026-09-08 — paper (PDF) · Lean repository · Clay's statement, 11 Sept 2026 · TechSpot · YourStory on the dispute · Core
-
What was claimed:
-
Finite-time blow-up for the 3D Navier–Stokes equations under a carefully chosen external force. The 166-page paper's Theorem 1.1 states the following for every viscosity ν > 0:
- there is a force f ∈ C_c^∞(ℝ³ × (0, ∞)) with zero initial velocity;
- its smooth solution on [0, 1) keeps the L² norm bounded;
- the L^∞ norm blows up as t → 1.
- The paper says this establishes alternative (C) of Fefferman's Clay statement (C7), where such a force is allowed, and (D) on the torus via Corollary 10.6. If correct, it settles the Clay problem as posed. The unforced regularity question, statements (A) and (B), stays open; many mathematicians consider it the deeper one.
- The Lean repository (Lean 4 with Mathlib) also claims an unforced singularity for the Euler equations. Its README says nothing about axioms or statement fidelity.
-
Finite-time blow-up for the 3D Navier–Stokes equations under a carefully chosen external force. The 166-page paper's Theorem 1.1 states the following for every viscosity ν > 0:
- How it was produced:
- Press reports (TechSpot and others) say an unreleased internal model ran about 10,000 agents for 88 hours. They cite OpenAI's announcement, which could not be opened here because it blocks automated reads. Some reports call the 10,000 figure an estimate.
- The paper itself has no methods section.
- Status:
- Clay's 11 September statement says the problem "has apparently been settled" and calls the review "deliberately unhurried". It names neither OpenAI nor the method, and the Millennium page still lists the problem as active.
- OpenAI said it will not seek the prize.
- Experts still have to confirm the argument, and that the Lean statement matches Clay's.
- There is a credit dispute with related forced-equation work by Tristan Buckmaster and Levent Alpöge, who used several AI models including OpenAI's.
- Two preprints bear on it:
- Bastounis, Circelli & Hansen (6 Oct) find that the Lean proof does not follow the paper's argument at intermediate steps (NPR: in one equation a 4 became a 5). They do not claim the theorem is false (see C11).
- Constantin, Ignatova & Vicol argue that constructions with the paper's two structural features cannot blow up without a force (see C12).
- Press: Quanta and Science News report about 10,000 agents running at once on an internal model, a cost of "millions of dollars", and a separate Lean-formalization run. Scientific American covers the forcing debate.
- Caveat: No independent verification had been reported by mid-November.
C3. OpenAI's math release: 721 manuscripts — OpenAI, October 2026 — github.com/openai/math · history.md · Core
-
Why: The clearest picture of what generating results at scale looks like:
- about 4,000 problems posed;
- 721 manuscripts in 373 families;
- about three hours of ChatGPT Pro thinking per result;
- about 42% of top-line results formalized in Lean (300 of 719 on 7 October).
- The withdrawal:
- On 2026-10-07, three manuscripts were withdrawn because "a sign error invalidates a stabilization-trace cancellation argument and the construction used by two dependent papers."
- Fourteen others were corrected.
- Read: The README, then history.md from the bottom up.
- Caveat: The repository doesn't state its first release date; 2026-10-06 comes from earlier research for this list.
C4. The Exponential Time Hypothesis refutation claims — OpenAI and Anthropic, 2026-10-26 — OpenAI's entry in history.md · Anthropic's README (formal-math/eth) · HackerNoon overview · OfficeChai on Anthropic’s paper · Core
-
What was claimed: Each lab claims an algorithm for 3-SAT that runs faster than exponential time, which would refute ETH (and SETH). Both proofs are checked in Lean.
- OpenAI: "The Exponential Time Hypothesis is False", added to openai/math on 26 October. It claims a deterministic algorithm that exactly counts the satisfying assignments of fixed-width CNF formulas in subexponential time and polynomial space; the 3-SAT case refutes ETH.
- Anthropic: "A 2^o(n) Time Algorithm for 3-SAT". The README claims a deterministic algorithm counting 3-SAT solutions in time 2^O(n log log n / √log n), "discovered by an internal research Claude model".
- The Lean files use only the axioms propext, Classical.choice and Quot.sound (Lean v4.33.1).
- A check script runs the Comparator tool and a second kernel (nanoda).
- The README warns that no machine checks that the Lean file says what the paper says.
- Secondary coverage reports 49 pages and about 14 million output tokens.
- Status:
- Russell Impagliazzo, Ramamohan Paturi and Ryan Williams have opened a community review (Message to the TCS Community on ETH). They say their involvement is not a certification and propose a virtual meeting before the end of 2026. OpenAI says the materials were made public at the group's request.
- OpenAI's Lean documentation says the Lean file proves a slightly weaker bound than the paper, 2^O(n (log log n)² / √log n). That is still subexponential.
- Dates: OpenAI's manuscript was posted 26 October. Anthropic's files are dated 27 October, and it reports the core algorithm was found on 16 October.
- No independent verdict had been found by 2026-11-16.
- The algorithm's practical running time may be astronomically large (see C8).
- Caveat: Anthropic's README is undated and names no reviewers. The two claims appeared a day apart, not on the same day as some reports say.
C5. The four colour theorem: proof by computer — Appel & Haken, 1976; Gonthier, 2004–05 — Gonthier's formal proof (PDF) · Deeper
- Why: The first famous theorem whose proof rested on computer checks no person could do by hand: an unavoidable set of 1,936 (later 1,482) reducible configurations. Mathematicians argued about whether it counted. Gonthier's Coq formalization removed doubt about the case analysis.
- Caveat: The formal proof still trusts Coq and the hardware and software it runs on.
C6. The Kepler conjecture and Flyspeck — Hales, 1998–2017 — Nature news, 2003 · Flyspeck, Forum of Mathematics, Pi 5:e2 (2017) · Deeper
-
Why: Hales's computer-assisted proof was too long to check fully.
- After years of work, the referees, led by Gábor Fejes Tóth, said they were "99 percent certain"; the Annals editors said the referees could not certify the proof.
- Hales answered with Flyspeck, a formal proof in HOL Light and Isabelle, announced complete on 10 August 2014.
- Caveat: It took about a decade to formalize one theorem; that is what formal checking used to cost.
C7. The official Navier–Stokes problem statement — Fefferman, Clay Mathematics Institute — PDF · Core
-
Why: The Millennium Problem is settled by proving any one of four statements:
- (A) smooth solutions for all time on ℝ³;
- (B) the same on the torus;
- (C) a breakdown on ℝ³;
- (D) a breakdown on the torus. The two breakdown statements allow a smooth external force with the stated decay (periodic for D). That is why a forced blow-up claim targets the problem as posed.
- Read: The statements (A)–(D) and the smoothness, decay and finite-energy conditions they reference.
- Caveat: Many analysts regard the unforced question, whether smooth data with no force can blow up, as the deeper one.
C8. The Exponential Time Hypothesis: definitions and the previous best bound — Impagliazzo & Paturi, 2001 — J. Comput. Syst. Sci. 62(2) · Hansen, Kaplan, Zamir & Zwick, 2019 — STOC 2019 · Deeper
- Why: Defines s_k as the infimum of δ such that k-SAT on n variables is solvable in time 2^(δn), and states ETH as s_3 > 0. SETH, the strong form, says s_k → 1. Before 2026 the best randomized 3-SAT algorithms (the PPSZ family; biased-PPSZ, Hansen et al.) ran in about 1.307^n.
- Read: Impagliazzo & Paturi, §1–2; Hansen et al., the abstract and the results table.
- Caveat: Refuting ETH needs only time 2^(o(n)), which is still superpolynomial, so it says nothing about P versus NP. A bound whose hidden constants are unknown may never be practical.
Balance reading: what mathematicians are saying
C9. A severe misalignment of AI in mathematics — 25 Fields medallists, posted by Tao, 11 Sept 2026 — post · Core
- Why: The strongest statement of the talk's takeaway from inside mathematics. Solving problems is only a proxy for understanding, and announcement-driven problem solving can harm the field.
- Caveat: A position statement. It does not discuss Navier–Stokes specifically, and not every leading mathematician signed it.
C10. The technical debt of AI-generated mathematics — Henry Cohn, 15 Sept 2026 — post · Deeper
- Why: Hard-to-read AI proofs shift the cost of understanding them onto the community.
- Caveat: An essay; written before the October release.
C11. The Lean proof versus the paper — Bastounis, Circelli & Hansen, Oct 2026 — arXiv:2610.08144 · NPR Short Wave, 4 Nov 2026 · Core
- Why: A passing Lean check of an AI autoformalization need not certify the natural-language proof. In the Navier–Stokes case the Lean proof differs from the paper's argument at intermediate steps; NPR describes an equation in which a 4 became a 5.
- Caveat: It does not claim the theorem is false. It is about what the check certifies.
C12. Limits of the forcing method — Constantin, Ignatova & Vicol, Sept 2026 — arXiv:2609.20803 · Deeper
- Why: Constructions with the two structural features of OpenAI's construction cannot blow up under real-analytic forcing, including no force. This is why the unforced question stays open even if the forced claim holds.
- Caveat: Conditional on those features; it does not say OpenAI's paper is wrong.
Could not verify
These were left out of the main list because no primary source could be opened:
- A Science version of AlphaProof Nexus (reported 8 October 2026), found only in secondary coverage; A39 cites the arXiv preprint instead.
- The DeepSeek-V4 technical report (April 2026). Secondary coverage says it replaced mixed RL with on-policy distillation from RL-trained experts.
- Kimi K3, Qwen 3.8 and GLM-5.2 technical reports (mid-2026): only secondary comparison posts, with no primary RL details.
- The AxiomProver technical report, announced as forthcoming in January 2026.
- NVIDIA Nemotron 3 Ultra Technical Report (June 2026): the PDF link works but was not read. It is the base model behind A26.
- How OpenAI organized the Navier–Stokes run. Reports describe about 10,000 agents working for 88 hours and sharing insights, but OpenAI's PDF has no methods section, and no OpenAI page describing the setup was found. C2 gives what has been reported.
- How Anthropic's ETH result was produced. Reports say an internal Claude research model used about 14M output tokens; only secondary coverage was found.
- The exact wording of Anthropic's S-1/A No. 2 on its AI R&D threshold (see B20).
- A claim that METR could not measure GPT-5.6 Sol's time horizon because the model cheated. It appeared only in a search-engine summary, not on METR's pages.