OpenSIR

Open-Ended Self-Improving Reasoner

Wai-Chung Kwan¹ Joshua Ong Jun Leang^1,2 Pavlos Vougiouklis³ Jeff Z. Pan¹ Marco Valentino⁴ Pasquale Minervini^1,5

¹University of Edinburgh · ²Imperial College London · ³Huawei Technologies R&D (UK) · ⁴University of Sheffield · ⁵Miniml.AI

Paper Code

OpenSIR overview: one policy alternates teacher and student roles, generates problems from a pool, samples solutions, scores novelty and correctness, and updates through GRPO.

OpenSIR trains a single policy to play two roles. As a teacher, it proposes novel, learnable problems near the model's current frontier from a growing problem pool. As a student, it solves them; majority voting supplies pseudo-reference answers. The teacher is rewarded for calibrated difficulty and semantic diversity, while the student is rewarded for correctness.

Main Results

The only self-play method that works for post-trained models: OpenSIR improves all four reported models across seven math benchmarks, averaging +3.35 points over base; prior self-play baselines stagnate or regress on reasoning models.
Beats annotated-data training: from a single seed problem, OpenSIR surpasses both GRPO baselines trained on 7,000+ human-annotated examples on every reported model in overall average.
Transfers beyond math: OpenSIR improves general reasoning by up to +4.79 points, while the best baseline reaches only +1.40 on instruction models and is negligible or negative on reasoning models.

Main Insights

Diversity sustains open-ended self-play: removing the diversity reward drops math from 29.57 to 26.84, general reasoning from 17.02 to 14.52, and concept coverage from 5,914 to 3,328.
Difficulty must be calibrated, not maximized: lowering s_min from 0.5 to 0.1 reduces validity from 70.82% to 42.31% and math average from 29.57 to 25.97.

One policy, two co-evolving roles

OpenSIR uses the same policy as both teacher and student. The teacher expands a shared problem pool with novel, learnable questions and concepts; the student solves sampled problems and turns majority-vote answers into pseudo-references. Starting from $1 + 1 = ?$, the two roles co-evolve into a self-generated curriculum spanning algebra, calculus, optimisation, probability, statistics, trigonometry, and other topics.

Teacher reward

Novelty = difficulty + complexity + diversity + format

\[ \operatorname{score}_{\mathrm{novel}}(q) = \alpha \operatorname{score}_{\mathrm{sol}}(q) + \lambda \operatorname{score}_{\mathrm{len}}(q) + \gamma \operatorname{score}_{\mathrm{div}}(q) + \delta \operatorname{score}_{\mathrm{Tfmt}}(q) \]

\[ \begin{aligned} \operatorname{score}_{\mathrm{novel}}(q) &= \alpha \operatorname{score}_{\mathrm{sol}}(q) \\ &\quad + \lambda \operatorname{score}_{\mathrm{len}}(q) \\ &\quad + \gamma \operatorname{score}_{\mathrm{div}}(q) \\ &\quad + \delta \operatorname{score}_{\mathrm{Tfmt}}(q) \end{aligned} \]

Solvability: a triangular score over $[s_{\min}, s_{\max}]$, peaking at $s_{\mathrm{mid}} = (s_{\min} + s_{\max}) / 2$. Problems outside the window receive zero.
Solution length: $\operatorname{score}_{\mathrm{len}}(q) = \min(\bar{l}(q) / l_{\mathrm{base}}, l_{\mathrm{cap}} / l_{\mathrm{base}})$, rewarding problems that elicit longer multi-step solutions.
Diversity: $\operatorname{score}_{\mathrm{div}}(q) = \min_{q'} d(e_q, e_{q'})$ over previous pool problems, pushing generation away from familiar concepts.
Format: generated problems must contain parseable <question> and <concepts> tags.

Student reward

Correctness = answer match + format

\[ \operatorname{score}_{\mathrm{correct}}(o) = \mathbf{1}[a = a^\star] + \delta \operatorname{score}_{\mathrm{Sfmt}}(o) \]

\[ \begin{aligned} \operatorname{score}_{\mathrm{correct}}(o) &= \mathbf{1}[a = a^\star] \\ &\quad + \delta \operatorname{score}_{\mathrm{Sfmt}}(o) \end{aligned} \]

The pseudo-reference $a^\star$ is the majority-vote answer across sampled solutions.

Correctness: reward is given when the parsed answer matches the pseudo-reference answer from sampled solutions.
Format: final answers must be parseable and presented in boxed-answer form.
Selection: high-novelty valid problems are used for student training, while high-variance problem groups keep teacher training diverse.

OpenSIR improves where prior self-play stalls

Across instruction-tuned and reasoning models, OpenSIR is the only self-play method here that consistently improves both math and general reasoning.

Model

View

Average benchmark scores

OpenSIR's aggregate gains separate math accuracy, general-reasoning transfer, and overall performance.

Analysis

The core analyses explain why OpenSIR keeps improving: diversity sustains exploration, solve-rate scoring targets learnable challenge, and teacher-student co-evolution keeps difficulty calibrated.

Diversity sustains open-ended learning

Removing the diversity reward narrows concept coverage and hurts both math and general reasoning, showing that exploration keeps self-play useful.

Impact of diversity reward

Diversity improves accuracy and nearly doubles concept coverage.

Topic distribution

OpenSIR broadens from arithmetic and algebra into calculus, number theory, probability, and trigonometry.

Problem-space evolution

The t-SNE view shows how the diversity reward changes the geometry of generated problems: with diversity, OpenSIR explores a much broader semantic region instead of staying near repeated templates.

Problem embedding spread

The plot is rendered from the underlying t-SNE point data, rather than embedding a static result image.

Difficulty must be calibrated, not maximized

The solvability reward peaks at a target solve rate of about 70%, favoring problems that are challenging but still yield reliable pseudo-reference answers. Lowering s_min from 0.5 to 0.1 makes the GPT-5 evaluator solve rate fall from 89.82% to 78.31%, but validity collapses from 70.82% to 42.31% and math accuracy drops from 29.57 to 25.97. This implies that harder generated problems are only useful when they remain reliably solvable enough to produce trustworthy pseudo-references.

Difficulty-validity trade-off

GPT-5 is used only as an external evaluator here: eight sampled solutions per problem estimate solve rate, while majority-answer agreement estimates validity.

Difficulty and invalid problems

Generated problems start too hard and often invalid, then become easier and cleaner before difficulty rises again as the solver improves.

Co-evolution keeps difficulty calibrated

Updating teacher and student together keeps generated problems aligned with the solver's improving ability.

Dual-role training

Co-training improves accuracy and keeps solve-rate variance much lower than student-only training.

Example problems

Here are examples from different training steps. Early questions are often underspecified or invalid; later generations become cleaner, more constrained, and span a wider range of reasoning topics.

invalidstep 0solve rate 0.25

A concert venue charges an admission price per seat and also offers a VIP ticket package that includes food, drinks, and other perks. If the food and other perks are included in the VIP ticket price and are worth $10 per person, and a group of friends want to buy the same number of VIP tickets as they would with regular tickets, what is the minimum admission price per regular ticket to make the total cost of the VIP tickets equal to or less than the total cost of the regular tickets?

Why invalid: the VIP ticket price is never given, so the answer is undetermined. Early training is full of these.

invalidstep 0solve rate 0.125

Consider two positive integers m and n (m >= n). Given a 2x2 matrix of numbers where each element is a non-negative integer, find the maximum value of ax^2 + by^2 + cxy, where a, b, c are constants, subject to the constraint that the sum of any two elements in each row and column of the matrix are equal. What is the maximum possible value of ax^2 + by^2 + cxy?

Why invalid: a, b, c are unspecified, m, n are unused, and the optimisation is not bounded.

validstep 124solve rate 0.375

A farmer has 10 different types of vegetables to store in a warehouse with 1,000 m² of floor space. Four container sizes are available (5/10/15/20 m², with caps of 50/40/30/25). Bulky vegetables need at least 15 m² containers; medium need at least 10 m²; small fit anywhere. Each vegetable type must be in at least one container. What is the maximum number of containers that can be used while satisfying all constraints and not exceeding 1,000 m² total?

Now valid: a bounded optimisation problem with explicit constraints.

validstep 156solve rate 0.375

Find the equation of the curve y = f(x) where the derivative is given by f'(x) = (3x^2 - x - 2) / 2x and the curve passes through the point (2, 3).

A clean calculus problem requiring antiderivatives and a boundary condition.

validstep 172solve rate 0.50

A golfer hits a ball from the top of a 50-meter high cliff with an initial velocity of 30 m/s at an angle of 45 degrees above the horizontal. What is the horizontal distance traveled by the ball when it hits the ground?

Projectile motion: trigonometry plus physics, outside the original arithmetic seed.

validstep 188solve rate 0.25

Consider a randomly ordered sequence of n = 3q distinct integers. Let f be the number of adjacent pairs sharing the same remainder mod 3. If the integers 1..3q are randomly permuted, what is the expected value of f?

A harder probability problem involving expectations over permutation statistics.

Cite OpenSIR

@misc{kwan2025opensir,
  title         = {OpenSIR: Open-Ended Self-Improving Reasoner},
  author        = {Kwan, Wai-Chung and Leang, Joshua Ong Jun
                   and Vougiouklis, Pavlos and Pan, Jeff Z.
                   and Valentino, Marco and Minervini, Pasquale},
  year          = {2025},
  eprint        = {2511.00602},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2511.00602}
}