ReasoningPaperAdvanced2025~15 min theory

Inference-Time Scaling for Generalist Reward Modeling

A reward model can think too. Train a 27B model to write principles and critiques, sample it many times, and vote, and it beats reward models hundreds of times larger.

arXiv:2504.02495

Where this sits

A research satellite on the reward side of the pipeline. R1 showed that verifiable rewards can drive reasoning; this paper asks how to get reliable rewards for the general case, which is the open problem R1's own limitations section leaves behind. Its 'from' edge is left out because it was not built on a specific earlier curriculum paper.

Proposes pointwise generative reward modeling, where the model writes principles and critiques and then scores, instead of acting as a fixed classifier or judge prompt.
Introduces SPCT (Self-Principled Critique Tuning), an RFT (rejective fine-tuning) plus rule-based online RL schedule that makes reward generation scale with inference compute.
Shows parallel sampling with voting improves the same 27B model monotonically: RewardBench 86.0 at one sample, 87.7 at 8, 88.5 at 32, and 90.4 with meta-RM filtering.
Adds a meta reward model that predicts whether a judgment is correct and filters bad judgments before voting, improving scaling beyond plain voting.
Demonstrates that inference-time scaling of a 27B reward model beats training-time scaling to 671B, closing a gap that used to require the bigger model.

What it is

This paper is about the part of RL that R1 deliberately sidestepped. Verifiable tasks can be graded by rules, but most useful behavior cannot. The authors build a generative reward model (GRM) that writes principles for judging a query, critiques each candidate response against those principles, and outputs a score. Training (Self-Principled Critique Tuning) combines rejection-sampled fine-tuning with rule-based online RL, and inference scales by sampling many judgments in parallel and aggregating them with a vote guided by a meta reward model.

1Theory from first principles

Why reward models matter, and why they are hard

RL post-training needs a number for every response the policy produces. When the task is verifiable, a rule supplies that number exactly, as in R1's math and code rewards. For writing, dialogue, or open-ended advice, there is no rule, so a reward model must stand in for human judgment. Classical reward models are discriminative: they read a prompt and a response and output a scalar, which makes them cheap but rigid, since one forward pass must encode every criterion at once. This paper instead treats reward modeling as a generative task, which is what makes it scale with more compute at inference time.

Generative reward modeling: principles, then critiques

The pointwise GRM takes a query and one response, proposes the principles it will judge by (for example, correctness, safety, relevance, instruction following), writes a critique against each principle, and only then emits a scalar reward. Judging is now an explicit reasoning process, and like any reasoning process it can be sampled repeatedly. Two properties follow: the model adapts its own criteria per query instead of applying one fixed rubric, and the quality of the judgment can improve simply by generating more attempts, which is the same inference-time scaling idea as letting a reasoner think longer.

SPCT: how the model learns to judge

SPCT has two stages. In the first, rejective fine-tuning, the model samples judgments on reward-model (RM) data and keeps the ones that agree with the ground-truth preference, with an optional hint about the correct label attached to some samples. In the second, rule-based online RL sharpens the behavior: reward is computed by comparing the model-generated reward with the known correct ordering, so the training signal stays exact and needs no learned critic of its own. The resulting DeepSeek-GRM-27B is based on Gemma-2-27B. The paper also trains a meta RM, a small classifier over (query, response, judgment) that predicts whether a sampled judgment is correct, and this is what guides voting at inference.

Compute comparisons. Voting over 32 samples from the 27B generative reward model reaches the overall level of much larger reward models, with meta-RM filtering on top; the paper reports this inference-side scaling beats size scaling up to 671B parameters.

Plurality voting over sampled judgments

V(c) = sum_{j=1..N} 1[ c = argmax_{c'} r_j(c') ]

Each sampled judgment j assigns a score r_j to every candidate response c'. The first term inside the indicator asks which candidate that sample prefers. Summing over N samples counts how many judgments rank each candidate first, and the aggregate reward picks the candidate with the largest V(c). This is inference-time scaling in its cheapest form: more samples, less variance, no retraining. The meta RM changes the aggregation by weighting or filtering samples before the vote, which is why 8 guided samples can match many unguided ones.

Tallying votes from sampled judgments

python

def plurality_vote(judgments):
    # judgments: list of samples; each sample is a dict candidate -> score
    tally = {}
    for sample in judgments:
        best = max(sample, key=sample.get)
        tally[best] = tally.get(best, 0) + 1
    return max(tally, key=tally.get)

samples = [
    {"A": 0.8, "B": 0.2},
    {"A": 0.6, "B": 0.7},
    {"A": 0.9, "B": 0.1},
]
print(plurality_vote(samples))  # A wins two of three judgments

Ties resolve by dictionary order here; production code should break them explicitly.
Every sample is one full generative judgment, so this is where inference cost grows with N.
The meta RM can pre-filter samples before this function runs.

What the numbers show

On RewardBench the 27B model scores 86.0 with a single greedy judgment, 87.7 with 8 samples, 88.5 with 32, and 90.4 when meta-RM filtering guides the vote. The paper's overall average across its RM benchmark suite moves from 69.9 to 70.6 (8 samples) to 71.0 (32 samples) to 72.8 with the meta RM, against 71.3 for GPT-4o and 70.5 for Nemotron-4-340B-Reward, which the paper cites as public reference points. Ablations show principle generation is load-bearing: removing it drops the overall score to 67.5, and removing the rejective sampling or general instruction data hurts as well. Inference-time scaling also outperforms simply training a bigger GRM, the paper's central comparison.

The honest limits

The paper admits that DeepSeek-GRM still struggles on some tasks and does not eliminate bias, since a generative judge inherits the biases of its training data. The meta RM is another learned component with its own failure modes, and sampling 32 judgments per evaluation multiplies inference cost, which matters whenever the reward model sits inside an RL loop rather than an offline eval. Voting assumes the majority of samples are right, which can fail systematically on questions where the model's priors are wrong. The authors frame these as open problems for generalist reward systems.

2Inside the paper

Problem the paper sets out to solve

RL post-training beyond verifiable tasks needs accurate reward signals for arbitrary queries, and classical scalar reward models neither adapt their criteria nor benefit much from extra inference compute.

Key idea to look for while reading

Make reward modeling generative and the rest follows: principles and critiques can be sampled, samples can be voted, and a meta model can filter. Section 3 defines pointwise GRM and SPCT, Section 4 covers the inference-time scaling strategies including voting and meta-RM guidance.

Evidence the paper musters

DeepSeek-GRM-27B on the paper's suite: RewardBench 86.0 to 90.4 as samples and filtering scale from one to 32; overall average 69.9 to 72.8. Baseline comparisons include LLM-as-a-Judge at 67.8, CLoud-Gemma-2-27B at 68.7, DeepSeek-PairRM-27B at 69.0, and public reference results GPT-4o 71.3 and Nemotron-4-340B-Reward 70.5. Ablations isolate principle generation (67.5 without it) and rejective sampling (68.7 without it) as contributors.

Limits the authors admit

Residual task failures are reported and attributed to data difficulty and synthesis quality; biases are reduced but not removed. The meta RM adds a learned filter, not a guarantee. Scaling by parallel sampling has real serving cost, and the paper does not claim the approach covers every subjective evaluation.

Practitioner takeaway

If you need a reward signal for a non-verifiable task, consider a generative judge that states its criteria, then spend inference compute sampling and aggregating rather than immediately reaching for a larger model. A small meta model that filters judgments is a cheap upgrade to plain voting. And when you evaluate a reward model, always report the number of samples behind each score, because in this design the sample count is part of the model.

3Implementation check

0 of 5 questions answered

1.Why does voting help a generative reward model more than a scalar one?
2.What is the job of the meta reward model?
3.In SPCT's online RL stage, what serves as the reward for the reward model?
4.The pointwise GRM scores candidates one at a time rather than comparing pairs. What does that buy?
5.The paper claims inference-time scaling beats training-time scaling for this task. What is the practical reading?

4Build it yourself

Plurality Voting and DUCB Budget Allocation

Intermediate3-4 hours

Give five candidates hidden scores, judge pairs with Gaussian noise, and aggregate wins into a vote. Compare spending a fixed comparison budget uniformly against a discounted-UCB rule that keeps matching the leaders.

Milestones

Give each candidate a hidden score and write the noisy pairwise judge.
Implement vote as the candidate with the most match wins, with ties broken deterministically.
Run uniform allocation at budgets 2 through 64 and average accuracy over 60 seeds.
Implement the adaptive branch: leader by win rate, challenger by win rate plus an exploration bonus.
Print both compute and accuracy curves side by side.
Raise the noise and find the point where adaptive allocation stops helping.

Starter code

python · editable

import math
import random

CANDIDATES, SIGMA, SEEDS = 5, 0.22, 60
BUDGETS = [2, 4, 8, 16, 32, 64]

def true_scores(seed):
    rng = random.Random(seed)
    return [rng.uniform(0.0, 1.0) for _ in range(CANDIDATES)]

def judge(i, j, scores, rng):
    return 1 if scores[i] + rng.gauss(0, SIGMA) > scores[j] + rng.gauss(0, SIGMA) else 0

def vote(wins):
    # TODO: return the candidate with the most wins from the pairwise matches.
    return -1

def uniform_accuracy(budget, seed):
    rng = random.Random(seed)
    scores = true_scores(seed)
    wins = [0] * CANDIDATES
    for _ in range(budget):
        i, j = rng.sample(range(CANDIDATES), 2)
        if judge(i, j, scores, rng):
            wins[i] += 1
        else:
            wins[j] += 1
    best = max(range(CANDIDATES), key=lambda c: scores[c])
    return 1 if vote(wins) == best else 0

def ducb_accuracy(budget, seed):
    # TODO: same budget, but pick every next pair adaptively: the leader is
    # the best win rate, the challenger is the best win rate plus an
    # exploration bonus (discounted UCB), then score the pair with judge().
    return 0

def main():
    print("budget  uniform  ducb")
    for budget in BUDGETS:
        u = sum(uniform_accuracy(budget, s) for s in range(SEEDS)) / SEEDS
        d = sum(ducb_accuracy(budget, s) for s in range(SEEDS)) / SEEDS
        print(budget, " ", round(u, 3), " ", round(d, 3))
    # EXPECTED: after the TODOs, both curves rise and ducb leads (about
    # 0.83 vs 0.58 at 64 comparisons); the unfilled starter prints 0.0.
main()

Runs locally in your browser.

Success criteria

Both accuracy curves increase with budget across the six budgets.
DUCB accuracy at budget 64 is at least 0.1 above uniform and never lower at any budget.
Reducing the judge noise to 0.05 pushes both methods above 0.9, showing the gap is about judgement noise, not the voting rule.

Stretch goals

Resample every pair twice and drop judgments that disagree before the vote, a small stand-in for the meta reward model.
Add a sixth candidate nearly tied for first and see which method identifies it.
Replace plurality with a Bradley-Terry fit and compare at a fixed budget.

Problems

5Keep going

previousCodeI/O: Condensing Reasoning Patterns via Code Input-Output Prediction nextDeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decomposition

Problems

Articles

Post-Training: RLHF → DPO → GRPO

Where this sits

What it is

1·Theory from first principles

Why reward models matter, and why they are hard

Generative reward modeling: principles, then critiques

SPCT: how the model learns to judge

What the numbers show

The honest limits

2·Inside the paper

Problem the paper sets out to solve

Key idea to look for while reading

Evidence the paper musters

Limits the authors admit

Practitioner takeaway

3·Implementation check

4·Build it yourself

Milestones

Success criteria

Stretch goals

Problems

5·Keep going

Problems

Articles

1Theory from first principles

2Inside the paper

3Implementation check

4Build it yourself

5Keep going