FrontierAnnouncementCore2025~12 min theory

DeepSeek-V3.1 Release: Hybrid Inference for the Agent Era

One set of weights, two modes: Think when the task needs deliberation, Non-Think when latency matters, and the chat template decides which one runs.

Source

Where this sits

DeepSeek-V3 Technical ReportDeepSeek-V3.1 Release: Hybrid Inference for the Agent EraDeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse Attention

The reasoning era ended with two separate products: a fast instruct model and a slow but stronger reasoner. V3.1 is the moment DeepSeek folds the reasoning capability back into the V3 line as a mode of one model, and points the post-training budget at tool use, which is what the V3.2 and V4 generations build on.

Serves thinking and non-thinking behavior from a single set of weights, selected by the chat template rather than by routing to a different checkpoint.
Extends the V3 base model with 840B tokens of continued pretraining for long-context handling and ships a new tokenizer and chat template configuration.
Post-trains specifically for tool use and multi-step agent tasks, reporting SWE-bench Verified 66.0, SWE-bench Multilingual 54.5, and Terminal-bench 31.3.
Adds an Anthropic-format API endpoint and strict function calling (beta), lowering the integration cost for agent frameworks of the time.
Improves thinking efficiency: DeepSeek reports V3.1-Think reaches answers in less time than R1-0528 rather than by simply emitting more tokens.

What it is

DeepSeek-V3.1 is the August 21, 2025 upgrade that turned chat and reasoning into one checkpoint. It continues pretraining the V3 base model for 840B tokens to extend context handling, updates the tokenizer and chat template, and post-trains for tool use and multi-step agent tasks. The same weights serve both a fast non-thinking mode (the deepseek-chat API) and a deliberating thinking mode (the deepseek-reasoner API), each with 128K context. DeepSeek reports that V3.1-Think answers in less time than R1-0528, that SWE-bench Verified reaches 66.0, SWE-bench Multilingual 54.5, and Terminal-bench 31.3, and the release calls itself the first step toward the agent era. A follow-up patch, V3.1-Terminus (September 22, 2025), fixed Chinese-English mixing and further tuned the Code and Search agents.

1Theory from first principles

Two products became one

Before V3.1, a DeepSeek deployment had a fork in it. The instruct model answered directly and cheaply but was weak on multi-step problems; the R1-lineage reasoner wrote a long private chain of thought first, which raised quality on math and code but multiplied output cost and latency. Every request had to choose a checkpoint, and every team had to build routing logic on top. V3.1 collapses the fork: one base model, one set of weights, and a mode switch realized in how the prompt is rendered. The architectural claim is deliberately modest - V3.1 keeps the V3 architecture and size (671B total, 37B activated parameters) - and the release spends its budget on post-training and context instead.

The mode switch is a template, not a model

The two modes share weights; what changes is the chat template that serializes the conversation before the forward pass. In thinking mode the model emits reasoning between special tags before the final answer, and the API exposes it as deepseek-reasoner; in non-thinking mode the same weights answer directly, and the API exposes it as deepseek-chat. DeepSeek documents a practical detail for multi-turn use: the thinking-mode template drops the previous turn's reasoning tokens while retaining the tag in context, so a long conversation does not balloon just because the model once thought hard. This is the same trick a serving stack would otherwise implement with two model versions and an external router - here it is one artifact and a rendering choice.

Hybrid inference: one checkpoint, two prompt templates. Non-Think answers immediately; Think emits a reasoning trace between special tags, then the answer. The template, not a separate model, selects the path.

When each mode wins

Thinking is not free: reasoning tokens are output tokens, billed and serialized like the answer. So the honest decision rule is about the value of deliberation per token. Non-Think wins for short factual replies, extraction, classification, formatting, and high-QPS serving (many queries per second) where tail latency (p99, the 99th percentile) is the product. Think wins when the task has many dependent steps - math, code repair, multi-hop search, tool orchestration - because one correct final answer is worth many cheap wrong ones. DeepSeek's own framing in the release is efficiency: V3.1-Think is compared against R1-0528 on time-to-answer, not just accuracy, which is an admission that the previous reasoner overthought. The right mental model is a dial on deliberation, and the work is in learning where to set it.

The cost of deliberation

cost(mode) = t_in * p_in + t_think(mode) * p_out + t_answer(mode) * p_out
utility(mode) = P(correct | mode) - lambda * cost(mode)

Input tokens are paid once; the mode changes how many tokens the model writes before answering. Thinking multiplies output cost by roughly (t_think + t_answer) / t_answer while raising the probability of a correct answer. A mode router should choose Think only when the accuracy gain times the value of being right exceeds lambda times the extra output cost. That is why the agent-era framing matters: when the answer feeds a tool call or a code patch, the value of being right is high, so thinking pays; when the answer is a one-line lookup, it usually does not.

A cheap mode router

python

def choose_mode(prompt, task, value_of_correct, latency_budget_ms):
    # features a serving stack can compute before generation
    multi_step = task in {"math", "code", "agent", "search"}
    long_context = task == "long_doc_qa"
    short_answer = task == "extract" or task == "classify"

    if short_answer and latency_budget_ms < 400:
        return "non-think"          # deliberation cannot pay for itself
    if multi_step and value_of_correct > 10:
        return "think"              # a wrong tool call is expensive
    if long_context:
        return "think"              # retrieval over many hops
    return "non-think"

# the model itself does not need to change; only the template does

This is the router DeepSeek avoided shipping twice: the mode lives in the prompt, but the decision of which prompt to render is still yours.
The thresholds encode the economics, not model quality: same weights, different willingness to pay for deliberation.
In V4 generations this router mutates into a numeric reasoning-effort setting on the request.

The agent-era post-training

The second half of the release is about tool use. An agent loop is not a single question: the model calls a function, reads the result, decides the next call, and repeats. That workload stresses different skills than chat - deciding when to call a tool at all, formatting arguments exactly, and recovering when a call fails. DeepSeek post-trained V3.1 on this behavior and shipped strict function calling plus an Anthropic-format API so existing agent frameworks could switch with little code. The reported gains (SWE-bench Verified 66.0, Multilingual 54.5, Terminal-bench 31.3) are the earliest public signal that the same checkpoint can compete on agentic coding, and the Terminus patch shows how quickly the remaining rough edges - language mixing, search behavior - were found in production.

Continued pretraining for context

Before any post-training, V3.1-Base received 840B tokens of continued pretraining on top of the V3 checkpoint, aimed at long-context extension, plus an updated tokenizer configuration. This is the quiet work behind the headline: a 128K context is not a flag you flip; the model has to see enough long documents during training to use the middle of a long window. The same recipe - take the previous model, continue pretraining on long data, then post-train modes and tools - is exactly what the V3.2 experimental release repeats when it adds DeepSeek Sparse Attention, and it is why V3.1 matters in the lineage even though its architecture is unchanged.

2Inside the paper

Problem the release sets out to solve

A deployment could serve a fast instruct model or a slow reasoner, but not both well. The release asks whether the two can be the same artifact, and whether the post-training budget is better spent on agentic behaviors - tool use, multi-step loops - than on another round of chat improvements.

What is actually public here

This is an announcement, not a paper: the primary artifacts are the official release note of August 21, 2025, the Hugging Face model card, and the API changelog. There is no V3.1 technical report, and the paper linked from the model card is the original DeepSeek-V3 report, not a V3.1 write-up. Read the note for the mode design, the 840B-token continued pretraining, and the agent benchmarks; do not expect ablations of the mode switch, because none are published.

Evidence the release notes offer

Reported numbers: SWE-bench Verified 66.0, SWE-bench Multilingual 54.5, Terminal-bench 31.3; thinking mode is faster than R1-0528 on time-to-answer while reaching comparable answer quality; the base model is 671B parameters with 37B activated, 128K context for both modes. The agent scores are the same table that later served as the baseline for V3.2-Exp, which is useful: the Exp release deliberately kept training aligned with V3.1-Terminus so the two could be compared head to head.

Limits the release admits (and some it does not)

The release is vendor reporting: no third-party evaluations are cited for the headline numbers. The thinking-efficiency claim is comparative without a published measurement protocol. The 840B-token continued pretraining lacks data-mix and context-length details. And the mode switch is only as good as the template that invokes it: if the caller renders the wrong template, the model happily runs the wrong mode, and the release offers no learned router. The Terminus patch, one month later, is an implicit admission that the first cut had language-consistency and agent rough edges.

Practitioner takeaway

Treat thinking as a billed resource with a value, not as a quality dial. If your stack serves both fast and deep responses, prefer one checkpoint with prompt-selected modes over two models: it halves the weight memory, unifies evals, and lets you change the mix by traffic policy. And invest post-training in the interface, not just the answer: strict tool schemas and an API format your agent framework already speaks are what make a model usable inside a loop.

3Implementation check

0 of 5 questions answered

1.In the V3.1 API, which mapping between model names and modes is correct?
2.Why can V3.1 offer two behaviors without shipping two models?
3.A team serves a classification endpoint at high QPS with a tight latency budget. Using the cost model from the theory section, what does the router do?
4.What did the 840B-token continued pretraining of V3.1-Base primarily target?
5.Which change most directly lowers the cost of plugging V3.1 into an existing agent framework?

4Build it yourself

Route a two-mode agent on a toy code puzzle

Starter2-3 hours

Rebuild the V3.1 mode switch as a router. Each puzzle hides a two-digit code and hands the agent a noisy clue whose quality varies; fast mode checks one candidate, thinking mode checks k candidates in clue order. Implement the quality router, then compare its accuracy and token cost against fixed budgets.

Milestones

Generate 300 seeded puzzles: a hidden code, a clue with per-puzzle quality, and a deterministic tie-broken candidate ranking.
Precompute the true code's rank in each ranking so every policy is scored on identical puzzles.
Evaluate fixed budgets k = 1, 4, 8 and record accuracy against average tokens per puzzle.
Implement thinking_cost: use clue quality as the cheap feature and return 1, 4, or 12 candidates.
Place the router in the accuracy/token curve and check it sits on or above the interpolation between k=4 and k=8.
Sweep the thresholds and report how far the router can move along the curve.

Starter code

python · editable

import random, itertools
random.seed(31)
# EXPECTED before TODOs: fast 0.433 at 1.00 tokens; think-4 0.643/4.00;
# think-8 0.853/8.00; then "unimplemented: TODO: thinking_cost".
# EXPECTED after TODOs: the router reaches 0.77 accuracy at 5.86 tokens.
N, BASE = 300, 4
CODES = list(itertools.product(range(BASE), repeat=2))
def one_puzzle(rng):
    code = (rng.randrange(BASE), rng.randrange(BASE))
    quality = rng.random()
    clue = tuple(code[i] if rng.random() < quality else rng.randrange(BASE) for i in range(2))
    tie = [(rng.random(), c) for c in CODES]
    return (code, quality, clue, [c for _, c in sorted(tie)])
def thinking_cost(p):
    # TODO: the router. Use p[1] (clue quality) as the cheap feature: check 1
    # candidate if quality > 0.7, 4 if quality > 0.4, else 12. Tokens buy
    # accuracy; the job is to spend them only where they pay.
    raise NotImplementedError("TODO: thinking_cost")
PUZZLES = [one_puzzle(random) for _ in range(N)]
RANKS = [sorted(o, key=lambda c: (c[0] != clue[0]) + (c[1] != clue[1])).index(code)
         for code, _, clue, o in PUZZLES]
def evaluate(cost):
    acc = sum(r < cost(p) for r, p in zip(RANKS, PUZZLES)) / N
    return acc, sum(cost(p) for p in PUZZLES) / N
def main():
    for name, fixed in (("fast   ", 1), ("think-4", 4), ("think-8", 8)):
        a, t = evaluate(lambda p: fixed)
        print(name, "acc", round(a, 3), "tokens", round(t, 2))
    try:
        a, t = evaluate(thinking_cost)
        print("router ", "acc", round(a, 3), "tokens", round(t, 2))
    except NotImplementedError as exc:
        print("unimplemented:", exc)
main()

Runs locally in your browser.

Success criteria

The script exits 0 and prints the three fixed-budget rows plus one router row.
The router's (tokens, accuracy) point is at least as good as linear interpolation between the k=4 and k=8 rows.
Moving the thresholds moves the router along the curve, showing the token/accuracy trade directly.

Stretch goals

Stop thinking early when the top candidate's score margin is decisive, and report the tokens saved.
Learn the thresholds from a held-out split instead of hard-coding them.
Hide the clue quality behind puzzle length so the router has to estimate difficulty from scratch.

Problems

5Keep going

previousNative Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention nextDeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse Attention

Problems

Group Size For Learning Signal

Articles

Post-Training: RLHF → DPO → GRPO

Where this sits

What it is

1·Theory from first principles

Two products became one

The mode switch is a template, not a model

When each mode wins

The agent-era post-training

Continued pretraining for context

2·Inside the paper

Problem the release sets out to solve

What is actually public here

Evidence the release notes offer

Limits the release admits (and some it does not)

Practitioner takeaway

3·Implementation check

4·Build it yourself

Milestones

Success criteria

Stretch goals

Problems

5·Keep going

Problems

Articles

1Theory from first principles

2Inside the paper

3Implementation check

4Build it yourself

5Keep going