Mini Language Model
Next-token prediction over a four-token alphabet generated by a 2nd-order Markov process. Beat the unigram frequency baseline.
Your goal
Beat the unigram frequency baseline — score higher accuracy than 0.3000 on the hidden 30 test rows. The first winning run records 35 points in this browser.
- Train
- 70 × 4
- Hidden test
- 30 rows
- Metric
- accuracy · higher
- Baseline score
- 0.3000
Data
Rows are four-token contexts from the alphabet {0, 1, 2, 3} and the label is the next token. The next token is a deterministic function of the previous two tokens 85% of the time and a uniform draw otherwise. 70 train rows and 30 hidden test rows from one LCG stream (seed 55).
rows: last token of the context · columns: next token · cells: transition counts
Transition counts: rows are the last token of the context, columns the next token; darker cells are more frequent. The heaviest cell is 1 → 0 with 9 of 70 rows.
Method & theory
read before you submitNext-token prediction on a four-token alphabet from a second-order Markov process. The unigram baseline ignores order entirely - contexts are the whole point.
The hidden rule and its noise
Rows are four-token contexts [a, b, c, d] over {0, 1, 2, 3} and the label is the next token. With probability 0.85 the next token is a deterministic function of the last two context tokens: next = (c + d + 1) mod 4, which is the 4x4 table with rows [1,2,3,0], [2,3,0,1], [3,0,1,2], [0,1,2,3]. Otherwise (15% of the time) the next token is uniform over the four tokens. That caps the accuracy of a model that knows the pair exactly at 0.85 + 0.15/4 = 0.8875.
Why the unigram baseline lands on 0.3
The 70 training rows contain token counts [19, 19, 11, 21], so the most frequent next token is 3. The hidden 30 rows have counts [7, 5, 9, 9], so the same always-3 rule is right 9 times: exactly 0.300. It never looks at the context, so it cannot see any of the Markov structure.
Counting pairs is a language model
Build a table over the last two context tokens (c, d) to the observed next tokens, and predict the most frequent continuation, falling back to the unigram pick for unseen pairs. All 16 pairs appear in the 70 training rows and the hidden rows reuse 14 of them, so a majority lookup scores 29/30 = 0.967 - only a three-way tie on the pair (0, 1) and the mixed (0, 0) rows cost anything.
Why the last-token matrix looks flat
The preview's 4x4 matrix counts transitions from the last context token d to the next token. Each row averages over all four possible values of c, and since next = (c + d + 1) mod 4, different c values land in different columns. Row totals of 14, 23, 18, and 15 spread almost evenly across the four columns - that flatness is evidence the process is second order, not first order.
Pitfalls
- Keying the lookup on one token (the last one) instead of the last two. Every row of that 4x4 table mixes all four possible continuations, so accuracy stalls near the uniform rate no matter how many rows you count.
- Keying on the first two context tokens (a, b) instead of the last two (c, d). Those tokens generated the context but do not determine the next token, so the table leaks no usable signal.
- Ignoring ties in a pair's counts. With 70 rows over 16 pairs, ties happen - pair (0, 1) has three different continuations with one vote each - and an arbitrary or unstable tie-break quietly costs hidden rows.
Takeaway. A language model is a conditional distribution; counting a transition table on the last two tokens is the smallest honest version of it.
Hint
The next token depends on the last two tokens. Count (previous two tokens → next token) transitions and predict the most frequent continuation, falling back to the unigram pick for pairs you never saw.