Quantization: INT8 to FP8

Serving a model is a memory problem before it is a computing problem. Every weight sits in memory, and every generated token pulls those weights through the processor. Cut the bytes per weight in half and you roughly double the speed at which tokens come out.

Quantization means storing numbers with fewer possible values. A standard 32-bit float takes 4 bytes. A quantized weight can use one byte, which allows 256 possible values. That is four times smaller than float32 and half the size of the 16-bit formats used for training.

The price is rounding error. Quantization is lossy compression, and the engineering job is to put the error where it hurts the model least.

The usual recipe is a straight-line map: x ≈ scale · (q − zero_point). Here x is the original value, q is the stored integer code, scale says how much real value one step of code is worth, and zero_point is the code that represents real zero.

Symmetric quantization centers the range on zero. Then zero maps to exactly zero, the zero point is 0, and the scale is absmax / 127, where absmax is the largest absolute value in the data and 127 is the largest code in the signed 8-bit range. It is popular for weights because zero stays exact.

Asymmetric quantization fits the actual smallest and largest values, so its zero point is usually not zero. It spends all 256 codes on the range you really have, which pays off when the values are lopsided, at the cost of the extra bookkeeping.

Outliers decide how much precision everything else loses. If one weight is 5 while the rest sit between −2 and 2, the scale must cover 5, and every ordinary value gets a coarse step. The repair is clipping: cap the range at a threshold and accept error on the few values beyond it.

Clipping is a trade. A narrower range gives typical values finer steps, and everything outside becomes a constant at the edge. That is the error you can steer.

Granularity decides how local the scale is. A tensor is just the general word for a grid of numbers. Per-tensor quantization shares one scale across the whole grid, so a single outlier damages every value. Per-channel gives each row or column its own scale, so the outlier only distorts its own channel. Per-tensor is cheaper; per-channel is the default for weights.

FP8 — an 8-bit float — enters here. It spends some bits on an exponent, which records a number's scale, the way 10⁷ records the scale in scientific notation. This packs levels tightly near zero and stretches them far out, covering a wide range without a custom scale. Recent GPUs run FP8 at full speed.

Drag the clip threshold and watch the error bars. Pull it down and typical values get finer steps while the outliers pay more. Push it up and the outliers survive at the cost of coarser steps everywhere.

The error histogram shows who pays. A tight clump means the number format fits the data. Long tails mean the range is too wide.

SQNR — signal-to-quantization-noise ratio — captures the trade in decibels: signal power divided by error power, on a log scale. Above roughly 40 dB, a quantized model usually matches the original on benchmarks.

Turn on per-channel and the histogram narrows immediately. That is the outlier story in one click: with four channels there are four scales, so the big value only distorts its own channel.

In practice, weights use per-channel 8-bit codes, or 4-bit codes in small groups. Activations — the intermediate values computed while the model runs — are harder, because they change with every input, so they use one scale per tensor, calibrated on sample data or computed on the fly.

W8A8 means weights and activations both use 8 bits, which speeds up both prompt processing and token generation. W4A16 keeps weights in 4 bits and runs the arithmetic in 16, which saves memory but not compute. GPTQ and AWQ are the common recipes for that split.

Error compounds through layers. A layer that receives quantized inputs quantizes its own outputs on top of them, so end-to-end accuracy is the number that matters; per-layer error is only a hint.

Calibration data should look like production data. If the input distribution shifts, a scale fitted on the old data clips the new values. Monitoring the ranges catches the drift.

The format must also match the hardware. INT8 kernels are everywhere. FP8 needs Hopper, Blackwell, or newer chips. NVFP4 and MXFP4 are Blackwell-native. Choosing a format the deployment chip does not accelerate is the most expensive quantization mistake.

Quantization is also how large models fit on small hardware. QLoRA keeps the frozen base model in a 4-bit format called NF4 and trains small 16-bit adapters on top. The base never changes, so its error is fixed and the adapters learn around it.

KV cache quantization is the decoding-time cousin. Caching attention keys and values in FP8 instead of the usual 16-bit format nearly halves the memory per token, which is why FP8 is a common default for long contexts.

The problems below compute an INT8 scale, quantize and dequantize, work through per-channel scales, and compare 4-bit against 8-bit memory.

When you write the arithmetic by hand, watch two things: how you round and where you clamp. A scale without a clamp silently overflows, and a clamp without the right scale throws away range for nothing.