Per-Token Quant Scales

Problem

Compute per-token (per-row) quantization scales as max(|row|) / (2^(bits-1)-1), the LLM.int8 style scaling that isolates outlier tokens. Return 0.0 for an all-zero row.

The signature is per_token_scales(activations, bits=8).

Hint

Per-token scales stop one loud token from widening every other token range.

Test cases

case 1

in: [[[1,-2],[0.5,0.5]],8]

expected: [0.015748031496062992,0.003937007874015748]

case 2

in: [[[0,0]],8]

expected: [0]

case 3

in: [[[4]],4]

expected: [0.5714285714285714]

Starter codePython

def per_token_scales(activations, bits=8):
    # Your code here
    pass

3 test cases · run your code against them in the editor