TTFT Estimate

Problem

Estimate time to first token in milliseconds: prefill_flops / gpu_flops_per_sec * 1000 + overhead_ms.

The signature is ttft_ms(prefill_flops, gpu_flops_per_sec, overhead_ms=0.0).

Hint

Prefill processes the whole prompt in parallel, unlike memory-bound decode.

case 1

in: [1000000000000,100000000000000,5]

expected: 15

case 2

in: [100000000000,100000000000000,0]

expected: 1

case 3

in: [10000000000000,50000000000000,2]

expected: 202

case 4

in: [0,100000000000000,1]

expected: 1

Starter codePython

def ttft_ms(prefill_flops, gpu_flops_per_sec, overhead_ms=0.0):
    # Your code here
    pass

4 test cases · run your code against them in the editor