Max Sequence From KV Budget

Problem

Find the longest sequence length that fits a KV cache budget: floor((budget_bytes - model_bytes) / (batch_size * kv_bytes_per_token)). Return 0 if the model alone exceeds the budget or the per-token cache is non-positive.

The signature is max_sequence_from_kv_budget(budget_bytes, model_bytes, kv_bytes_per_token, batch_size). Return an integer.

Hint

The batch shares the cache, so divide the leftover bytes by batch times bytes-per-token.

Test cases

case 1

in: [100000000000,60000000000,1000000,8]

expected: 5000

case 2

in: [100000000000,100000000000,1000000,1]

expected: 0

case 3

in: [10000000000,20000000000,1000000,1]

expected: 0

case 4

in: [80000000000,40000000000,500000,4]

expected: 20000

Starter codePython

def max_sequence_from_kv_budget(budget_bytes, model_bytes, kv_bytes_per_token, batch_size):
    # Your code here
    pass

4 test cases · run your code against them in the editor