GQA Cache Savings Fraction

Problem

Compute the percentage of key/value cache bytes saved by grouped-query attention, which stores num_kv_heads heads instead of num_query_heads heads.

The signature is gqa_cache_savings(num_query_heads, num_kv_heads). The result is 100 * (1 - num_kv_heads / num_query_heads).

Hint

Only the K and V projections shrink; the ratio of stored heads is the whole story.

case 1

in: [32,8]

expected: 75

case 2

in: [32,32]

expected: 0

case 3

in: [32,1]

expected: 96.875

case 4

in: [12,4]

expected: 66.6666666667

Starter codePython

def gqa_cache_savings(num_query_heads, num_kv_heads):
    # Your code here
    pass

4 test cases · run your code against them in the editor