Communication To Compute Ratio

Problem

Compute the gradient all-reduce time and its ratio to compute time for one data-parallel step: comm_ms = 2 * (num_gpus - 1) / num_gpus * param_bytes / (bandwidth_gbps * 1e6).

The signature is communication_compute_ratio(param_bytes, num_gpus, bandwidth_gbps, compute_ms). Return [comm_ms, ratio].

Hint

A ratio near or above 1 means communication cannot hide behind compute.

Test cases

case 1

in: [4000000000,8,200,100]

expected: [35,0.35]

case 2

in: [1000000000,2,100,50]

expected: [10,0.2]

case 3

in: [8000000000,16,400,20]

expected: [37.5,1.875]

case 4

in: [1000000,4,50,10]

expected: [0.03,0.003]

Starter codePython

def communication_compute_ratio(param_bytes, num_gpus, bandwidth_gbps, compute_ms):
    # Your code here
    pass

4 test cases · run your code against them in the editor