Comm-Compute Overlap

Problem

Compute exposed communication time and overlap ratio for a training step: exposed = compute + comm - overlap and ratio = overlap / (compute + comm).

The signature is comm_compute_overlap(compute_ms, comm_ms, overlap_ms). Return [exposed_ms, overlap_ratio].

Hint

Backward all-reduce can hide behind the backward compute of later layers.

case 1

in: [100,50,25]

expected: [125,0.1666666667]

case 2

in: [100,100,100]

expected: [100,0.5]

case 3

in: [10,10,0]

expected: [20,0]

case 4

in: [200,50,50]

expected: [200,0.2]

Starter codePython

def comm_compute_overlap(compute_ms, comm_ms, overlap_ms):
    # Your code here
    pass

4 test cases · run your code against them in the editor