Distributed Sync Interval

Problem

Return the 0-based micro-batch indices at which gradients are synchronized, given that synchronization happens every sync_every micro-batches (indices sync_every - 1, 2 * sync_every - 1, ...).

The signature is sync_interval(micro_batches, sync_every).

Hint

With sync_every = 1 every micro-batch triggers an all-reduce.

Test cases

case 1

in: [4,2]

expected: [1,3]

case 2

in: [5,5]

expected: [4]

case 3

in: [3,1]

expected: [0,1,2]

case 4

in: [0,2]

expected: []

Starter codePython

def sync_interval(micro_batches, sync_every):
    # Your code here
    pass

4 test cases · run your code against them in the editor