SWA Cyclic Phase LR

Problem

Compute the SWA learning rate with a cyclic phase after the switch point.

Before swa_start (or when period is not positive) return base_lr. Afterwards let phase = (step - swa_start) % period and return swa_lr * 0.5 * (1 + cos(pi * phase / period)), a cosine cycle running from swa_lr down to 0.

Hint

The cycle length period resets the cosine every period steps.

Test cases

case 1

in: [0,10,5,0.1,0.05]

expected: 0.1

case 2

in: [10,10,5,0.1,0.05]

expected: 0.05

case 3

in: [12,10,5,0.1,0.05]

expected: 0.032725424859373686

case 4

in: [15,10,5,0.1,0.05]

expected: 0.05

case 5

in: [17,10,5,0.1,0.05]

expected: 0.032725424859373686

case 6

in: [20,10,0,0.1,0.05]

expected: 0.1

Starter codePython

def swa_cyclic_phase_lr(step, swa_start, period, base_lr, swa_lr):
    # Your code here
    pass

6 test cases · run your code against them in the editor