DeiT Distillation Token

Problem

Prepend both special tokens used by DeiT: first the class token, then the distillation token, followed by the patch embeddings.

The signature is deit_distill_token(x, cls_token, distill_token). Return the new token sequence.

Hint

The distillation token learns from the teacher's hard labels while the class token learns from ground-truth labels.

case 1

in: [[[1,2],[3,4]],[0,0],[9,9]]

expected: [[0,0],[9,9],[1,2],[3,4]]

case 2

in: [[[1]],[1],[2]]

expected: [[1],[2],[1]]

case 3

in: [[],[0],[0]]

expected: [[0],[0]]

case 4

in: [[[5,6]],[0.5,0.5],[-0.5,-0.5]]

expected: [[0.5,0.5],[-0.5,-0.5],[5,6]]

Starter codePython

def deit_distill_token(x, cls_token, distill_token):
    # Your code here
    pass

4 test cases · run your code against them in the editor