CLIP Prompt Ensemble

Problem

Compute zero-shot class probabilities with prompt ensembling.

For each class average its prompt embeddings, then normalize image_embed and each class mean, scale the cosine similarities by logit_scale and apply softmax. Return the class probabilities rounded to 4 decimal places.

Test cases

case 1

in: [[1,0],[[[1,0],[0.8,0.2]],[[0,1]]],1]

expected: [0.7299,0.2701]

case 2

in: [[1,0],[[[1,0]],[[0,1]]],2]

expected: [0.8808,0.1192]

case 3

in: [[1,1],[[[1,0],[0,1]],[[1,0]]],1]

expected: [0.5727,0.4273]

Starter codePython

def clip_prompt_ensemble(image_embed, class_prompts, logit_scale):
    # Your code here
    pass

3 test cases · run your code against them in the editor