IP-Adapter Decoupled Cross-Attention

Problem

Combine text and image attention streams: out = attn(query, text_keys, text_values) + weight * attn(query, image_keys, image_values), where each attention is standard scaled dot-product attention.

The signature is ip_adapter_combine(query, text_keys, text_values, image_keys, image_values, weight).

Hint

Decoupled cross-attention keeps the pretrained text path intact and learns a separate image path.

Test cases

case 1

in: [[1,0],[[1,0]],[[1,2]],[[1,0]],[[3,4]],0.5]

expected: [2.5,4]

case 2

in: [[1,0],[[1,0]],[[1,2]],[[1,0]],[[3,4]],0]

expected: [1,2]

case 3

in: [[1,0],[[1,0],[0,1]],[[1,2],[3,4]],[[1,0],[0,1]],[[5,6],[7,8]],1]

expected: [7.3209538027,9.3209538027]

case 4

in: [[1,1],[[1,0]],[[2,0]],[[0,1]],[[0,4]],2]

expected: [2,8]

Starter codePython

import math
def ip_adapter_combine(query, text_keys, text_values, image_keys, image_values, weight):
    # Your code here
    pass

4 test cases · run your code against them in the editor