HardNLPproj-021

BM25 Score

Problem

Score one document against a query with BM25.

document_frequency maps a term to the number of documents containing it, num_docs is the collection size, and avg_doc_length is the mean document length. For each query term use idf = ln((N - df + 0.5) / (df + 0.5) + 1) and tf = the raw count in the document, contributing idf * (tf * (k1 + 1)) / (tf + k1 * (1 - b + b * dl / avg_doc_length)). Skip terms absent from the document or collection, use k1=1.5 and b=0.75 by default, and return 0.0 for an empty document or zero average length.

Hint

Term saturation and length normalization live in the denominator of each term.

Test cases

case 1

in: [["cat"],["cat","sat"],{"cat":1},2,2]

expected: 0.6931471805599453

case 2

in: [["a","b"],["a","a","b","c"],{"a":2,"b":2},4,3]

expected: 1.4971201375348047

case 3

in: [["x"],["a","b"],{"x":1},2,2]

expected: 0

case 4

in: [["a"],[],{"a":1},1,1]

expected: 0

Starter codePython

import math
def bm25_score(query_tokens, doc_tokens, document_frequency, num_docs, avg_doc_length, k1=1.5, b=0.75):
    # Your code here
    pass

4 test cases · run your code against them in the editor