MediumNLPproj-018

TF-IDF Document Vector

Problem

Build a sparse TF-IDF vector for one document.

tokens is the tokenized document, document_frequency maps each term to the number of documents containing it, and num_docs is the collection size. For each unique term use tf = count / len(tokens) and idf = ln(num_docs / df), then return a dict mapping term to tf * idf. Empty tokens return an empty dict.

Hint

Terms that appear in every document get idf = 0 and vanish from the vector.

Test cases

case 1

in: [["a","a","b"],{"a":2,"b":1},2]

expected: {"a":0,"b":0.23104906018664842}

case 2

in: [["a"],{"a":1},2]

expected: {"a":0.6931471805599453}

case 3

in: [["a","b"],{"a":1,"b":2},4]

expected: {"a":0.6931471805599453,"b":0.34657359027997264}

case 4

in: [[],{},3]

expected: {}

Starter codePython

import math
def tfidf_vector(tokens, document_frequency, num_docs):
    # Your code here
    pass

4 test cases · run your code against them in the editor