MediumNLPnlp-003

TF-IDF

Problem

Compute the TF-IDF matrix for a list of documents.

docs is a list of token lists. vocab is a list of unique terms.

For term t and document d: tf(t, d) = count(t in d) / len(d) (0 if len(d) == 0) idf(t) = log(N / df(t)) (N = number of docs, df(t) = number of docs containing t) tfidf(t, d) = tf(t, d) * idf(t)

Return an n_docs x n_vocab matrix.

Hint

df(t) counts docs containing t (not total occurrences). log is natural log.

Test cases

case 1

in: [[["the","cat","sat"],["the","dog","ran"],["the","bird"]],["the","cat","dog","ran","bird","sat"]]

expected: [[0,0.3662040962227032,0,0,0,0.3662040962227032],[0,0,0.3662040962227032,0.3662040962227032,0,0],[0,0,0,0,0.549306144333982,0]]

case 2

in: [[["a","b"],["a","c"]],["a","b","c"]]

expected: [[0,0.34657359027997264,0],[0,0,0.34657359027997264]]

case 3

in: [[["a","a","b"],["b","c"]],["a","b","c"]]

expected: [[0.46209812037329684,0,0],[0,0,0.34657359027997264]]

Starter codePython

import math
def tfidf(docs, vocab):
    # Your code here
    pass

3 test cases · run your code against them in the editor