TF-IDF

April 13, 2020

Page content

Definition

Document (sentence) -> words.

This term is irrelavant to the corpus (defined at each document).

There are several definitions of “term frequency.”

f_{t, d_{i}} := t h e n u m b e r o f t h e w o r d s t i n t h e d o c u m e n t d_{i}

The most simple and “meaningful” for NLP is frequency form,

t f (t, d_{i}) = \frac{f_{t, d_{i}}}{t h e t o t a l n u m b e r o f w o r d s i n a d o c u m e n t d_{i}}

There are also logarithmically scaled version,

t f (t, d) = \log (1 + f_{t, d})

The preferred base of the logarithm are 2 or 10.

There are several definitions also,

i d f (t, D) = \log (\frac{t h e t o t a l n u m b e r o f d o c u m e n t s i n t h e c o r p u s D}{T h e n u m b e r o f d o c u m e n t s w h i c h c o n t a i n t h e w o r d t})

When the word t is rare, IDF of the word become high.

In scikit-learn,

i d f (t, D) = 1 + \log (\frac{t h e t o t a l n u m b e r o f d o c u m e n t s i n t h e c o r p u s D}{T h e n u m b e r o f d o c u m e n t s w h i c h c o n t a i n t h e w o r d t})

TF times IDF means how important a word is.

We can assign a importance vector (TFIDF vector) to a document in corpus.

SVM categorizing.