Menu

Converting Reviews into TF-IDF Features

Converting Reviews into TF-IDF Features

The TfidfVectorizer converts each review into a numerical vector based on the importance of its words.

Each review becomes a row, and each unique word becomes a feature (column). The values represent the TF-IDF scores.

Import TF-IDF Vectorizer

from sklearn.feature_extraction.text import TfidfVectorizer

Create the Vectorizer

tfidf = TfidfVectorizer(
max_features=5000
)

Explanation

  • max_features=5000 limits the vocabulary to the 5,000 most important words, reducing memory usage and improving training speed.

Transform the Reviews

X = tfidf.fit_transform(df['clean_review'])

Display the Shape of the Feature Matrix

X.shape

Explanation

The output represents:

  • Number of reviews (rows)
  • Number of selected TF-IDF features (columns)

For example:

(40432, 5000)

This means:

  • 40,432 reviews
  • 5,000 numerical features