Chapter 3: Fundamentals of Natural Language Processing

Activity 3: Process a Corpus

Solution

1. Import the sklearn TfidfVectorizer and TruncatedSVD methods:

   from sklearn.feature_extraction.text import TfidfVectorizer

   from sklearn.decomposition import TruncatedSVD

2. Load the corpus:

   docs = []

   ndocs = ["doc1", "doc2", "doc3"]

   for n in ndocs:

   aux = open("dataset/"+ n +".txt", "r", encoding="utf8")

   docs.append(aux.read())

3. With spaCy, let's add some new stop words, tokenize the corpus, and remove the stop words. The new corpus without these words will be stored in a new variable:

   import spacy

   import en_core_web_sm

   from spacy.lang.en.stop_words import STOP_WORDS

   nlp = en_core_web_sm.load()

   nlp.vocab["\n\n"].is_stop = True

   nlp.vocab["\n"].is_stop = True

   nlp.vocab["the"].is_stop = True

   nlp.vocab["The"].is_stop = True

   newD = []

   for d, i in zip(docs, range(len(docs))):

   doc = nlp(d)

   tokens = [token.text for token in doc if not token.is_stop and not token.is_punct]

   newD.append(' '.join(tokens))

4. Create the TF-IDF matrix. I'm going to add some parameters to improve the results:

   vectorizer = TfidfVectorizer(use_idf=True,

   ngram_range=(1,2),

   smooth_idf=True,

   max_df=0.5)

   X = vectorizer.fit_transform(newD)

5. Perform the LSA algorithm:

   lsa = TruncatedSVD(n_components=100,algorithm='randomized',n_iter=10,random_state=0)

   lsa.fit_transform(X)

6. With pandas, we are shown a sorted DataFrame with the weights of the terms of each concept and the name of each feature:

   import pandas as pd

   import numpy as np

   dic1 = {"Terms": terms, "Components": lsa.components_[0]}

   dic2 = {"Terms": terms, "Components": lsa.components_[1]}

   dic3 = {"Terms": terms, "Components": lsa.components_[2]}

   f1 = pd.DataFrame(dic1)

   f2 = pd.DataFrame(dic2)

   f3 = pd.DataFrame(dic3)

   f1.sort_values(by=['Components'], ascending=False)

   f2.sort_values(by=['Components'], ascending=False)

   f3.sort_values(by=['Components'], ascending=False)

   The output is as follows:

Figure 3.26: Output example of the most relevant words in a concept (f1)

Note:

Do not worry if the keywords are not the same as yours, if the keywords represent a concept, it is a valid result.