Understanding Natural Language Processing (NLP): The Bridge Between Human Language and Artificial Intelligence

📗 Chapter 2: Core NLP Tasks and Techniques

Essential Capabilities That Power Modern Natural Language Understanding

🧠 Introduction

In Chapter 1, we laid the foundation for Natural Language Processing (NLP) by discussing the structure of language and preprocessing. Now, it’s time to dive into the core tasks and techniques that make NLP systems functional, intelligent, and truly interactive.

This chapter covers the essential NLP tasks like Part-of-Speech (POS) tagging, Named Entity Recognition (NER), text classification, sentiment analysis, summarization, and more—complete with real-world code examples and tool comparisons.

📘 Section 1: Part-of-Speech (POS) Tagging

🔍 What is POS Tagging?

Part-of-Speech tagging assigns a word its corresponding grammatical role—such as noun, verb, adjective, or adverb—based on its context in a sentence.

🧪 Code: POS Tagging with spaCy

python

import spacy

nlp = spacy.load("en_core_web_sm")

doc = nlp("The quick brown fox jumps over the lazy dog")

for token in doc:

    print(token.text, token.pos_)

📘 Section 2: Named Entity Recognition (NER)

🧠 What is NER?

NER identifies and classifies entities in text into predefined categories like:

• PERSON
• ORGANIZATION
• LOCATION
• DATE
• MONEY

🧪 Code: NER with spaCy

python

doc = nlp("Apple Inc. was founded by Steve Jobs in Cupertino in 1976.")

for ent in doc.ents:

    print(ent.text, ent.label_)

📘 Section 3: Text Classification

📄 What is Text Classification?

This involves assigning a category label to a piece of text.

Common classification tasks:

• Spam detection
• Topic categorization
• Emotion detection
• Intent classification (chatbots)

🧪 Code: Text Classification with scikit-learn

python

from sklearn.feature_extraction.text import CountVectorizer

from sklearn.naive_bayes import MultinomialNB

texts = ["I love this product", "Worst experience ever", "Absolutely fantastic", "I want a refund"]

labels = ["positive", "negative", "positive", "negative"]

vectorizer = CountVectorizer()

X = vectorizer.fit_transform(texts)

model = MultinomialNB()

model.fit(X, labels)

test = vectorizer.transform(["This is horrible"])

print(model.predict(test))  # Output: ['negative']

📘 Section 4: Sentiment Analysis

Sentiment analysis determines whether the sentiment behind a piece of text is positive, negative, or neutral.

🧪 Code: Sentiment Analysis with TextBlob

python

from textblob import TextBlob

text = TextBlob("I am so happy with the customer service!")

print("Polarity:", text.sentiment.polarity)  # Range: -1 to +1

📘 Section 5: Text Summarization

Summarization condenses a long article or document into a shorter version, preserving key information.

✂️ Types:

• Extractive: Selects key sentences (e.g., TextRank)
• Abstractive: Rewrites content in a new form (e.g., BART, T5)

🧪 Code: Extractive Summarization with Gensim

python

from gensim.summarization import summarize

text = """

Natural language processing (NLP) is a field of artificial intelligence that enables computers to understand, interpret, and generate human language. It has applications in chatbots, translation, speech recognition, and more.

"""

print(summarize(text, ratio=0.5))

📘 Section 6: Topic Modeling

Topic modeling uncovers hidden themes or topics in large collections of text using unsupervised learning.

🔍 Common Methods:

• Latent Dirichlet Allocation (LDA)
• Non-negative Matrix Factorization (NMF)

🧪 Code: Topic Modeling with LDA (Gensim)

python

from gensim import corpora, models

docs = ["NLP is fun and exciting", "Machine learning is a subset of AI", "NLP includes machine translation"]

tokens = [doc.lower().split() for doc in docs]

dictionary = corpora.Dictionary(tokens)

corpus = [dictionary.doc2bow(text) for text in tokens]

lda_model = models.LdaModel(corpus, num_topics=2, id2word=dictionary)

for idx, topic in lda_model.print_topics(-1):

    print(f"Topic {idx}: {topic}")

📘 Section 7: POS Tagging vs NER vs Text Classification – Quick Comparison

📘 Section 8: Tools and Libraries Overview

✅ Chapter Summary Table