Understanding Natural Language Processing (NLP): The Bridge Between Human Language and Artificial Intelligence

📗 Chapter 3: Language Modeling and Vector Representations

How Machines Learn and Represent Human Language

🧠 Introduction

At the core of every NLP system lies the ability to understand and predict language. This understanding is powered by two pillars:

• Language Modeling: Predicting sequences of words.
• Vector Representations: Converting words into numerical form that machines can process.

Without these, NLP tasks like translation, text classification, chatbots, and question answering wouldn’t be possible. This chapter dives deep into the concepts, mathematics, and practical implementations of language models and word embeddings.

📘 Section 1: What is a Language Model?

A language model (LM) assigns a probability to a sequence of words. It helps answer questions like:

• What word is likely to come next?
• Is this sentence grammatically correct?
• What is the probability of this sequence?

📌 Formal Definition

Given a sequence of words:
w₁, w₂, ..., wₙ
A language model estimates:
P(w₁, w₂, ..., wₙ) = P(w₁) * P(w₂|w₁) * ... * P(wₙ|w₁, ..., wₙ₋₁)

📊 Example:

📘 Section 2: N-gram Language Models

The simplest LMs are n-gram models, which predict the next word using the previous (n-1) words.

🔹 Types:

• Unigram (n=1): Assumes all words are independent.
• Bigram (n=2): Depends on the previous word.
• Trigram (n=3): Depends on the previous two words.

🧪 Code: Bigram Model in Python

python

from collections import defaultdict

corpus = "I love NLP and NLP loves me".lower().split()

bigrams = list(zip(corpus[:-1], corpus[1:]))

model = defaultdict(lambda: defaultdict(int))

for w1, w2 in bigrams:

    model[w1][w2] += 1

# Normalize

for w1 in model:

    total = float(sum(model[w1].values()))

    for w2 in model[w1]:

        model[w1][w2] /= total

print(model["nlp"])  # e.g., {'and': 0.5, 'loves': 0.5}

📘 Section 3: Neural Language Models

N-gram models struggle with long-range dependencies. Neural language models address this by using embeddings and hidden layers.

🔹 Key Architectures:

🧪 Code: Language Modeling with GPT2 (Hugging Face)

python

from transformers import GPT2Tokenizer, GPT2LMHeadModel

import torch

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")

model = GPT2LMHeadModel.from_pretrained("gpt2")

input_ids = tokenizer("The future of NLP is", return_tensors="pt").input_ids

output = model.generate(input_ids, max_length=10)

print(tokenizer.decode(output[0]))

📘 Section 4: What are Vector Representations?

Before a machine can work with words, they need to be converted into numbers. Vector representations are dense vectors where similar words are closer in space.

🧠 Why Not One-Hot Encoding?

Problems:

• High dimensionality
• No similarity encoding
• Sparse representation

📘 Section 5: Word Embeddings

Word embeddings solve the above by mapping words to dense, low-dimensional vectors.

🔹 Popular Embedding Methods:

🧪 Code: Word2Vec with Gensim

python

from gensim.models import Word2Vec

sentences = [["I", "love", "natural", "language", "processing"],

             ["NLP", "is", "fun"]]

model = Word2Vec(sentences, vector_size=50, window=2, min_count=1, workers=2)

print(model.wv.most_similar("NLP"))

📘 Section 6: Contextual vs Non-Contextual Embeddings

🧪 Code: BERT Embedding with Hugging Face

python

from transformers import BertTokenizer, BertModel

import torch

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

model = BertModel.from_pretrained('bert-base-uncased')

inputs = tokenizer("The bank was flooded after the storm", return_tensors="pt")

outputs = model(**inputs)

print(outputs.last_hidden_state.shape)  # [batch_size, sequence_length, hidden_size]

📘 Section 7: Visualizing Embeddings

To understand embeddings, visualize them using dimensionality reduction:

• PCA (Principal Component Analysis)
• t-SNE (t-distributed stochastic neighbor embedding)

🧪 Code: t-SNE for Word Embeddings

python

from sklearn.manifold import TSNE

import matplotlib.pyplot as plt

words = list(model.wv.key_to_index.keys())[:100]

vectors = [model.wv[word] for word in words]

tsne = TSNE(n_components=2)

reduced = tsne.fit_transform(vectors)

plt.figure(figsize=(14,10))

for i, word in enumerate(words):

    plt.scatter(reduced[i][0], reduced[i][1])

    plt.annotate(word, (reduced[i][0], reduced[i][1]))

plt.show()

📘 Section 8: Embeddings for Sentences and Documents

You can embed entire sentences or documents using models like:

• Sentence-BERT
• Universal Sentence Encoder
• Doc2Vec

🧪 Code: Sentence Embedding with Sentence-BERT

python

from sentence_transformers import SentenceTransformer

model = SentenceTransformer('all-MiniLM-L6-v2')

sentences = ["This is a good book", "I enjoyed reading it"]

embeddings = model.encode(sentences)

print(embeddings.shape)  # (2, 384)

✅ Chapter Summary Table