Top 4 Web Scraping Interview Questions in Python (With Solutions and Pro Tips)

Chapter 5: Scraping Paginated Data (Multiple Pages)

Most useful datasets aren’t found on a single page. Think product listings on Amazon, articles on a blog, or job postings on LinkedIn. These use pagination to divide results across multiple pages.

In this chapter, you’ll learn:

• How to detect and work with URL-based pagination
• How to scrape multiple pages with a for loop
• How to use BeautifulSoup to combine results
• When to stop pagination (next button or last page)
• Best practices for looping, throttling, and saving multi-page data

📄 What is Pagination?

Pagination splits content into parts, usually with:

• Page numbers in URLs
  E.g., example.com/page/1, page=2, etc.
• "Next" buttons or links
• Load more buttons (infinite scroll – dynamic; handled in another chapter)

🔍 Example 1: Pagination via URL Parameter

Let’s say we’re scraping blog articles from a site like:

https://example.com/blog?page=1

https://example.com/blog?page=2

...

Each page shows a list of posts like:

<div class="post">

  <h2>Post Title</h2>

  <p>Published: 2024-01-01</p>

</div>

✅ Step-by-Step Scraper for Multiple Pages

import requests

from bs4 import BeautifulSoup

import time

base_url = "https://example.com/blog?page="

for page in range(1, 6):  # scrape pages 1 to 5

    print(f"Scraping page {page}...")

    url = base_url + str(page)

    headers = {'User-Agent': 'Mozilla/5.0'}

    response = requests.get(url, headers=headers)

    soup = BeautifulSoup(response.text, 'html.parser')

    posts = soup.find_all('div', class_='post')

    for post in posts:

        title = post.find('h2').text

        date = post.find('p').text

        print(f"{title} | {date}")

    time.sleep(1)  # be polite!

📊 Output (Sample):

Scraping page 1...

Learn Python Fast | Published: 2024-01-01

Master Data Science | Published: 2024-01-02

...

Scraping page 2...

...

🔄 Example 2: Paginate Until No More Results

Some sites don’t limit the number of pages explicitly. Instead, you keep scraping until no results appear.

page = 1

while True:

    url = f"https://example.com/blog?page={page}"

    response = requests.get(url)

    soup = BeautifulSoup(response.text, 'html.parser')

    posts = soup.find_all('div', class_='post')

    if not posts:

        break  # no more results

    for post in posts:

        print(post.find('h2').text)

    page += 1

    time.sleep(1)

🧠 Detecting Pagination Patterns

Inspect the pagination area in browser DevTools or look at the network requests.

🧑‍💻 Example 3: Follow “Next” Links Dynamically

Some sites change the pagination URLs each time.

url = "https://example.com/blog"

while url:

    response = requests.get(url)

    soup = BeautifulSoup(response.text, 'html.parser')

    for post in soup.select('.post h2'):

        print(post.text)

    next_btn = soup.find('a', text='Next')

    if next_btn:

        url = "https://example.com" + next_btn['href']

    else:

        url = None

    time.sleep(1)

📁 Store Results to CSV

import csv

with open('posts.csv', 'w', newline='') as f:

    writer = csv.writer(f)

    writer.writerow(['Title', 'Date'])

    for page in range(1, 6):

        url = f"https://example.com/blog?page={page}"

        soup = BeautifulSoup(requests.get(url).text, 'html.parser')

        posts = soup.find_all('div', class_='post')

        for post in posts:

            title = post.find('h2').text

            date = post.find('p').text

            writer.writerow([title, date])

💡 Best Practices

⚠️ Common Mistakes

📦 Real-World Use Cases

• Scraping all product listings from e-commerce sites
• Extracting all job posts from a hiring board
• Collecting all articles in a news archive
• Gathering all course listings from an online platform

✅ Summary

In this chapter, you learned:

• How to scrape multiple pages using loops
• How to detect and use pagination in URLs
• How to follow “Next” links dynamically
• Best practices for performance, ethics, and error handling
• How to save multi-page data into CSV