Top 10 Data Cleaning Techniques in Python: Master the Art of Preprocessing for Accurate Analysis

📘 Chapter 9: Regex-Based Cleaning in Python

Master Pattern-Based Data Cleaning with Regular Expressions (Regex)

🧠 Introduction

When dealing with messy, unstructured, or semi-structured data, traditional cleaning methods can fall short. This is where Regular Expressions (Regex) come in — a powerful tool that lets you search, match, extract, and clean data based on patterns. Whether you need to extract phone numbers, clean email fields, validate formats, or strip out special characters, regex is your best friend.

In this chapter, you’ll learn:

• What regex is and how it works in Python
• How to clean, extract, and validate data using regex
• Regex patterns for emails, numbers, dates, and more
• How to apply regex cleaning in Pandas
• Real-world data wrangling examples

🔍 What is Regex?

Regex is a sequence of characters that forms a search pattern. You can use it to:

• Search for specific sequences
• Extract matched patterns
• Replace or remove unwanted patterns

Python provides built-in regex support through the re module, and Pandas enhances it with .str.replace(), .str.extract(), and .str.contains().

🛠 Step 1: Importing Required Libraries

python

import re

import pandas as pd

🔎 Step 2: Basic Regex Syntax and Rules

📦 Step 3: Cleaning Text Fields with Regex in Pandas

▶ Sample Data

python

df = pd.DataFrame({

    'Email': ['john.doe@gmail.com', 'jane@domain', 'invalid@.com'],

    'Phone': ['+1-202-555-0193', '2025550194', 'abc123'],

    'Bio': ['Hi! 😊 I work in <b>AI</b>.', 'Contact me at john@example.com.', 'N/A']

})

🔍 Step 4: Validate and Extract Email Addresses

▶ Regex to Match Valid Emails

python

email_pattern = r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}'

# Extract emails

df['Extracted_Email'] = df['Email'].str.extract(f'({email_pattern})')

# Check if valid

df['Is_Valid_Email'] = df['Email'].str.match(email_pattern)

☎️ Step 5: Clean and Extract Phone Numbers

▶ Regex for US Phone Number

python

phone_pattern = r'(\+?\d{1,2}[-.\s]?)?(\(?\d{3}\)?[-.\s]?)?\d{3}[-.\s]?\d{4}'

# Extract phone numbers

df['Extracted_Phone'] = df['Phone'].str.extract(f'({phone_pattern})')

🧼 Step 6: Remove HTML Tags from Text

python

df['Clean_Bio'] = df['Bio'].str.replace(r'<.*?>', '', regex=True)

✨ Step 7: Remove Emojis and Non-ASCII Characters

python

df['Clean_Bio'] = df['Clean_Bio'].str.replace(r'[^\x00-\x7F]+', '', regex=True)

📊 Step 8: Replace Multiple Spaces with One

python

df['Clean_Bio'] = df['Clean_Bio'].str.replace(r'\s+', ' ', regex=True).str.strip()

🧠 Step 9: Extract Hashtags or Mentions

python

text_series = pd.Series(['Follow @openai and tag #chatgpt!'])

# Extract mentions

text_series.str.extractall(r'(@\w+)')

# Extract hashtags

text_series.str.extractall(r'(#\w+)')

🔢 Step 10: Extract Numbers, Dates, or Codes

python

# Extract digits only

df['Digits'] = df['Bio'].str.extract(r'(\d+)')

# Extract formatted date (e.g., 2023-04-15)

date_series = pd.Series(['Posted on 2023-04-15.'])

date_series.str.extract(r'(\d{4}-\d{2}-\d{2})')

📊 Summary Table: Common Regex Cleaning Tasks

🧪 Real-World Applications

🚫 Common Pitfalls

🧠 Bonus: Use Python’s re Module for Advanced Matching

python

text = 'Email: alice@example.com, Phone: 123-456-7890'

# Extract using re

emails = re.findall(email_pattern, text)

phones = re.findall(phone_pattern, text)

print(emails, phones)

🏁 Conclusion

Regex is a superpower for any data professional. It allows you to search, extract, clean, and validate data with flexible patterns — especially when working with messy text inputs, log files, or scraped content. With Pandas and Python’s re module, regex becomes an essential tool in your data cleaning workflow.

Whether you're building a model, cleaning user input, or validating records, regex gives you surgical precision.