Handling Missing Data Like a Pro: Smart Strategies Every Data Scientist Should Know

📗 Chapter 2: Identifying and Profiling Missing Data

Detecting Data Gaps Before They Derail Your Project

🧠 Introduction

Before you can clean or impute missing data, you must first identify and understand what’s missing, how much is missing, and where patterns exist.

“You can’t fix what you don’t measure.” – That’s especially true for missing data.

In this chapter, you'll learn:

• How to detect missing values in structured datasets
• Techniques to visualize and quantify missingness
• How to distinguish between visible and hidden missing values
• Tools and libraries for profiling missingness
• Real-world strategies to evaluate missing data impact

Let’s explore how to shine a light on what’s not in your data.

🔍 1. What Does “Identifying Missing Data” Mean?

Identifying missing data isn’t just spotting NaN or None. It means profiling your dataset to detect:

• Explicit missing values (nulls, blanks)
• Implicit missing values (e.g., "Unknown", -999)
• Structural gaps (columns completely null for a segment)
• Patterned missingness (e.g., one column missing only when another is a certain value)

📦 2. Common Representations of Missing Values

✅ Replace Custom Missing with np.nan

python

import numpy as np

df.replace(['Unknown', 'N/A', -999, 0], np.nan, inplace=True)

📊 3. Quantifying Missing Data

Count & Percentage:

python

missing_count = df.isnull().sum()

missing_percent = df.isnull().mean() * 100

missing_df = pd.DataFrame({

    'Missing Values': missing_count,

    'Percentage': missing_percent

}).sort_values(by='Percentage', ascending=False)

Example Output:

📈 4. Visualizing Missing Data

Visualizations make patterns obvious.

🔹 Using missingno:

python

import missingno as msno

msno.matrix(df)

msno.heatmap(df)

msno.bar(df)

• Matrix view shows null positions
• Heatmap shows correlation of nulls across columns
• Bar chart gives a quick total missing count per feature

🔹 Using Seaborn Heatmap:

python

import seaborn as sns

sns.heatmap(df.isnull(), cbar=False)

🔄 5. Row-Level and Segment Analysis

Some rows may have many missing values:

python

df['missing_per_row'] = df.isnull().sum(axis=1)

df['missing_percent_row'] = df.isnull().mean(axis=1) * 100

Segment Missing by Category:

python

df.groupby('Gender')['Age'].apply(lambda x: x.isnull().mean())

Helps identify if missingness is biased toward a group (MAR condition)

🧪 6. Finding Structured or Patterned Missingness

Look for relationships between missingness:

python

# Correlation between missing values

df_missing = df.isnull().astype(int)

df_missing.corr()

Visualize as heatmap:

python

sns.heatmap(df_missing.corr(), annot=True)

Example: Missing Income highly correlates with missing CreditScore.

🔁 7. Time Series & Index-Based Gaps

If your dataset has time-based or sequential data, missingness could be episodic:

python

df.set_index('Date', inplace=True)

df['Sales'].plot()

Look for:

• Missing timestamps
• Data holes during holidays or system downtime

Fill in timestamps:

python

df = df.asfreq('D')  # Daily frequency

🛠 8. Profiling Tools

🧰 Automated Profiling with pandas_profiling (now ydata-profiling):

python

from ydata_profiling import ProfileReport

profile = ProfileReport(df)

profile.to_notebook_iframe()

Includes missing value stats, charts, and correlations — all in one place.

🧰 Great Expectations (For data validation)

bash

great_expectations suite new

Helps enforce rules like “No more than 5% missing in Age”

📚 9. Sample Missingness Report Format