Handling Missing Data Like a Pro: Smart Strategies Every Data Scientist Should Know

📗 Chapter 3: Assessing the Impact of Missing Data

Quantify, Understand, and Strategize Before You Clean

🧠 Introduction

Before you impute, drop, or modify any missing data, ask yourself:
“What’s the cost of this missing data to my model, insights, or business decision?”

Treating all missing data equally is like treating all medicine the same — context, dosage, and timing matter.

In this chapter, we’ll explore:

• How missing data affects analysis and models
• Quantifying the severity of missingness
• Assessing bias, distortion, and data loss
• Real-world examples and Python code
• Visual and statistical tools to evaluate impact

📊 1. Why Assessing Missingness Matters

Missing data affects:

• Data quality and validity
• Model accuracy and fairness
• Statistical significance
• Business trust and interpretability

Ignoring the impact of missing data can lead to:

• Biased models
• Misleading insights
• Data leakage
• Legal/compliance risk (e.g., biased models in healthcare)

🧩 2. Quantifying Missingness Across Columns

Start by profiling overall impact:

python

missing = df.isnull().sum()

percent = df.isnull().mean() * 100

missing_df = pd.DataFrame({'Missing Count': missing, 'Percent': percent})

Visualize using a bar chart:

python

missing_df[missing_df['Percent'] > 0].sort_values('Percent').plot.barh()

Table Example:

🔍 3. Assessing Statistical Bias in Missingness

Compare the distribution of missing vs. non-missing groups.

Example: Compare income between those with and without Age

python

df['age_missing'] = df['Age'].isnull()

df.groupby('age_missing')['Income'].describe()

If large differences exist, missingness may be not at random (MNAR) or at random but biased (MAR).

Chi-Square for Categorical Variables:

python

from scipy.stats import chi2_contingency

# Gender vs. Age missingness

pd.crosstab(df['Gender'], df['Age'].isnull()).pipe(chi2_contingency)

T-Test for Numerical Variables:

python

from scipy.stats import ttest_ind

missing_age = df[df['Age'].isnull()]

non_missing_age = df[df['Age'].notnull()]

ttest_ind(missing_age['Income'], non_missing_age['Income'], nan_policy='omit')

A significant p-value suggests statistical impact of missingness.

🧪 4. Impact on Correlations and EDA

Missing values distort correlation and relationships.

Example: Compare pairwise correlation before and after dropping missing

python

corr1 = df.corr()

corr2 = df.dropna().corr()

diff = corr1 - corr2

diff.abs().style.background_gradient(cmap='coolwarm')

Impact on Visual Trends

python

import seaborn as sns

sns.scatterplot(x='Age', y='Income', data=df)  # With missing

sns.scatterplot(x='Age', y='Income', data=df.dropna())  # Cleaned

Visual inspection reveals how missingness alters patterns.

📉 5. Modeling the Impact: Train With and Without Missing Columns

Try training two models:

1. On the raw data with missing columns dropped
2. On the same data with those columns imputed

Example with Random Forest:

python

from sklearn.ensemble import RandomForestClassifier

from sklearn.model_selection import train_test_split

from sklearn.metrics import accuracy_score

from sklearn.impute import SimpleImputer

X = df.drop('target', axis=1)

y = df['target']

# Drop rows with missing data

model1 = RandomForestClassifier()

model1.fit(X.dropna(), y[X.dropna().index])

# Impute

imputer = SimpleImputer(strategy='median')

X_imputed = pd.DataFrame(imputer.fit_transform(X), columns=X.columns)

X_train, X_test, y_train, y_test = train_test_split(X_imputed, y, test_size=0.2)

model2 = RandomForestClassifier()

model2.fit(X_train, y_train)

# Compare

print("Drop Accuracy:", model1.score(X.dropna(), y[X.dropna().index]))

print("Impute Accuracy:", model2.score(X_test, y_test))

Output Example:

🧠 6. Visualizing Impact with Drift Analysis

Use EvidentlyAI to visualize feature drift post-missing handling.

python

from evidently.report import Report

from evidently.metric_preset import DataDriftPreset

report = Report(metrics=[DataDriftPreset()])

report.run(reference_data=reference_df, current_data=df)

report.show()

💾 7. Business-Level Impact Assessment

Missingness affects decisions in:

• Marketing: Incomplete user profiles lead to poor segmentation
• Finance: Missing credit history can inflate or deflate risk
• Healthcare: Missing vitals may bias patient scoring

Solution: Assign a dollar/risk score to each percentage of data loss. Use cost-benefit tables.

🧮 8. Cost-Benefit Matrix for Handling Strategies

📋 9. Final Impact Evaluation Checklist