Top 10 Data Cleaning Techniques in Python: Master the Art of Preprocessing for Accurate Analysis

📘 Chapter 5: Handling Outliers in Python

Detect, Visualize, and Treat Extreme Values for Cleaner, Smarter Data

🧠 Introduction

Outliers are data points that differ significantly from the majority of observations. While they can be rare and sometimes valid, they often distort statistical analysis, mislead data modeling, and impact machine learning results. Whether you're forecasting revenue, analyzing customer behavior, or training a predictive model — outlier detection and treatment are essential steps in the data cleaning process.

In this chapter, you'll learn how to:

• Understand what outliers are and how they arise
• Detect outliers using statistical and visual methods
• Decide whether to remove, cap, or keep outliers
• Use Python (Pandas, NumPy, Matplotlib, Seaborn, and Scikit-learn) for practical outlier handling

🧩 What Are Outliers?

An outlier is a data point that lies an abnormal distance from other values in a dataset.

Example:
In a dataset of ages: [22, 25, 27, 30, 32, 35, 120], the value 120 is clearly an outlier.

🔍 Common Causes of Outliers

📌 Why Handle Outliers?

📊 Step 1: Detecting Outliers

Method 1: Using the Interquartile Range (IQR)

python

import pandas as pd

data = {'Salary': [30000, 32000, 31000, 30500, 700000, 31500, 34000]}

df = pd.DataFrame(data)

Q1 = df['Salary'].quantile(0.25)

Q3 = df['Salary'].quantile(0.75)

IQR = Q3 - Q1

lower_bound = Q1 - 1.5 * IQR

upper_bound = Q3 + 1.5 * IQR

outliers = df[(df['Salary'] < lower_bound) | (df['Salary'] > upper_bound)]

print(outliers)

Method 2: Using Z-Score

Z-score measures how far a point is from the mean in terms of standard deviations.

python

from scipy import stats

import numpy as np

df['z_score'] = stats.zscore(df['Salary'])

print(df[df['z_score'].abs() > 3])

Method 3: Visual Methods (Boxplot, Scatterplot)

python

import matplotlib.pyplot as plt

import seaborn as sns

sns.boxplot(x=df['Salary'])

plt.title("Boxplot of Salary")

plt.show()

Boxplots show outliers as individual points outside whiskers.

🛠 Step 2: Treating Outliers

Method 1: Removing Outliers

python

df_cleaned = df[(df['Salary'] >= lower_bound) & (df['Salary'] <= upper_bound)]

Use case: When outliers are errors or irrelevant to the analysis.

Method 2: Capping (Winsorization)

Limit extreme values to a threshold.

python

cap_upper = df['Salary'].quantile(0.95)

cap_lower = df['Salary'].quantile(0.05)

df['Salary_capped'] = df['Salary'].clip(lower=cap_lower, upper=cap_upper)

Use case: When you want to reduce the influence without removing data.

Method 3: Transformation (Log, Square Root)

Reduces impact of extreme values.

python

df['log_salary'] = np.log1p(df['Salary'])  # log(1 + x)

Use when the data is heavily right-skewed (e.g., income, prices).

Method 4: Imputation with Mean/Median

Replace outlier with mean or median of non-outlier data.

python

median_salary = df[(df['Salary'] >= lower_bound) & (df['Salary'] <= upper_bound)]['Salary'].median()

df['Salary'] = np.where((df['Salary'] > upper_bound) | (df['Salary'] < lower_bound),

                        median_salary,

                        df['Salary'])

Method 5: Isolation Forest (for high-dimensional datasets)

python

from sklearn.ensemble import IsolationForest

iso = IsolationForest(contamination=0.1)

df['outlier'] = iso.fit_predict(df[['Salary']])

df_outliers = df[df['outlier'] == -1]

📊 Summary Table: Outlier Detection & Handling Techniques

🧠 Best Practices

📉 Example Before & After

📦 Bonus: Outliers in Multivariate Data

Sometimes a value isn’t an outlier in isolation but is strange in relation to others.

Example:
A student with a GPA of 4.0 but zero attendance.

Use:

• Isolation Forest
• DBSCAN
• Mahalanobis Distance

🏁 Conclusion

Outliers can either be noise or insight — the key is knowing the difference. With Python’s robust tools, you can efficiently identify and treat outliers using statistical, visual, or algorithmic methods. Mastering outlier detection ensures more accurate models, clearer insights, and higher data quality across your projects.