Data Science Workflow: From Problem to Solution – A Complete Step-by-Step Journey for Beginners

📗 Chapter 4: Exploratory Data Analysis (EDA)

Uncovering Patterns, Trends, and Insights Before Modeling

🧠 Introduction

Before jumping into model building, a data scientist must first explore the data — not just technically, but intellectually and visually.

Exploratory Data Analysis (EDA) is the process of analyzing datasets to summarize their main characteristics, detect anomalies, and discover patterns using statistics and visualization.

EDA helps answer questions like:

• What are the distributions of key features?
• Are there any outliers?
• How do features relate to one another?
• Do we need feature transformation or binning?

This chapter will teach you how to:

• Perform univariate, bivariate, and multivariate analysis
• Visualize relationships using Python
• Understand feature correlations
• Prepare insights for stakeholders and modeling

📂 1. What is EDA?

Exploratory Data Analysis is the process of:

• Profiling your data: Types, distributions, missingness
• Visualizing relationships: Target variable vs. features
• Generating hypotheses: What matters for prediction?
• Identifying issues: Imbalances, collinearity, anomalies

It’s non-linear and iterative — you may return to data cleaning or feature engineering as you uncover insights.

🔍 2. Overview of the Dataset

We’ll use the Titanic dataset as an example.

python

import pandas as pd

df = pd.read_csv('titanic.csv')

df.head()

Basic Info

python

df.info()

df.describe()

df.isnull().sum()

📊 3. Univariate Analysis

Analysis of individual features.

▶ Numerical Features

Histogram + KDE plot:

python

import seaborn as sns

import matplotlib.pyplot as plt

sns.histplot(df['Age'], kde=True)

plt.title('Age Distribution')

Boxplot for outliers:

python

sns.boxplot(x=df['Age'])

Summary stats:

python

df['Fare'].describe()

▶ Categorical Features

Bar chart for counts:

python

sns.countplot(x='Sex', data=df)

Value counts:

python

df['Embarked'].value_counts(normalize=True)

🔗 4. Bivariate Analysis

Explore how one feature affects another, especially with respect to the target variable.

▶ Categorical vs Target

python

sns.barplot(x='Sex', y='Survived', data=df)

▶ Numerical vs Target

python

sns.boxplot(x='Survived', y='Fare', data=df)

Groupby statistics:

python

df.groupby('Pclass')['Survived'].mean()

🔄 5. Multivariate Analysis

Analyzing interactions between multiple variables.

▶ Pairplot

python

sns.pairplot(df[['Age', 'Fare', 'Survived']], hue='Survived')

▶ Heatmap of Correlation Matrix

python

corr = df.corr()

sns.heatmap(corr, annot=True, cmap='coolwarm')

What to look for:

• Strong linear relationships
• Redundant features (correlation > 0.85)
• Potential predictors of target variable

📈 6. Feature vs. Feature

Sometimes you need to explore relationships between features.

python

sns.scatterplot(x='Age', y='Fare', hue='Survived', data=df)

▶ Grouped boxplot:

python

sns.boxplot(x='Pclass', y='Age', hue='Survived', data=df)

🧪 7. Target Variable Analysis

Understanding your target variable is crucial.

python

df['Survived'].value_counts(normalize=True).plot(kind='bar')

If imbalanced, consider:

• Using stratified splits
• Adjusting evaluation metrics
• Using SMOTE or under/oversampling later

📊 8. Dealing with Skewed Features

python

from scipy.stats import skew

skewness = df['Fare'].skew()

print("Skewness:", skewness)

Apply transformation:

python

df['Fare_log'] = np.log1p(df['Fare'])

sns.histplot(df['Fare_log'], kde=True)

🧠 9. Questions to Ask During EDA

✅ 10. Summary Table Example

Here’s a snapshot you might prepare from EDA:

🧪 EDA Summary Example for Titanic:

• Age is right-skewed, with some outliers.
• Sex has a strong relationship with survival.
• Pclass is inversely related to survival.
• Fare varies widely and benefits from log transformation.
• Embarked has minor missingness; mode imputation is reasonable.
• Class imbalance exists in the Survived variable.