Building Your First Data Science Project: A Beginner's Step-by-Step Guide to Turn Raw Data into Real Insights

📗 Chapter 5: Exploratory Data Analysis (EDA)

Uncover Hidden Stories, Trends, and Patterns in Your Data

🧠 Introduction

After loading and cleaning your dataset, it's time to explore it deeply. This is where Exploratory Data Analysis (EDA) comes in.

EDA is the detective work in data science. It helps you:

• Understand distributions and relationships
• Detect patterns, anomalies, and outliers
• Generate hypotheses
• Select features and model strategies

In short, EDA builds your intuition about the data, which is essential before modeling.

In this chapter, you’ll learn:

• How to analyze numerical and categorical data
• Visual tools for insight
• Correlation analysis
• Techniques for identifying trends and outliers
• How to use Python tools like Pandas, Matplotlib, and Seaborn for EDA

🔍 1. Define the Goal of EDA

Before diving in, ask:

• What is the target variable (if any)?
• Are you trying to predict or explain?
• What types of variables are present?
• What business or research questions should guide the analysis?

📦 2. Understand Variable Types

Check variable types:

python

df.dtypes

📈 3. Univariate Analysis (One Variable at a Time)

🔹 Numerical Variables

▶ Histogram & Density Plot

python

import matplotlib.pyplot as plt

import seaborn as sns

sns.histplot(df['Age'], kde=True)

plt.title('Age Distribution')

plt.show()

▶ Boxplot

python

sns.boxplot(x=df['Age'])

Use .describe() for quick summary:

python

df['Age'].describe()

🔹 Categorical Variables

python

df['Gender'].value_counts().plot(kind='bar')

plt.title('Gender Count')

plt.xticks(rotation=0)

plt.show()

Use .value_counts():

python

df['Gender'].value_counts(normalize=True)

🔗 4. Bivariate Analysis (Two Variables)

Explore relationships between:

• Numeric vs Numeric
• Categorical vs Numeric
• Categorical vs Categorical

🔹 Numeric vs Numeric: Scatter Plot

python

sns.scatterplot(x='Age', y='Income', data=df)

🔹 Categorical vs Numeric: Boxplot

python

sns.boxplot(x='Gender', y='Income', data=df)

🔹 Categorical vs Categorical: Cross Tab

python

pd.crosstab(df['Gender'], df['Survived'], normalize='index')

🔁 5. Multivariate Analysis (Three or More Variables)

▶ Pairplot

python

sns.pairplot(df[['Age', 'Fare', 'Survived']], hue='Survived')

▶ Grouped Summary

python

df.groupby(['Pclass', 'Sex'])['Survived'].mean().unstack()

📊 6. Correlation Analysis

Use .corr() to analyze numeric relationships:

python

corr_matrix = df.corr()

sns.heatmap(corr_matrix, annot=True, cmap='coolwarm')

plt.title('Correlation Matrix')

plt.show()

⚠️ Note:

• Correlation ≠ Causation
• Strong correlations help in feature selection

🔍 7. Outlier Detection

Use boxplots and IQR method.

python

sns.boxplot(x='Fare')

python

Q1 = df['Fare'].quantile(0.25)

Q3 = df['Fare'].quantile(0.75)

IQR = Q3 - Q1

lower_bound = Q1 - 1.5 * IQR

upper_bound = Q3 + 1.5 * IQR

df_outliers = df[(df['Fare'] < lower_bound) | (df['Fare'] > upper_bound)]

📅 8. Time Series Analysis (if applicable)

If you have a datetime column:

python

df['Date'] = pd.to_datetime(df['Date'])

df.set_index('Date', inplace=True)

df['Sales'].resample('M').sum().plot()

🧠 9. Ask Analytical Questions

🧪 10. Tools Summary Table

✅ Full Example: Titanic Dataset EDA

python

import pandas as pd

import seaborn as sns

import matplotlib.pyplot as plt

df = pd.read_csv('titanic.csv')

# Overview

print(df.shape)

print(df.info())

# Univariate: Age

sns.histplot(df['Age'].dropna(), kde=True)

plt.title('Age Distribution')

plt.show()

# Bivariate: Gender vs Survival

sns.barplot(x='Sex', y='Survived', data=df)

# Multivariate: Pclass & Gender vs Survival

df.groupby(['Pclass', 'Sex'])['Survived'].mean().unstack().plot(kind='bar')

# Correlation

sns.heatmap(df.corr(), annot=True, cmap='YlGnBu')

# Outliers

sns.boxplot(x='Fare', data=df)