Model Evaluation Techniques in ML

📕 Chapter 4: Evaluation in Imbalanced and Noisy Datasets

🎯 Objective

This chapter focuses on evaluating machine learning models in challenging conditions — specifically imbalanced datasets (where one class dominates) and noisy datasets (with corrupted or mislabeled data). These scenarios are common in real-world applications like fraud detection, medical diagnoses, and anomaly detection.

Standard evaluation metrics like accuracy fail in these cases, so this chapter explores robust strategies and metrics designed for imbalanced and noisy data.

🔍 Understanding Imbalanced Datasets

In an imbalanced dataset, the majority class heavily outweighs the minority class. A model that predicts everything as the majority class could still achieve high accuracy — but be useless.

Example: In a dataset where only 1% of transactions are fraud, a model predicting “not fraud” for everything will be 99% accurate — but completely ineffective.

⚠️ Problems with Standard Accuracy

• Hides poor minority class performance
• Encourages bias toward majority class
• Doesn’t differentiate error severity

✅ Better Metrics for Imbalanced Datasets

1. Precision, Recall, and F1 Score

• Precision evaluates how many predicted positives are actually correct.
• Recall measures how many actual positives the model found.
• F1 Score balances the two.

These metrics help in fraud detection, disease prediction, and rare event classification.

2. Confusion Matrix Insights

Confusion matrices become even more valuable for imbalanced data. Focus on:

• False negatives: Missed actual positives
• False positives: Incorrect alerts

3. Precision-Recall (PR) Curve

More informative than the ROC curve in imbalanced settings. The PR curve plots precision vs recall, showing performance across various thresholds.

4. ROC Curve and AUC

While the ROC curve still works, it can be misleading in imbalanced data. The AUC should be interpreted with caution — always compare it with PR AUC.

5. G-Mean and Balanced Accuracy

These metrics consider the balance between sensitivity (recall for positive class) and specificity (recall for negative class).

🧪 Sampling Techniques

✅ Oversampling the Minority Class

SMOTE (Synthetic Minority Oversampling Technique) creates synthetic examples of the minority class. This boosts recall but can cause overfitting.

✅ Undersampling the Majority Class

Removes samples from the majority class to rebalance the dataset. It helps with training speed but may discard valuable data.

✅ Combined Sampling

Uses both over- and under-sampling to balance class distribution.

🤖 Ensemble Methods for Imbalanced Data

• Random Forest with Class Weights: Adjust class weights to penalize majority class.
• XGBoost: Supports imbalanced class weights with built-in hyperparameters.
• BalancedBaggingClassifier: Applies bootstrapping with balance-aware sampling.

🧠 Evaluating Noisy Datasets

Noise refers to irrelevant, mislabeled, or inconsistent data. Label noise is especially harmful in supervised learning.

Types of Noise

• Attribute noise: Incorrect or distorted input features
• Label noise: Incorrect class assignments

🧼 Strategies to Handle Noisy Data

✅ 1. Robust Evaluation Metrics

Use metrics less sensitive to outliers, like MAE in regression or median absolute error.

✅ 2. Noise Detection and Filtering

Apply noise filtering methods like:

• k-NN label filtering
• Consensus voting from ensemble models
• Rule-based filters

✅ 3. Data Cleaning with Domain Knowledge

Leverage expert input to flag or remove suspicious records, especially in high-stakes fields like healthcare.

✅ 4. Use of Robust Models

Algorithms like Random Forest, Gradient Boosting, or RANSAC Regression are more resilient to noise.

📊 Comparison Table Summary

✅ Tips and Best Practices

• Use stratified sampling during cross-validation to preserve class distribution
• Always compare ROC AUC with PR AUC in imbalanced classification
• Prefer F1 score over accuracy when classes are imbalanced
• Use log transformations to minimize noise in skewed numeric data
• Visualize decision boundaries and residuals to detect noise and misclassification