Chapters
Model Evaluation Techniques in ML
📘 Chapter 1: Evaluation for Classification Models – Accuracy Isn’t Enough
🎯 Objective
This chapter provides an in-depth look at the key
evaluation metrics for classification models. While accuracy is often the
default metric used by beginners, it can be dangerously misleading, especially
in imbalanced datasets. This chapter introduces better alternatives like
precision, recall, F1-score, and ROC AUC, and shows when and how to use each
effectively.
🔍 The Pitfall of Accuracy
Accuracy measures the percentage of correct predictions. It
is calculated as:
Accuracy=True Positives + True Negatives
/ Total Samples
Although intuitive, accuracy assumes balanced class
distributions. In real-world scenarios like fraud detection, spam
filtering, or rare disease diagnosis, accuracy can be misleading.
Example: In a dataset where 95% of emails are not
spam, a model that always predicts “not spam” will be 95% accurate but totally
useless.
🧱 Better Classification
Metrics
✅ 1. Confusion Matrix
A confusion matrix breaks down predictions into four
categories:
|
Predicted Positive |
Predicted Negative |
|
|
Actual Positive |
True Positive (TP) |
False Negative (FN) |
|
Actual Negative |
False
Positive (FP) |
True Negative
(TN) |
This structure forms the basis for all other metrics.
✅ 2. Precision
Precision=TP / (TP+FP)
Indicates how many predicted positives are actually correct.
Useful when false positives are costly — e.g., email marketing where you
don’t want to annoy users by misclassifying a good message as spam.
✅ 3. Recall (Sensitivity)
Recall=TP / (TP+FN)
Measures how many actual positives were correctly predicted.
Critical when false negatives are costly — like in medical screening
where missing a disease can be fatal.
✅ 4. F1 Score
This is the harmonic mean of precision and recall. F1 score
provides a balanced view when both precision and recall matter, such as
in fraud detection.
✅ 5. ROC Curve and AUC
- ROC
Curve (Receiver Operating Characteristic): Plots the true positive
rate (recall) against the false positive rate.
- AUC
(Area Under the Curve): The closer to 1.0, the better the model. AUC =
0.5 means random guessing.
✅ 6. Log Loss (Cross-Entropy)
Penalizes false classifications based on confidence
levels. If your model predicts the correct class with low confidence, it
still gets penalized.
✅ 7. Matthews Correlation
Coefficient (MCC)
Works even with imbalanced classes and takes into
account all confusion matrix values. A value of +1 indicates perfect
prediction, 0 means random, and -1 indicates complete disagreement.
🧠 When to Use What?
|
Metric |
When to Use |
|
Accuracy |
Balanced datasets |
|
Precision |
Costly false
positives |
|
Recall |
Costly false negatives |
|
F1 Score |
Need balance
of precision & recall |
|
ROC AUC |
General model
comparison, threshold tuning |
|
MCC |
Imbalanced
datasets |
🔁 Threshold Tuning
Classification models often output probabilities
rather than hard labels. You can set a custom decision threshold to
adjust the balance between precision and recall. For instance:
- Lowering
threshold increases recall but may reduce precision
- Raising
threshold increases precision but may lower recall
Tools like precision-recall curves help visualize
this trade-off.
🛠 Implementation Example
(Python)
python
from
sklearn.metrics import classification_report, confusion_matrix, roc_auc_score
print(confusion_matrix(y_true,
y_pred))
print(classification_report(y_true,
y_pred))
print("ROC
AUC:", roc_auc_score(y_true, y_proba))
📌 Summary Table
|
Metric |
Strength |
Weakness |
|
Accuracy |
Simple, intuitive |
Misleading on
imbalanced data |
|
Precision |
Controls
false positives |
Ignores false
negatives |
|
Recall |
Controls false
negatives |
Ignores false
positives |
|
F1 Score |
Balance of
precision and recall |
Hard to
interpret on its own |
|
ROC AUC |
Threshold-independent
performance |
Not actionable without
context |
|
MCC |
Balanced and
reliable metric |
Harder to
interpret intuitively |
FAQs
1. Why is model evaluation important in machine learning?
Model evaluation ensures that your model not only performs well on training data but also generalizes effectively to new, unseen data. It helps prevent overfitting and guides model selection.
2. What is the difference between training accuracy and test accuracy?
Training accuracy measures performance on the data used to train the model, while test accuracy evaluates how well the model generalizes to new data. High training accuracy but low test accuracy often indicates overfitting.
3. What is the purpose of a confusion matrix?
A confusion matrix summarizes prediction results for classification tasks. It breaks down true positives, true negatives, false positives, and false negatives, allowing detailed error analysis.
4. When should I use the F1 score over accuracy?
Use the F1 score when dealing with imbalanced datasets, where accuracy can be misleading. The F1 score balances precision and recall, offering a better sense of performance in such cases.
5. How does cross-validation improve model evaluation?
Cross-validation reduces variance in model evaluation by testing the model on multiple folds of the dataset. It provides a more reliable estimate of model performance than a single train/test split.
6. What is the ROC AUC score?
ROC AUC measures the model’s ability to distinguish between classes across different thresholds. A score closer to 1 indicates excellent discrimination, while 0.5 implies random guessing.
7. What’s the difference between MAE and RMSE in regression?
MAE calculates the average absolute errors, treating all errors equally. RMSE squares the errors, giving more weight to larger errors. RMSE is more sensitive to outliers.
8. Why is adjusted R² better than regular R²?
Adjusted R² accounts for the number of predictors in a model, making it more reliable when comparing models with different numbers of features. It penalizes unnecessary complexity.
9. What’s a good silhouette score?
A silhouette score close to 1 indicates well-separated clusters in unsupervised learning. Scores near 0 suggest overlapping clusters, and negative values imply poor clustering.
10. Can model evaluation metrics vary between domains?
Yes, different problems require different metrics. For example, in medical diagnosis, recall might be more critical than accuracy, while in financial forecasting, minimizing RMSE may be preferred.
Comments(0)