Deploying Containers with Kubernetes

✅ Chapter 5: Monitoring, Troubleshooting, and Best Practices

🔍 Introduction

Kubernetes is powerful, but with great power comes complexity. Once your applications are deployed, monitoring their health and troubleshooting issues efficiently becomes critical. Additionally, following best practices ensures your clusters remain secure, stable, and scalable.

In this chapter, you’ll learn:

• Key methods to monitor Kubernetes clusters and workloads
• Essential troubleshooting techniques
• Best practices for production-grade Kubernetes deployments

By mastering these skills, you’ll be able to proactively maintain and secure your Kubernetes environments.

📋 Part 1: Monitoring Kubernetes Clusters and Workloads

Monitoring is not just optional—it's essential. Kubernetes environments are dynamic, and things can change rapidly.

📊 What to Monitor in Kubernetes?

🔹 Key Monitoring Tools

📋 Installing Metrics Server (for kubectl top)

bash

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Check node and pod metrics:

bash

kubectl top nodes

kubectl top pods

📈 Setting Up Prometheus and Grafana

• Prometheus scrapes metrics from Kubernetes objects
• Grafana visualizes them

Deploy using Helm:

bash

helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack

Access Grafana dashboard:

bash

kubectl port-forward svc/kube-prometheus-stack-grafana 3000:80

Visit http://localhost:3000.

Default credentials:

• Username: admin
• Password: prom-operator

🧩 Important Metrics to Monitor

🛠️ Part 2: Troubleshooting Kubernetes Applications

Things go wrong — here’s how you methodically troubleshoot.

🔎 Checking Pod Status

bash

kubectl get pods

Look for unusual statuses like CrashLoopBackOff, Error, or Pending.

📜 Inspect Pod Events

bash

kubectl describe pod <pod-name>

• Events section shows scheduling issues, image pull errors, OOM kills, etc.

🛠️ Viewing Pod Logs

bash

kubectl logs <pod-name>

For multi-container Pods:

bash

kubectl logs <pod-name> -c <container-name>

Tail logs:

bash

kubectl logs -f <pod-name>

🛡️ Executing Commands Inside a Pod

bash

kubectl exec -it <pod-name> -- /bin/sh

Useful for debugging container filesystem, environment, or manual probes.

🚥 Troubleshooting Deployments

Check rollout status:

bash

kubectl rollout status deployment <deployment-name>

Undo rollout:

bash

kubectl rollout undo deployment <deployment-name>

View revision history:

bash

kubectl rollout history deployment <deployment-name>

🔥 Common Problems and How to Fix Them

📦 Part 3: Kubernetes Best Practices for Stability, Security, and Scalability

Building on a strong foundation ensures long-term success.

✅ General Best Practices

• Use Namespaces to separate environments (e.g., dev, staging, prod).
• Tag resources properly with labels and annotations.
• Prefer Deployments over manual Pod creation.
• Always set resource requests and limits.
• Use rolling updates for zero downtime deployments.

✅ Application Best Practices

✅ Security Best Practices

• Use RBAC (Role-Based Access Control) for users and services.
• Limit access to cluster components and APIs.
• Use PodSecurityPolicies or OPA Gatekeeper for workload restrictions.
• Scan images regularly for vulnerabilities (e.g., Trivy, Clair).
• Encrypt Secrets at rest.

✅ Resource Management Best Practices

• Always define requests and limits in Pod specs:

yaml

resources:

  requests:

    cpu: "100m"

    memory: "256Mi"

  limits:

    cpu: "500m"

    memory: "512Mi"

• Use Horizontal Pod Autoscaler (HPA):

bash

kubectl autoscale deployment my-app --cpu-percent=50 --min=2 --max=5

✅ Networking Best Practices

• Use NetworkPolicies to restrict traffic between Pods.
• Prefer Ingress Controllers over exposing NodePorts for public apps.
• Regularly audit network traffic patterns.

✅ Observability Best Practices

• Centralize logging with ELK or Loki stack.
• Use Prometheus and Grafana for proactive monitoring.
• Set up alerts for critical conditions like high CPU, node disk fill, etc.

🚀 Summary: What You Learned in Chapter 5

• Kubernetes needs continuous monitoring at cluster, node, and Pod levels
• Effective troubleshooting involves examining pod status, events, and logs
• Production-ready clusters follow strict best practices on configuration, security, and resource management
• Proactive monitoring and alerting systems are essential to maintain uptime and reliability
• Kubernetes is not “set and forget”—it demands lifecycle management and operational excellence