Creating Scalable Applications with Kubernetes

📒 Chapter 4: Monitoring, Logging & Observability at Scale in Kubernetes

🌐 Introduction

Modern Kubernetes applications are complex, distributed, and dynamic. At scale, traditional monitoring tools struggle to capture the real-time health, performance metrics, and debugging data you need.

That’s where observability comes in — a deeper, structured approach to understanding what’s happening inside your applications and infrastructure.

In this chapter, we’ll explore:

• Core principles of observability in Kubernetes
• Monitoring tools like Prometheus, Grafana, kube-state-metrics
• Centralized logging with EFK/Loki stacks
• Tracing with Jaeger and OpenTelemetry
• Alerting, dashboarding, and scaling observability for production

🔍 Section 1: Observability vs. Monitoring

Kubernetes observability means going beyond metrics — it means integrating logs, events, traces, and alerts to achieve full operational visibility.

📊 Section 2: Monitoring with Prometheus & kube-state-metrics

✅ What is Prometheus?

Prometheus is a pull-based metrics collection system. It scrapes HTTP endpoints that expose metrics in a specific format.

🛠️ Prometheus Operator (recommended method)

Install with:

bash

kubectl apply -f https://github.com/prometheus-operator/prometheus-operator/blob/main/bundle.yaml

Or use Helm:

bash

helm install prometheus prometheus-community/kube-prometheus-stack

This installs:

• Prometheus server
• Node exporter
• kube-state-metrics
• Grafana (optional)
• Alertmanager

🔧 Sample Prometheus scrape config (for a custom app)

yaml

apiVersion: monitoring.coreos.com/v1

kind: ServiceMonitor

metadata:

  name: my-app-monitor

spec:

  selector:

    matchLabels:

      app: my-app

  endpoints:

    - port: web

      interval: 30s

📊 Common Metrics Tracked

📈 Section 3: Grafana Dashboards

Grafana is the visualization layer in your monitoring stack.

✅ Built-in dashboards for:

• Cluster health
• Node CPU/memory
• Pod health and restart counts
• Network/ingress traffic

Access Grafana:

bash

kubectl port-forward svc/prometheus-grafana 3000:80

Default credentials: admin / prom-operator

📝 Section 4: Logging with EFK Stack or Loki

🔹 EFK: Elasticsearch, Fluent Bit/Fluentd, Kibana

🔹 Loki: Lightweight Alternative to Elasticsearch

• Developed by Grafana Labs
• Integrates natively with Grafana
• Uses labels for log indexing (not full-text search)

🛠️ Fluent Bit DaemonSet Config (example)

yaml

apiVersion: v1

kind: ConfigMap

metadata:

  name: fluent-bit-config

data:

  fluent-bit.conf: |

    [SERVICE]

        Flush        5

        Daemon       Off

    [INPUT]

        Name         tail

        Path         /var/log/containers/*.log

        Tag          kube.*

        Parser       docker

    [OUTPUT]

        Name         stdout

        Match        *

Apply:

bash

kubectl apply -f fluent-bit-config.yaml

🧵 Section 5: Tracing with Jaeger & OpenTelemetry

Traces allow you to follow a single request across multiple services and pods.

✅ Tools:

• Jaeger (open source distributed tracing system)
• OpenTelemetry SDK (instrument your code)

🛠️ Sample Jaeger Deployment (All-in-One)

bash

kubectl create namespace observability

kubectl apply -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.44.0/jaeger-operator.yaml

Instrument code (Python example):

python

from opentelemetry import trace

from opentelemetry.exporter.jaeger.thrift import JaegerExporter

🚨 Section 6: Alerting & Notifications

Prometheus + Alertmanager supports routing alerts to:

• Email
• Slack
• PagerDuty
• Opsgenie
• Webhooks

🔔 Sample Alert Rule (CPU Usage)

yaml

groups:

  - name: cpu-alerts

    rules:

      - alert: HighPodCPU

        expr: rate(container_cpu_usage_seconds_total[1m]) > 0.9

        for: 2m

        labels:

          severity: warning

        annotations:

          summary: "Pod CPU usage high"

🧠 Section 7: Observability Best Practices

✅ Summary

Scalable Kubernetes systems demand more than uptime monitoring. You need rich telemetry, actionable alerts, and real-time visibility.

Key takeaways:

• Use Prometheus + Grafana for full-stack metrics
• Centralize logs with Fluent Bit and Elasticsearch or Loki
• Trace requests with Jaeger or OpenTelemetry
• Route alerts through Alertmanager
• Build observability into your CI/CD pipeline