How to Set Up CI/CD on AWS

📘 Chapter 5: Monitoring, Notifications, and Cost Optimization

🔍 Overview

Deploying a CI/CD pipeline on AWS is only part of the job. Ensuring it's reliable, observable, and cost-effective is just as critical. AWS provides robust tools for monitoring pipeline performance, setting up automated notifications, and analyzing usage to reduce unnecessary expenses.

In this chapter, we’ll cover:

• Monitoring your CI/CD pipeline across AWS services
• Setting up real-time notifications
• Debugging and tracing with CloudWatch and CloudTrail
• Estimating and optimizing CI/CD costs
• Best practices for sustainable pipeline management

📡 1. Monitoring Your CI/CD Pipeline

✅ Key Tools for Observability

🧭 CloudWatch Metrics for CodePipeline

CloudWatch provides predefined metrics for:

• PipelineExecutionTime
• PipelineExecutionSucceeded
• PipelineExecutionFailed
• ActionExecutionFailed

You can create dashboards or alarms using these metrics.

🧪 Sample: Create a CloudWatch Alarm

bash

aws cloudwatch put-metric-alarm \

  --alarm-name "PipelineFailureAlarm" \

  --metric-name PipelineExecutionFailed \

  --namespace AWS/CodePipeline \

  --statistic Sum \

  --period 300 \

  --threshold 1 \

  --comparison-operator GreaterThanOrEqualToThreshold \

  --evaluation-periods 1 \

  --alarm-actions arn:aws:sns:us-east-1:123456789012:NotifyMe

🔔 2. Real-Time Notifications with SNS

You can use Amazon SNS (Simple Notification Service) to send alerts via:

• Email
• SMS
• Lambda functions
• HTTP/HTTPS endpoints

✅ Creating an SNS Topic

bash

aws sns create-topic --name PipelineAlerts

✅ Subscribe to the Topic

bash

aws sns subscribe \

  --topic-arn arn:aws:sns:us-east-1:123456789012:PipelineAlerts \

  --protocol email \

  --notification-endpoint you@example.com

You’ll receive a confirmation email. Approve it to start receiving notifications.

✅ Integrate SNS with CodePipeline

In the CodePipeline console:

• Go to Edit Pipeline → Advanced
• Add notification rules to trigger on Execution failed, Succeeded, etc.

Or use:

bash

aws codestar-notifications create-notification-rule \

  --name PipelineFailureNotify \

  --resource arn:aws:codepipeline:us-east-1:123456789012:MyAppPipeline \

  --event-type-ids codepipeline-pipeline-execution-failed \

  --targets TargetType=SNS,TargetAddress=arn:aws:sns:us-east-1:123456789012:PipelineAlerts \

  --detail-type FULL \

  --status ENABLED

🛠️ 3. Debugging and Troubleshooting CI/CD Failures

✅ CodePipeline Errors

✅ Viewing Logs in CloudWatch

CodeBuild Logs

bash

aws logs get-log-events \

  --log-group-name /aws/codebuild/MyBuildProject \

  --log-stream-name $(your-log-stream-name)

CodeDeploy Logs (on EC2)

bash

cat /opt/codedeploy-agent/deployment-root/deployment-logs/codedeploy-agent-deployments.log

🛑 Common CloudWatch Metrics for Debugging

💰 4. Cost Management for CI/CD

✅ Cost Components by Service

💡 Tips to Reduce Cost

• Use smaller build environments (BUILD_GENERAL1_SMALL)
• Enable build caching to reduce redundant installs
• Use retention policies for logs in CloudWatch
• Schedule builds only on commit, not time-based
• Clean up unused pipelines, build projects, and S3 artifacts

🧾 Create Budget Alerts (Billing Guardrails)

bash

aws budgets create-budget \

  --account-id 123456789012 \

  --budget file://budget.json

budget.json:

json

{

  "BudgetName": "CICDBudget",

  "BudgetLimit": {

    "Amount": "50.0",

    "Unit": "USD"

  },

  "TimeUnit": "MONTHLY",

  "BudgetType": "COST"

}

🧠 5. Best Practices for Observability & Cost Control

• Enable detailed CloudTrail logs for auditing
• Use AWS Config to track resource changes
• Use resource tags to group costs per environment or app
• Review Cost Explorer monthly
• Consolidate builds and environments when possible

📋 Summary Table – Monitoring & Cost Optimization