How To Configure AWS CloudWatch Agent To Monitor System Level Metrics

Posted By : Mohd Adnan | 03-May-2018
AWS CloudWatch
Image Credits: amazon.com
 
Amazon Web Services has inbuilt functionality to good metrics on the console by default, like CPU, but it’s missing some key metrics that can cause bottlenecks like memory usage or disk space; these are essential to monitor to ensure instance uptime and health.
 
The monitoring scripts are written by Amazon themselves, but aren’t included unless you set them up yourself, which isn’t always obvious. The scripts are available for different operating systems, however we will focus on Linux-based systems in this post. Amazon's documentation on CloudWatch Agent implementation, while comprehensive, is hard to find; hopefully this post will help you with your own instance monitoring. 
 
Note: This post will be helpful for developers/devOps who would like to configure CloudWatch for disk and memory monitoring as the previous perl scripts provided by AWS is deprecated and not recommended and there is no other sources which mentions the steps clearly.
 
AWS has how-to instructions on creating memory, disk and swap checks via Perl scripts located at https://aws.amazon.com/code/8720044071969977. These scripts are customised and need cron jobs to be configured to use designated intervals, every 5 minutes.
 
Amazon introduced AWS CloudWatchAgent to automate this:
 
Here is a few important list of the available metrics within the manual monitoring scripts:
 
  • Memory Utilization (%)
  • Memory Used (MB)
  • Memory Available (MB)
  • Swap Utilization (%)
  • Swap Used (MB)
  • Disk Space Utilization (%)
  • Disk Space Used (GB)
  • Disk Space Available (GB)
 
Installation Process Overview
 
The general flow of installing the CloudWatch agent is as follows:
 
1. Create and Configure the IAM roles and users for the CloudWatch agent.
 
2. If you are installing on an EC2 instance, attach an IAM role to the instance. Else, If you are installing on an on-premises server, you need to create an IAM user to enable the CloudWatch agent to write information to CloudWatch.
 
3. Downloading the CloudWatch agent package, using either AWS Systems Manager Run Command or a public Amazon S3 download link.
 
4. Modify the CloudWatch agent configuration files as per our requirement.
 
5. Initiate the agent, using from the command line.
 
 
How to Simplify Memory and Disk Utilization Monitoring in AWS
 
1. Create an IAM Role and attach policy 'CloudWatchAgentServerPolicy'
2. Attach the instance you want to monitor to the created role from EC2 instance console. 
Select Instance --> Actions --> Instance Settings --> Attach/Replace IAM Role 
3. SSH to your EC2 instance and follow the steps
 
Downloading the CloudWatch Agent Package on an EC2 Instance
Make a directory for downloading the AmazonCloudWatchAgent and then navigate to that directory
Download the CloudWatch agent. 
 
wget https://s3.amazonaws.com/amazoncloudwatch-agent/linux/amd64/latest/AmazonCloudWatchAgent.zip
 
Unzip the package
 unzip AmazonCloudWatchAgent.zip
 
Install the package
 sudo ./install.sh
 
Open the CloudWatch Wizard and choose options
 sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard
 
The config file generated will be at:
 /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json
 
This file can be manually edited according to your requirements
 
Start the CloudWatch
 sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a start
 
CloudWatch Status can be checked at:
 sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status
You'll see following status if CloudWatch is running
 
{
  "status": "running",
  "starttime": "2018-05-03T11:03:30+0000",
  "version": "1.200442.0"
}

 
 
If required, CloudWatch monitoring can be stopped
 sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a stop
 
CloudWatch logs will be generated at this path:
 /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log
 
CloudWatch other paths are defined at this URL:
 
Here is an example of standard configuration file to monitor cpu, disk, memory & swap
 
 

{
	"metrics": {
		"append_dimensions": {
			"AutoScalingGroupName": "${aws:AutoScalingGroupName}",
			"ImageId": "${aws:ImageId}",
			"InstanceId": "${aws:InstanceId}",
			"InstanceType": "${aws:InstanceType}"
		},
		"metrics_collected": {
			"cpu": {
				"measurement": [
					"cpu_usage_idle",
					"cpu_usage_iowait",
					"cpu_usage_user",
					"cpu_usage_system"
				],
				"metrics_collection_interval": 60,
				"totalcpu": false
			},
			"disk": {
				"measurement": [
					"used_percent",
					"inodes_free"
				],
				"metrics_collection_interval": 60,
				"resources": [
					"*"
				]
			},
			"diskio": {
				"measurement": [
					"io_time"
				],
				"metrics_collection_interval": 60,
				"resources": [
					"*"
				]
			},
			"mem": {
				"measurement": [
					"mem_used_percent"
				],
				"metrics_collection_interval": 60
			},
			"swap": {
				"measurement": [
					"swap_used_percent"
				],
				"metrics_collection_interval": 60
			}
		}
	}
}


About Author

Author Image
Mohd Adnan

Adnan, an experienced Backend Developer, boasts a robust expertise spanning multiple technologies, prominently Java. He possesses an extensive grasp of cutting-edge technologies and boasts hands-on proficiency in Core Java, Spring Boot, Hibernate, Apache Kafka messaging queue, Redis, as well as relational databases such as MySQL and PostgreSQL. Adnan consistently delivers invaluable contributions to a variety of client projects, including Vision360 (UK) - Konfer, Bitsclan, Yogamu, Bill Barry DevOps support, enhedu.com, Noorisys, One Infinity- DevOps Setup, and more. He exhibits exceptional analytical skills alongside a creative mindset. Moreover, he possesses a fervent passion for reading books and exploring novel technologies and innovations.

Request for Proposal

Name is required

Comment is required

Sending message..