Introduction
Currently, AWS has three associate level certifications: Solutions Architect, Developer and SysOps Administrator. In 2017, I have passed the certification exams for the first two certifications and as I visited AWS re:Invent in Las Vegas last year, I decided that I would like to also pass the one that is missing from my curriculum right now.
On Course For Certification
One of the ways to prepare for the AWS Associate certifications (apart from reading the documentation), is completing one of the many courses being offered on the on-line learning platforms. For my first two certifications, I used the certification training offered by A Cloud Guru which were very good. However, in the meantime I have also completed some trainings on the Udemy Platform by Stephane Maarek on miscellaneous topics related to Amazon Web Services (CloudFormation Masterclass, AWS Lambda and the Serverless Framework) that I also like very much, so I decided to start my certification course with Stephane’s wonderful “Ultimate AWS Certified SysOps Administrator Associate 2019” (quite a mouthful, but it’s a 17 hour journey).
The Same Old Song?
Some of the topics for this certification come up in other tracks as well, e.g. “How do you obtain information instance information from the EC2 instance currently running?” and topics like “Is available memory a standard EC2 Metric?”.
The answer to the last question used to be that EC2-instance memory statistics are not available as CloudWatch metrics out of the box, however AWS does provide some PERL scripts to capture this information as Custom CloudWatch metrics. Although AWS now provides a brand new CloudWatch Agent that can be installed on both EC2 and on-premise environments, there is still value in creating your own custom metrics for CloudWatch, as the CloudWatch agent might not expose the metrics you’re interested in for your environment or use case.
For the older generation of developers and system administrators, PERL might have been a valid choice. However, as a developer I am much more comfortable with a well-structured and readable language like Python, so I was wondering how complex it would be to create my own custom metrics using some Python scripting.
Preparing the environment
One of the (may) good features on AWS is its fine-grained security model; you can easily tailor the permissions required for your specific use case and allow the service or program no more privileges than it requires to complete its task.
IAM – roles and policies
First, we start by defining a new role to be used for the EC2 instance. This will allow the EC2 instance to send its data to CloudWatch:
In this role, we include a new policy that only allows it to send metric data to CloudWatch:
Finally, we attach this IAM policy to the EC2 instance we’re launching:
Now launch your EC2 instance, I am launching mine in the eu-west-1 (Ireland) region.
Create an IAM user
As I am planning to send the custom metric data using the AWS Boto3 library, I will need to create an IAM user as well for programmatic access only; I’ve assigned this user the same policy (e.g. send data to CloudWatch) and copied its secret and access keys from the IAM console.
Dependencies
As I have launched my instance using a rather bare Amazon Linux 2 AMI, I do not yet have all the required packages installed. Let’s assume that we will be using this EC2 instance for serving some webserver traffic and it will be running a simple installation of Apache’s httpd.
To install the dependencies, SSH into your EC2 instance after launching has completed:
yum install gcc python3 python3-devel httpd -y # install required OS packages sudo systemctl start httpd # start HTTP server as a daemon/service sudo chkconfig httpd on # make sure it starts after reboot sudo pip3 install psutil boto3 requests # install Python dependencies mkdir ~/.aws # prepare for AWS configuration
After installing the OS packages and Python modules, there is some basic setup to perform: first, you have to create a file called credentials inside your ~/.aws directory for boto3. This file needs to contain two named keys for boto to be able to access your AWS account as a user (remember you restricted the user to only allow sending metric data). Additionally, create a file called config with a single key ‘region’ having the value ‘eu-west-1’ as I have launched my EC2 instance in the Ireland region (and so I want my metrics to be to the Emerald Isle as well):
$ more ~/.aws/* /home/ec2-user/.aws/config :::::::::::::: [default] region=eu-west-1 :::::::::::::: /home/ec2-user/.aws/credentials :::::::::::::: [default] aws_access_key_id=your-access-id-obtained-from-iam aws_secret_access_key=your-super-secret-key-from-iam
Finally, it is time for the script that actually calculates and sends the metrics to be created. This script uses the psutil package to obtain statistics on the current environment, like the percentage of memory being used, the percentage of storage used on the root device and the amount of memory being consumed by the HTTP processes.
Also, it obtains information on the instance currently running from the same and thus answers the first question: you can obtain this information by performing an HTTP GET call to http://169.254.169.254/latest/meta-data/<some-information-item-id> to retrieve the required information:
#!/usr/bin/env python3
import boto3
import psutil
import requests
_METADATAURL = 'http://169.254.169.254/latest/meta-data' # where to obtain instance metadata ...
cw = boto3.client('cloudwatch')
currMetrics = []
def appendMetrics(CurrentMetrics, Dimensions, Name, Unit, Value):
metric = { 'MetricName' : Name
, 'Dimensions' : Dimensions
, 'Unit' : Unit
, 'Value' : Value
}
CurrentMetrics.append(metric)
def memUsedByApache():
return round(sum([p.info['memory_info'].rss for p in psutil.process_iter(attrs=['name','memory_info']) if 'httpd' in p.info['name']]) / (1024*1024), 1)
def usedMemoryPercentage():
return psutil.virtual_memory().percent
def usedDiskSpace():
return psutil.disk_usage('/').percent
if __name__ == '__main__':
instance_id = requests.get( _METADATAURL + '/instance-id').text
instance_type = requests.get( _METADATAURL + '/instance-type').text
dimensions = [{'Name' : 'InstanceId', 'Value': instance_id}, {'Name' : 'InstanceType', 'Value': instance_type}]
appendMetrics(currMetrics, dimensions, Name='ApacheMemory', Value=memUsedByApache(), Unit='Megabytes')
appendMetrics(currMetrics, dimensions, Name='MemoryInUse', Value=usedMemoryPercentage(), Unit='Percent')
appendMetrics(currMetrics, dimensions, Name='DiskspaceUsed', Value=usedDiskSpace(), Unit='Percent')
response = cw.put_metric_data(MetricData = currMetrics, Namespace='CustomMetrics')
Scheduling
As a final step, I need to make sure that the custom metric data is sent to CloudWatch on a regular basis. By making this script I have created as ~/metrics.py executable and scheduling it to run every 5 minutes using Linux cron facility, I can send custom metric data to CloudWatch easily:
Metrics, anyone?
After some time has elapsed (while generating some load using curl for the HTTP-server and consuming some disk space by generating files using dd), several datapoints for the new metrics have been committed to CloudWatch. Switching to CloudWatch’ metrics, the new metrics can be graphed in an ad-hoc fashion:
Dashboard
However, if these metrics are truly important for your environment, you can also easily turn them into a custom dashboard:
Conclusion
Experimenting with some of the new concepts in AWS is not only good to satisfy my curiosity and increase my experience, but is also a nice opportunity to get into action and a nice getting away from watching over 200 videos (over 17 hours of video).
So yes, although you can rely on the good old Perl scripts provided by Amazon as examples for custom metrics, if the metrics you require are not available out of the box for your EC2 environment or provided by the new CloudWatch agents, it’s a piece of cake to roll your own … as long as you can extract or calculate the metric values using Python!
