Dear Lambda, Shutdown that EC2 for me!
Problem statement
Save cost on our on-demand EC2 instances by shutting them down based on a schedule defined within a tag.
Solution description
The solution can be found on my Github repo.
We came up with the following tag format to help us shutdown our EC2 instances. As each EC2 instance runs specific applications with varying workloads and access patterns, we wanted to define a set of tags to cover all of them. This is what we came up with.
Schedule
24x5_Mon-Fri
Shutdown at 12am on Saturday and start back up at 12am on Monday.
08–24_Mon-Fri
Shutdown at 12am on Saturday and start up at 8am on Monday.
08–18_Mon-Sun
Shutdown at 6pm Monday to Sunday and start up at 8am.
08–18_Mon-Fri
Shutdown at 6pm Monday to Friday and start back up at 8am.
18_Shutdown
Shutdown at 6pm and do not bring it back up.
24x7_Mon-Sun
Should not shutdown as the instance needs to be up 24x7.
Maintenance
Nothing to do but send an email notifying that the instance is tagged for maintenance.
Lambda
Lambda to the rescue. The serverless framework was used to package things up as it was an intuitive way to set it up and hook into our current CI/CD.
Python was the natural fit for this lambda with the libraries around AWS specially with boto3.
The lambda fires up at a defined interval as configured on your serverless.yaml which then checks to see if any of the above schedule constraints are met. If so, we perform the relevant operation(either a start or stop) on those EC2 instances. The scheduled activity on the lambda is done via the integration with the AWS EventBridge which is all done under the hood through the serverless framework.
Did I tell you how awesome this framework is? Well get ready to hear it a couple of more times through this post!
In terms of the actual implementation, I will leave it up to the reader to go through the GitHub repository. It was a mixture of the usage of pytz, datetime and boto3 in action.
Test driven development (TDD)
I am a huge advocate of TDD and thus was looking at implementing my lambda using TDD. I stumbled upon this amazing library called moto. Huge shout out to these guys as it is just an amazing piece of work.
What it essentially lets you do is run a virtual AWS environment and run tests/assertions against it. This was just what I needed and saved me a good several hours as I did not have to go through the deploy/test cycles to test my lambda.
As I used Pipfile to package my lambda, it was easy for me to separate out the dependencies needed for development and production.
As you can see from the following, I was able to package on moto and pyyaml just for testing purposes and leave the rest to be bundled up when I do a deployment to production.
[dev-packages]moto = "*"pyyaml = "*"[packages]boto3 = "*"pytz = "*"
Dockerize the tests
When integrating this to our existing CI/CD platform, I did not want to have to go in to all the servers and install the Pipenv related dependencies just to run the integration tests.
Docker to the rescue. The panacea to all things these days ain’t it!
I wrote a quick Dockerfile that installed all the required dependencies and ran the tests.
I also wanted to make sure that the docker instance ran with the UTC timezone. The Dockerfile used is as follows;
FROM python:3.6-slimENV AWS_DEFAULT_REGION=ap-southeast-2ENV TZ=UTCRUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezoneRUN pip install — upgrade pipRUN pip install pipenvADD Pipfile /home/ec2-stop-lambda/ADD *.py /home/ec2-stop-lambda/ADD serverless.yml /home/ec2-stop-lambda/WORKDIR /home/ec2-stop-lambdaRUN pipenv install — devENTRYPOINT exec pipenv run test
The python slim version of the docker image was used to keep the size of the artefact at a minimum.
Conclusion
The Lambda runs on all our environments and works as expected to start and stop all our EC2 instances on the schedules we define.
Do drop a comment below if you have anything you would like to add on and thank you for reading!
Have a good week ahead people. Stay safe!
