How to stop/start EC2 instance on a scheduled basis to save cost by using Boto3 and Lambda
This is one of the ask I came across in Dev env in terms of saving cost where I need to shut down all the EC2 instance at 6 pm and bring it back the next day at 9 am.
Problem: Shutdown all EC2 instances in the AWS DEV account at 6 pm and bring them back the next day at 9 am (Monday to Friday).
Solution: Using Lambda function in a combination with CloudWatch events.
- One of the major challenge in implementing this what would be the case if Developer is working late and he wants his instance to be run beyond 6 pm or if there is an urgent patch he needs to implement and need to work on the weekend?
- One common solution we come out with is manually specifying the list of an instance in Python Code(Lambda Function) and in case of exception it goes through a change management process where we need to remove developer instance manually(Agree not an ideal solution but so far works great)
Step1: Create IAM Role so that Lambda can interact with CloudWatch Events
Go to IAM Console https://console.aws.amazon.com/iam/home?region=us-west-2#/home --> Roles --> Create role
- In the next screen, select Create Policy and then JSON tab and then copy the below IAM Policy
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:*:*:*"
},
{
"Effect": "Allow",
"Action": [
"ec2:DescribeInstances",
"ec2:DescribeRegions",
"ec2:StartInstances",
"ec2:StopInstances"
],
"Resource": "*"
}
]
}
- Skip the tag section and click Next
- Give some meaningful name to your policy for e.g. ec2_stop_start and then click on Create policy
- Go back to the IAM console and Create a role
Go to IAM Console https://console.aws.amazon.com/iam/home?region=us-west-2#/home --> Roles --> Create role
- But this time select the policy(ec2_stop_start) we have created above. Click Next:Tags and again skip the Tagging part
- Give some meaningful name to the role(for e.g. ec2_stop_start_role)and click on Create role
- In case if you don’t want to create an IAM role manually, you can automate the entire process using awscli
- The below IAM Policy has the permission to create and push logs to AWS CloudWatch. It also has the permission to stop/start/describe the EC2 instance and Regions.
cat iam_policy.json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:*:*:*"
},
{
"Effect": "Allow",
"Action": [
"ec2:DescribeInstances",
"ec2:DescribeRegions",
"ec2:StartInstances",
"ec2:StopInstances"
],
"Resource": "*"
}
]
}
- This time let’s create IAM policy using awscli
aws iam create-policy --policy-name iam-policy-stop-start-ec2 --policy-document file://iam_policy.json
{
"Policy": {
"PolicyName": "iam-policy-stop-start-ec2",
"PolicyId": "ANPA47TQRQ43C3PXNTFLU",
"Arn": "arn:aws:iam::XXXXXXXXX:policy/iam-policy-stop-start-ec2",
"Path": "/",
"DefaultVersionId": "v1",
"AttachmentCount": 0,
"PermissionsBoundaryUsageCount": 0,
"IsAttachable": true,
"CreateDate": "2021-06-18T22:20:36Z",
"UpdateDate": "2021-06-18T22:20:36Z"
}
}
NOTE: Please note down the arn as we need it in the later step while creating the IAM role.
- The next step is to create an IAM Role and attach the trust policy to it
cat Role-Trust-Policy.json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
NOTE: Trust policy defines which principals can assume the role and under which conditions.
Now execute the aws iam create-role command to create the role
aws iam create-role --role-name ec2-stop-start --assume-role-policy-document file://Role-Trust-Policy.json
{
"Role": {
"Path": "/",
"RoleName": "ec2-stop-start",
"RoleId": "AROA47TQRQ43EZG6DFGOO",
"Arn": "arn:aws:iam::XXXXXXXXX:role/ec2-stop-start",
"CreateDate": "2021-06-18T22:32:17Z",
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
}
}
- Finally, create an IAM Role by attaching the arn of the IAM Policy create in the earlier step.
aws iam attach-role-policy --policy-arn arn:aws:iam::XXXXXXX:policy/iam-policy-stop-start-ec2 --role-name ec2-stop-start
Step2: Create Lambda function
- Go to Lambda https://us-west-2.console.aws.amazon.com/lambda/home?region=us-west-2#/home
- Select Create Function
* Select Author from scratch
* Name: Give your Lambda function any name(for e.g. EC2_stop)
* Runtime: Select Python3.7 as runtime
* Role: Choose the role we create in first step(ec2_stop_start_role)
* Click on Create function
- In this scenario, we need to create Function one to stop an instance and others to start an instance.
Approach 1
import boto3
ec2 = boto3.resource("ec2")
for instance in ec2.instances.all():
instance.stop()
Approach 2.1
import boto3
ec2 = boto3.resource("ec2")
ec2_instance = {"Name": "instance-type", "Values": ["t2.micro"]}
for instance in ec2.instances.filter(Filters=[ec2_instance]):
instance.stop()
Approach 2.2
import boto3
ec2 = boto3.resource("ec2")
ec2_instance = {"Name": "instance-type", "Values": ["t2.micro"]}
ec2_tag = {"Name": "tag:Name", "Values": ["boto3-prod"]}
for instance in ec2.instances.filter(Filters=[ec2_tag]):
instance.stop()
Approach 3
import boto3
ec2 = boto3.resource("ec2")
regions = []
for region in ec2.meta.client.describe_regions()['Regions']:
regions.append(region['RegionName'])
for region in regions:
ec2 = boto3.resource("ec2", region_name=region)
print(“EC2 region is:", region)
ec2_instance = {"Name": "instance-state-name", "Values": ["running"]}
instances = ec2.instances.filter(Filters=[ec2_instance])
for instance in instances:
instance.stop()
print("The following EC2 instances is now in stopped state", instance.id)
- Once the function is created copy-paste the code from Approach 3(or depend upon your requirement)to the Lambda console and click on Deploy. Keep all the other settings as default
- If you have a large set of EC2 instances, you might need to increase the Timeout value from the default value of 3 sec. In order to do that click on the Configuration tab and then Edit.
Similarly to start the instance you can use start method, so the only change in code is instead of using instance.stop() use instance.start().
Step3: Create the CloudWatch event to trigger this Lambda function
- Open the Amazon CloudWatch console.
- Choose Events, and then choose Create rule.
- Choose Schedule under Event Source.
* Under Cron expression choose * 18 * * ? * (If you want to shutdown your instance at 6pm everyday)
* Choose Add target, and then choose Lambda function(EC2_stop)that you created earlier to stop the instance
- The same steps need to be repeated for Starting the instance
* Only difference is different time schedule
* Under target different Lambda function(to start the instance)
NOTE: One very important point to note is that all scheduled event is in UTC timezone, so you need to customize it based on your timezone.
- Once the event is triggered, go to your
Lambda function --> Monitoring --> View logs in CloudWatch
- For stopping the instance, you will see something like this
- For starting the instance
- The simple automation system is ready in stopping/starting the instance and to save some company money.
GitHub: https://github.com/100daysofdevops/100daysofdevops/tree/master/aws-lambda/ec2_stop_start_instances
Please join me with my journey by following any of the below links
- Website: https://101daysofdevops.com/
- Twitter: @100daysofdevops OR @lakhera2015
- Facebook: https://www.facebook.com/groups/795382630808645/
- Medium: https://medium.com/@devopslearning
- GitHub: https://github.com/100daysofdevops/100daysofdevops
- Slack: https://join.slack.com/t/100daysofdevops/shared_invite/enQtNzg1MjUzMzQzMzgxLWM4Yjk0ZWJiMjY4ZWE3ODBjZjgyYTllZmUxNzFkNTgxZjQ4NDlmZjkzODAwNDczOTYwOTM2MzlhZDNkM2FkMDA
- YouTube Channel: https://www.youtube.com/user/laprashant/videos?view_as=subscriber
Tag:automation, boto3, devops, ec2, python
