• About WordPress
    • WordPress.org
    • Documentation
    • Support
    • Feedback
  • Log In
  • Register
  • Home
  • About Us
  • Blog
  • Courses
  • Contact Us

Have any question?

101daysofdevops@gmail.com
RegisterLogin
101DaysofDevops
  • Home
  • About Us
  • Blog
  • Courses
  • Contact Us

Blog

  • Home
  • Blog
  • Blog
  • How to stop/start EC2 instance on a scheduled basis to save cost by using Boto3 and Lambda

How to stop/start EC2 instance on a scheduled basis to save cost by using Boto3 and Lambda

  • Posted by lakhera2020
  • Date June 19, 2021
  • Comments 0 comment
Stop EC2 instances using Boto3 – Part 1
Stop EC2 instances using Boto3 – Part 2

This is one of the ask I came across in Dev env in terms of saving cost where I need to shut down all the EC2 instance at 6 pm and bring it back the next day at 9 am.

Problem: Shutdown all EC2 instances in the AWS DEV account at 6 pm and bring them back the next day at 9 am (Monday to Friday).

Solution: Using Lambda function in a combination with CloudWatch events.

  • One of the major challenge in implementing this what would be the case if Developer is working late and he wants his instance to be run beyond 6 pm or if there is an urgent patch he needs to implement and need to work on the weekend?
  • One common solution we come out with is manually specifying the list of an instance in Python Code(Lambda Function) and in case of exception it goes through a change management process where we need to remove developer instance manually(Agree not an ideal solution but so far works great)

Step1: Create IAM Role so that Lambda can interact with CloudWatch Events

Go to IAM Console https://console.aws.amazon.com/iam/home?region=us-west-2#/home --> Roles --> Create role
  • In the next screen, select Create Policy and then JSON tab and then copy the below IAM Policy 
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:*:*:*"
},
{
"Effect": "Allow",
"Action": [
"ec2:DescribeInstances",
"ec2:DescribeRegions",
"ec2:StartInstances",
"ec2:StopInstances"
],
"Resource": "*"
}
]
}
  • Skip the tag section and click Next
  • Give some meaningful name to your policy for e.g. ec2_stop_start and then click on Create policy
  • Go back to the IAM console and Create a role
Go to IAM Console https://console.aws.amazon.com/iam/home?region=us-west-2#/home --> Roles --> Create role
  • But this time select the policy(ec2_stop_start) we have created above. Click Next:Tags and again skip the Tagging part
  • Give some meaningful name to the role(for e.g. ec2_stop_start_role)and click on Create role
  • In case if you don’t want to create an IAM role manually, you can automate the entire process using awscli
  • The below IAM Policy has the permission to create and push logs to AWS CloudWatch. It also has the permission to stop/start/describe the EC2 instance and Regions.
cat iam_policy.json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:*:*:*"
},
{
"Effect": "Allow",
"Action": [
"ec2:DescribeInstances",
"ec2:DescribeRegions",
"ec2:StartInstances",
"ec2:StopInstances"
],
"Resource": "*"
}
]
}
  • This time let’s create IAM policy using awscli
aws iam create-policy --policy-name iam-policy-stop-start-ec2 --policy-document file://iam_policy.json
{
"Policy": {
"PolicyName": "iam-policy-stop-start-ec2",
"PolicyId": "ANPA47TQRQ43C3PXNTFLU",
"Arn": "arn:aws:iam::XXXXXXXXX:policy/iam-policy-stop-start-ec2",
"Path": "/",
"DefaultVersionId": "v1",
"AttachmentCount": 0,
"PermissionsBoundaryUsageCount": 0,
"IsAttachable": true,
"CreateDate": "2021-06-18T22:20:36Z",
"UpdateDate": "2021-06-18T22:20:36Z"
}
}

NOTE: Please note down the arn as we need it in the later step while creating the IAM role.

  • The next step is to create an IAM Role and attach the trust policy to it
cat Role-Trust-Policy.json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}

NOTE: Trust policy defines which principals can assume the role and under which conditions.

Now execute the aws iam create-role command to create the role

aws iam create-role --role-name ec2-stop-start --assume-role-policy-document file://Role-Trust-Policy.json
{
"Role": {
"Path": "/",
"RoleName": "ec2-stop-start",
"RoleId": "AROA47TQRQ43EZG6DFGOO",
"Arn": "arn:aws:iam::XXXXXXXXX:role/ec2-stop-start",
"CreateDate": "2021-06-18T22:32:17Z",
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
}
}
  • Finally, create an IAM Role by attaching the arn of the IAM Policy create in the earlier step.
aws iam attach-role-policy --policy-arn arn:aws:iam::XXXXXXX:policy/iam-policy-stop-start-ec2 --role-name ec2-stop-start

Step2: Create Lambda function

  • Go to Lambda https://us-west-2.console.aws.amazon.com/lambda/home?region=us-west-2#/home
  • Select Create Function
* Select Author from scratch
* Name: Give your Lambda function any name(for e.g. EC2_stop)
* Runtime: Select Python3.7 as runtime
* Role: Choose the role we create in first step(ec2_stop_start_role)
* Click on Create function
  • In this scenario, we need to create Function one to stop an instance and others to start an instance.

Approach 1

import boto3

ec2 = boto3.resource("ec2")

for instance in ec2.instances.all():
instance.stop()

Approach 2.1

import boto3

ec2 = boto3.resource("ec2")

ec2_instance = {"Name": "instance-type", "Values": ["t2.micro"]}

for instance in ec2.instances.filter(Filters=[ec2_instance]):
instance.stop()

Approach 2.2

import boto3

ec2 = boto3.resource("ec2")

ec2_instance = {"Name": "instance-type", "Values": ["t2.micro"]}
ec2_tag = {"Name": "tag:Name", "Values": ["boto3-prod"]}

for instance in ec2.instances.filter(Filters=[ec2_tag]):
instance.stop()

Approach 3

import boto3

ec2 = boto3.resource("ec2")

regions = []
for region in ec2.meta.client.describe_regions()['Regions']:
regions.append(region['RegionName'])


for region in regions:
ec2 = boto3.resource("ec2", region_name=region)

print(“EC2 region is:", region)

ec2_instance = {"Name": "instance-state-name", "Values": ["running"]}

instances = ec2.instances.filter(Filters=[ec2_instance])

for instance in instances:
instance.stop()
print("The following EC2 instances is now in stopped state", instance.id)
  • Once the function is created copy-paste the code from Approach 3(or depend upon your requirement)to the Lambda console and click on Deploy. Keep all the other settings as default
  • If you have a large set of EC2 instances, you might need to increase the Timeout value from the default value of 3 sec. In order to do that click on the Configuration tab and then Edit.

Similarly to start the instance you can use start method, so the only change in code is instead of using instance.stop() use instance.start().

Step3: Create the CloudWatch event to trigger this Lambda function

  • Open the Amazon CloudWatch console.
  • Choose Events, and then choose Create rule.
  • Choose Schedule under Event Source.
* Under Cron expression choose * 18 * * ? * (If you want to shutdown your instance at 6pm everyday)
* Choose Add target, and then choose Lambda function(EC2_stop)that you created earlier to stop the instance
  • The same steps need to be repeated for Starting the instance
* Only difference is different time schedule
* Under target different Lambda function(to start the instance)

NOTE: One very important point to note is that all scheduled event is in UTC timezone, so you need to customize it based on your timezone.

  • Once the event is triggered, go to your
Lambda function --> Monitoring --> View logs in CloudWatch
  • For stopping the instance, you will see something like this
  • For starting the instance
  • The simple automation system is ready in stopping/starting the instance and to save some company money.

GitHub: https://github.com/100daysofdevops/100daysofdevops/tree/master/aws-lambda/ec2_stop_start_instances

Please join me with my journey by following any of the below links

  • Website: https://101daysofdevops.com/
  • Twitter: @100daysofdevops OR @lakhera2015
  • Facebook: https://www.facebook.com/groups/795382630808645/
  • Medium: https://medium.com/@devopslearning
  • GitHub: https://github.com/100daysofdevops/100daysofdevops
  • Slack: https://join.slack.com/t/100daysofdevops/shared_invite/enQtNzg1MjUzMzQzMzgxLWM4Yjk0ZWJiMjY4ZWE3ODBjZjgyYTllZmUxNzFkNTgxZjQ4NDlmZjkzODAwNDczOTYwOTM2MzlhZDNkM2FkMDA
  • YouTube Channel: https://www.youtube.com/user/laprashant/videos?view_as=subscriber

Tag:automation, boto3, devops, ec2, python

  • Share:
author avatar
lakhera2020

Previous post

Boto3 Concepts(Waiter, Meta, and Paginator)
June 19, 2021

Next post

My road to Certified Kubernetes Security Specialist (CKS)
September 21, 2021

You may also like

Am I reading the iostat command output correctly?
25 April, 2022

Iostat command came from the same sysstat family package # rpm -qf `which iostat` sysstat-11.7.3-6.el8.x86_64 It mainly read data from /proc/diskstats # cat /proc/diskstats 259 0 nvme1n1 147 0 6536 …

Debugging Performance Issue using SAR
21 April, 2022

What is SAR? SAR is a utility used to collect and report system activity. It collects data relating to most core system functions and writes those metrics to binary data …

4 common Kubernetes Pods Error and Debugging
20 April, 2022

Why do Kubernetes Pods fail? The two most common reasons for Kubernetes pod failure is The container inside the pod doesn’t start, which we also call a startup failure. The …

Leave A Reply Cancel reply

Your email address will not be published. Required fields are marked *

Recent Posts

  • Am I reading the iostat command output correctly?
  • Debugging Performance Issue using SAR
  • 4 common Kubernetes Pods Error and Debugging
  • My road to Gremlin Chaos Engineering Practitioner Certificate
  • My road to Certified Kubernetes Security Specialist (CKS)

Recent Comments

  • lakhera2020 on Debugging Performance Issue using SAR
  • Anonymous on Debugging Performance Issue using SAR
  • Pety on Day 2 – MetalLB Load Balancer for Bare Metal Kubernetes
  • akashambasta on Day 1 – AWS IAM User
  • rd on 100 Days of AWS

 

101daysofdevops@gmail.com

  • Home
  • About Us
  • Courses
  • Blog

© 101daysofdevops. All rights reserved.

Login with your site account

Lost your password?

Not a member yet? Register now

Register a new account

Are you a member? Login now