• About WordPress
    • WordPress.org
    • Documentation
    • Support
    • Feedback
  • Log In
  • Register
  • Home
  • About Us
  • Blog
  • Courses
  • Contact Us

Have any question?

101daysofdevops@gmail.com
RegisterLogin
101DaysofDevops
  • Home
  • About Us
  • Blog
  • Courses
  • Contact Us

Blog

  • Home
  • Blog
  • Blog
  • Boto3 Concepts(Waiter, Meta, and Paginator)

Boto3 Concepts(Waiter, Meta, and Paginator)

  • Posted by lakhera2020
  • Date June 17, 2021
  • Comments 0 comment

In the last blog, you understand the concept of Boto3. Let’s solidify that idea by looking at some other Boto3 concepts.

Introduction to AWS Boto3

Boto3 Meta

Boto3 Paginator

Boto3 Waiters

Waiters: In the case of Boto3, there are some requests which are not instant. One such example is when you try to start or stop any instance. For such kinds of requests, you can initiate those requests and check back at some later time. But in cases such as EC2, we need to wait for the requests to be complete before we move on to the next part. So AWS SDK provides many waiters that allow you to block the code until the process is complete. 

A typical ec2 instance start script with waiter look like this

import boto3

ec2 = boto3.resource("ec2")

instance_id=input("Please enter the instance id: ")
instance=ec2.Instance(instance_id)
print("Starting EC2 instance")
instance.start()
instance.wait_until_running()
print("Your instance is up and running")

In this script, we use a waiter wait_until_running, which runs in the background and loops over 40 times every 5 seconds. The other available waiters are for EC2

wait_until_exists 
wait_until_running
wait_until_stopped
wait_until_terminated'

The above code is written using resources. Similarly, we have several waiters available for the client.EC2 – Boto3 Docs 1.17.94 documentation

import boto3

ec2 = boto3.client("ec2")

instance_id=input("Please enter the instance id: ")
print("Starting EC2 instance")
instance=ec2.start_instances(InstanceIds=[instance_id])
waiter = ec2.get_waiter('instance_running')
waiter.wait(InstanceIds=[instance_id])

print("Your instance is up and running")

In the case of a client, the waiter will run 15 loops in the background over the period of 40 seconds. 

We can also mix and match where we can use the resource object to start the instance but use client waiters. 

import boto3

ec2 = boto3.resource("ec2")
ec2_cli = boto3.client("ec2")

instance_id=input("Please enter the instance id: ")
instance=ec2.Instance(instance_id)
print("Starting EC2 instance")
instance.start()
waiter = ec2_cli.get_waiter('instance_running')
waiter.wait(InstanceIds=[instance_id])

print("Your instance is up and running")

Meta: As discussed in Introduction to Boto3 blog, Boto3 client support all the operation whereas resource doesn’t. Suppose we have the existing code written using the resource, and now if there is a requirement where we need to modify the code to use the operation which is not supported by the resource. Do we need to re-write the code to use the client? The answer is no; we can use meta to access the client directly via resource. In the below use case, we want to list out all the regions supported by EC2, and this operation is not available for resources; hence, in this case, we can access client operation using meta.

>>> import boto3
>>> ec2 = boto3.resource("ec2")
>>> for region in ec2.meta.client.describe_regions()['Regions']:
... print(region['RegionName'])
...
eu-north-1
ap-south-1
eu-west-3
eu-west-2
eu-west-1
ap-northeast-3
ap-northeast-2
ap-northeast-1
sa-east-1
ca-central-1
ap-southeast-1
ap-southeast-2
eu-central-1
us-east-1
us-east-2
us-west-1
us-west-2

Paginator: In the case of AWS clients, certain operations return the incomplete result, requiring subsequent requests to obtain the entire result set. The process of sending the subsequent requests to continue where a previous request left off is called pagination. E.g., in the case of the S3 list_object operation, it only returns 1000 objects, so we need to send the subsequent requests with the appropriate marker to retrieve the next page of results. 

Advantage: The advantage of the pagination approach is you will get the results more quickly rather than waiting for the entire result. 

Let understand this with the help of IAM.

Let try to print the list of IAM user present in my AWS account using the resource.

import boto3

iam_res = boto3.resource("iam")

count=1
for user in iam_res.users.all():
print(count, user.name)
count=count+1
python3 test.py |wc -l
145

As you can see in the output of the above script, I have 145 users present in my account. Let’s re-write the same code using the client.

import boto3

iam_client = boto3.client("iam")

count=1
for user in iam_client.list_users()['Users']:
print(count, user['UserName'])
count=count+1
python3 test.py |wc -l
100

As you can see in the above code, it only returns the first 100 results. If you use the paginator called list_users and paginate through the response, you will see all the IAM users. By default, the page size is set to 100, so it will return the result in two different pages that are 100 on the first page and the remaining user on the next page. 

import boto3

iam_client = boto3.client("iam")
paginator = iam_client.get_paginator('list_users')
page_iterator=paginator.paginate()
count=1
for user in page_iterator:
for username in user['Users']:
print(count, username['UserName'])
count+=1
python3 test.py  |wc -l
145

NOTE: For most of the AWS resources, it returns 50 or 100 results except S3 and the paginator is not available for all AWS services in those cases you need to write your own custom paginator. 

So the bottom line if you are trying to retrieve more than one page of results, you need to use a paginator to issue multiple API requests on your behalf.

GitHub:

https://github.com/100daysofdevops/100daysofdevops/tree/master/boto3

Please join me with my journey by following any of the below links

  • Website: https://101daysofdevops.com/
  • Twitter: @100daysofdevops OR @lakhera2015
  • Facebook: https://www.facebook.com/groups/795382630808645/
  • Medium: https://medium.com/@devopslearning
  • GitHub: https://github.com/100daysofdevops/100daysofdevops
  • Slack: https://join.slack.com/t/100daysofdevops/shared_invite/enQtNzg1MjUzMzQzMzgxLWM4Yjk0ZWJiMjY4ZWE3ODBjZjgyYTllZmUxNzFkNTgxZjQ4NDlmZjkzODAwNDczOTYwOTM2MzlhZDNkM2FkMDA
  • YouTube Channel: https://www.youtube.com/user/laprashant/videos?view_as=subscriber


Tag:automation, aws, boto3, meta, paginators, python, waiters

  • Share:
author avatar
lakhera2020

Previous post

Introduction to AWS Boto3
June 17, 2021

Next post

How to stop/start EC2 instance on a scheduled basis to save cost by using Boto3 and Lambda
June 19, 2021

You may also like

Am I reading the iostat command output correctly?
25 April, 2022

Iostat command came from the same sysstat family package # rpm -qf `which iostat` sysstat-11.7.3-6.el8.x86_64 It mainly read data from /proc/diskstats # cat /proc/diskstats 259 0 nvme1n1 147 0 6536 …

Debugging Performance Issue using SAR
21 April, 2022

What is SAR? SAR is a utility used to collect and report system activity. It collects data relating to most core system functions and writes those metrics to binary data …

4 common Kubernetes Pods Error and Debugging
20 April, 2022

Why do Kubernetes Pods fail? The two most common reasons for Kubernetes pod failure is The container inside the pod doesn’t start, which we also call a startup failure. The …

Leave A Reply Cancel reply

Your email address will not be published. Required fields are marked *

Recent Posts

  • Am I reading the iostat command output correctly?
  • Debugging Performance Issue using SAR
  • 4 common Kubernetes Pods Error and Debugging
  • My road to Gremlin Chaos Engineering Practitioner Certificate
  • My road to Certified Kubernetes Security Specialist (CKS)

Recent Comments

  • lakhera2020 on Debugging Performance Issue using SAR
  • Anonymous on Debugging Performance Issue using SAR
  • Pety on Day 2 – MetalLB Load Balancer for Bare Metal Kubernetes
  • akashambasta on Day 1 – AWS IAM User
  • rd on 100 Days of AWS

 

101daysofdevops@gmail.com

  • Home
  • About Us
  • Courses
  • Blog

© 101daysofdevops. All rights reserved.

Login with your site account

Lost your password?

Not a member yet? Register now

Register a new account

Are you a member? Login now