Day 6 – Python Subprocess module
Welcome to Day 6 of 101 Days of DevOps. The topic for today is a python subprocess module.
To view the complete course, please check the below url.
On Day 1, we discussed OS Module. As part of the discussion, we discussed the os.system(). One of the major drawbacks of the OS module is you can’t save the output of the executed command.
If you execute the below script rt variable
import os
rt = os.system("ls -l")
print(rt)
The output of rt is simply 0 and not the output of the command, and this where we need to use the subprocess module.
As per doc, “the subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. This module intends to replace several older modules and functions.”
os.system os.spawn*
Syntax
sp = subprocess.Popen(command,shell=True/False,stdout=subprocess.PIPE,stderr=subprocess.PIPE,universal_newlines=True)
- To understand the subprocess.Popen, we first need to understand the process. A process refers to the program in execution. To create new processes, we can use library functions such as subprocess.Popen(). These subprocess run completely independently with their own private system state and the main thread of execution.
- Let’s take a simple example before exploring the other options/arguments
import subprocess cmd="ls -l" sp = subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE) rt=sp.wait() out,err=sp.communicate() print(out)
- The output of the above script will look like this
python3 newtest1.py b'total 56\n-rw-r--r-- 1 plakhera plakhera 191 Jun 18 15:31 Role-Trust-Policy.json\n-rw-r--r-- 1 plakhera plakhera 552 Jun 12 08:51 boto3_test.py'
- Python treats output as a byte code; by adding universal_newlines=True, we will get an output in the form of a string. Add this parameter and re-execute the code
import subprocess cmd="ls -l" sp = subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE,universal_newlines=True) rt=sp.wait() out,err=sp.communicate() print(out)
python3 newtest1.py total 56 -rw-r--r-- 1 plakhera plakhera 191 Jun 18 15:31 Role-Trust-Policy.json -rw-r--r-- 1 plakhera plakhera 552 Jun 12 08:51 boto3_test.py
- Once the command is executed, you need to wait for that command to be executed completely(e.g., if we are trying to connect to the database from our process), and the command is going to return some return code(0=sucessfull, non-zero=unsucessful)
rc=sp.wait()
- Once the command is executed successfully or if it’s failed, its return tuple(stdout or stderr). Suppose we omit stdout=subprocess.PIPE or stderr=subprocess.PIPE in the Popen call; we will get the none back. sp.communicate() interact with our process and send data to stdin or read data from stdout/stderr till it read the whole file. It also waits for the process to be fully terminated.
out,err=sp.communicate()
- Now let’s look at the other parameter shell=True/False
- In case of your command depend upon the environment variable of the operating system, then only in those cases take shell=True
command="ls -l” command=["echo","$HOME"]
- In case if your command doesn’t need a new shell, use shell=False.
cmd=[“ls”,”-lrt”] shell=False
- If we are taking shell=True, we need to write the command as it’s in the form of a string.
import subprocess cmd="echo $HOME" sp = subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE,universal_newlines=True) rt=sp.wait() out,err=sp.communicate() print(out)
NOTE: Please try to avoid using shell=True. Executing programs via shell means that all the user input passed to the program is interpreted according to the syntax and semantic rules of the invoked shell. , special characters like quotation marks need to be escaped, and it can also cause security leaks because the user can execute the arbitrary program.
Assignment:
- Try to execute any operating system with shell=True and shell=False.
I am looking forward to you guys joining the amazing journey.