How to run shell commands in Python? [SOLVED]

Different methods to run shell commands in Python

There are different modules available which can be used to execute or call shell commands inside Python program.

subprocess module: The subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes.
os module: The os module provides a way of using operating system dependent functionality like reading or writing to files, starting new processes, killing processes etc.
os.popen() method: This method is similar to os.system(), but it also allows you to capture the output of the command and return it as a file-like object.
sh library: sh is a python library that allows you to run shell commands as if you were typing them in the terminal.

Method-1: Using subprocess module

Example-1: Using `subprocess.run()` function

import subprocess

# run the "ls -l" command and capture the output
result = subprocess.run(["ls", "-l"], capture_output=True, text=True)

# print the captured output
print(result.stdout)

In this example, we use the subprocess.run() function to run the shell command ls -l. The subprocess.run() function takes a list of arguments, where the first element is the command to run and the rest of the elements are its arguments. In this case, the command is “ls” and the argument is “-l”.

The capture_output parameter is set to True which means that the output of the command will be captured and returned in the stdout attribute of the CompletedProcess object returned by the subprocess.run() function. The text parameter is set to True which means that the output will be returned as a string instead of bytes.

The result.stdout attribute contains the captured output of the command, which in this case will be the output of the ls -l command, that is a list of files in the current directory along with their permissions, owner, group, size and last modification date.

Example-2: Using `subprocess.call()` function

In this example, we use the subprocess.call() function to run the shell command ls -l. The subprocess.call() function takes a list of arguments, where the first element is the command to run and the rest of the elements are its arguments. In this case, the command is “ls” and the argument is “-l”.

The subprocess.call() function returns the return code of the command, which is 0 if the command succeeded, or a non-zero value if the command failed. In this example, we check the return code and print a message indicating whether the command was executed successfully or not.

It’s important to note that the subprocess.call() function also waits for the command to complete before returning, this means that it blocks the execution of your Python script until the command finishes. Also, the subprocess.call() function doesn’t capture the output of the command.

import subprocess

# run the "ls -l" command
return_code = subprocess.call(["ls", "-l"])

# check the return code
if return_code == 0:
    print("Command executed successfully")
else:
    print("Command failed with return code", return_code)

Example-3: Using `subprocess.check_call()` function

The subprocess.check_call() function is similar to subprocess.call(), but it raises an exception if the return code of the command is non-zero. Here’s an example of using subprocess.check_call() to run a shell command:

import subprocess

try:
    subprocess.check_call(["ls", "-l"])
    print("Command executed successfully")
except subprocess.CalledProcessError as e:
    print(f"Command failed with return code {e.returncode}")

In this example, we use the subprocess.check_call() function to run the shell command ls -l. The subprocess.check_call() function takes a list of arguments, where the first element is the command to run and the rest of the elements are its arguments. In this case, the command is “ls” and the argument is “-l”.

The subprocess.check_call() function raises a CalledProcessError exception if the return code of the command is non-zero. We use a try-except block to catch this exception and print an appropriate message indicating whether the command was executed successfully or not.

It’s important to note that the subprocess.check_call() function also waits for the command to complete before returning, this means that it blocks the execution of your Python script until the command finishes. Also, the subprocess.check_call() function doesn’t capture the output of the command.

Example-4: Using `subprocess.check_output()` function

The subprocess.check_output() function is similar to subprocess.check_call(), but it also captures and returns the output of the command:

import subprocess

try:
    output = subprocess.check_output(["ls", "-l"])
    print(output)
    print("Command executed successfully")
except subprocess.CalledProcessError as e:
    print(f"Command failed with return code {e.returncode}")

In this example, we use the subprocess.check_output() function to run the shell command ls -l. The subprocess.check_output() function takes a list of arguments, where the first element is the command to run and the rest of the elements are its arguments. In this case, the command is “ls” and the argument is “-l”.

The subprocess.check_output() function raises a CalledProcessError exception if the return code of the command is non-zero. We use a try-except block to catch this exception and print an appropriate message indicating whether the command was executed successfully or not.

If the command is executed successfully, the output variable will contain the captured output of the command, which in this case will be the output of the ls -l command, that is a list of files in the current directory along with their permissions, owner, group, size and last modification date.

It’s important to note that the subprocess.check_output() function also waits for the command to complete before returning, this means that it blocks the execution of your Python script until the command finishes.

Example-5: Using `subprocess.Popen()` function

The subprocess.Popen() function allows you to start a new process and interact with its input/output/error pipes.

import subprocess

# start the "ls -l" command
process = subprocess.Popen(["ls", "-l"], stdout=subprocess.PIPE)

# read the output
output, _ = process.communicate()

# print the output
print(output)

# check the return code
if process.returncode == 0:
    print("Command executed successfully")
else:
    print("Command failed with return code", process.returncode)

In this example, we use the subprocess.Popen() function to start the shell command ls -l. The subprocess.Popen() function takes a list of arguments, where the first element is the command to run and the rest of the elements are its arguments. In this case, the command is “ls” and the argument is “-l”.

The stdout parameter is set to subprocess.PIPE which means that the output of the command will be captured and returned in the stdout attribute of the Popen object.

The process.communicate() function is used to read the output of the command. It waits for the command to complete and returns a tuple containing the output and the error. In this case, we are not interested in the error, so we use the _ variable to discard it.

The process.returncode attribute contains the return code of the command, which is 0 if the command succeeded, or a non-zero value if the command failed.

It’s important to note that the subprocess.Popen() function doesn’t wait for the command to complete before returning, this means that it allows you to continue executing your Python script while the command is running.

Also, the subprocess.Popen() function returns the output as bytes, if you want the output to be returned as a string, you can use the decode() function to convert it to a string.

Method-2: Using os module

The os module provides a way of using operating system dependent functionality like reading or writing to files, starting new processes, killing processes etc.

Example-1: Using `os.system()` function

import os

# run the "ls -l" command and capture the output
os.system("ls -l")

In this example, we use the os.system() function to run the shell command ls -l. This function runs the command in a subshell and waits for it to complete. The output of the command is sent to the standard output, which means that it will be displayed in the console. The function returns the return code of the command, which is 0 if the command succeeded, or a non-zero value if the command failed.

It’s important to note that the os.system() function is not recommended to use because it’s not secure, it can’t capture the output and it’s not as flexible as the subprocess module.

Example-2: Using os.popen() method

If you need to capture the output of the command, you can use os.popen() method instead, it is similar to os.system(), but it also allows you to capture the output of the command and return it as a file-like object.

import os

output = os.popen("ls -l").read()
print(output)

In this example, the os.popen() function is used to run the shell command ls -l and capture the output, output variable contains the captured output of the command, which in this case will be the output of the ls -l command, that is a list of files in the current directory along with their permissions, owner, group, size and last modification date.

It’s important to note that the os.popen() method is also not recommended to use because it’s not secure, it’s not as flexible as the subprocess module and it’s been deprecated since python 3.

Method-3: Using sh library

The sh library is a python library that allows you to run shell commands as if you were typing them in the terminal. Here is an example of using sh to run a shell command inside Python:

from sh import ls

output = ls("-l")
print(output)

In this example, we use the ls command from the sh library to run the command ls -l. The output of the command is captured in the output variable, which is an instance of the sh’s RunningCommand class. This class has a stdout attribute that contains the output of the command as a string.

If you get ModuleNotFoundError: No module named 'sh' while executing this program then you can manually install this module using pip:

~]# pip3 install sh
Collecting sh
  Downloading sh-1.14.3.tar.gz (62 kB)
     |████████████████████████████████| 62 kB 801 kB/s 
Using legacy 'setup.py install' for sh, since package 'wheel' is not installed.
Installing collected packages: sh
    Running setup.py install for sh ... done
Successfully installed sh-1.14.3

You can use the dir() function to get a list of all the functions (and other attributes) available in the sh library:

from sh import ls

# Get a list of all the attributes of the sh library
sh_attributes = dir(ls)

# Filter the list of attributes to get only the functions
sh_functions = [attr for attr in sh_attributes if callable(getattr(ls, attr))]

print(sh_functions)

Output:

['__call__', '__class__', '__delattr__', '__dir__', '__enter__', '__eq__', '__exit__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__unicode__', '__weakref__', '_extract_call_args', 'bake', 'thread_local']

Some practical examples executing shell commands in Python

Example-1: Run shell command with a variable

In this example we will run a shell command with a variable defined inside our python program:

import subprocess

# Define the variable
file_name = "example.txt"

# Run the command
return_code = subprocess.call(["cat", file_name])

# check the return code
if return_code == 0:
    print("Command executed successfully")
else:
    print("Command failed with return code", return_code)

In this example, we define a variable file_name with the value “example.txt”. Then we use the subprocess.run() function to run the shell command cat with the variable file_name as an argument.

Example-2: Run shell commands in background and wait for them to compete

In this example, we use the subprocess.Popen() function to start two shell commands, sleep 10 and sleep 15, in the background. The Popen() function takes a list of arguments, where the first element is the command to run and the rest of the elements are its arguments.

The Popen object returned by the function has a pid attribute which is the process ID of the command that was started. We print the pid of the command as and when it starts.

Then we use the wait() function to wait for each command to complete. The wait() function blocks the execution of your Python script until the command finishes. Once the command completes the returncode attribute of the Popen object contains the return code of the command, which is 0 if the command succeeded, or a non-zero value if the command failed.

import subprocess

# Start the first command
process1 = subprocess.Popen(["sleep", "10"])
print(f"command1 started with pid {process1.pid}")

# Start the second command
process2 = subprocess.Popen(["sleep", "15"])
print(f"command2 started with pid {process2.pid}")

# Wait for the first command to complete
process1.wait()
print(f"command1 with pid {process1.pid} completed with return code {process1.returncode}")

# Wait for the second command to complete
process2.wait()
print(f"command2 with pid {process2.pid} completed with return code {process2.returncode}")

Output:

~]# python3 script.py 
command1 started with pid 84082
command2 started with pid 84083
command1 with pid 84082 completed with return code 0
command2 with pid 84083 completed with return code 0

On another terminal we can verify the processes are running in background with same PID:

~]# ps -ef | grep sleep
root       84082   84081  0 23:08 pts/2    00:00:00 sleep 10
root       84083   84081  0 23:08 pts/2    00:00:00 sleep 15

Example-3: Run shell command and get STDOUT and STDERR separately

n this example, we use the subprocess.run() function to run a shell command and capture the output and error. The stdout and stderr arguments are set to subprocess.PIPE, which means that the output and error of the command will be captured and returned in the stdout and stderr attributes of the CompletedProcess object returned by the run() function.

The output and error are returned as bytes, so we use the decode() function to convert them to strings.

import subprocess

# Run the command and capture the output and error
result = subprocess.run(["ping", "-c", "2", "google.com"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)

# Print the output
print("stdout:", result.stdout.decode())

# Print the error
print("stderr:", result.stderr.decode())

Success Output:

stdout: PING google.com (142.250.194.142) 56(84) bytes of data.
64 bytes from del12s05-in-f14.1e100.net (142.250.194.142): icmp_seq=1 ttl=58 time=27.4 ms
64 bytes from del12s05-in-f14.1e100.net (142.250.194.142): icmp_seq=2 ttl=58 time=31.4 ms

--- google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 27.410/29.401/31.393/1.991 ms

stderr:

Failure Output:

stdout: 
stderr: ping: bad address 'google.com'

Example-4: Concatenate multiple shell commands inside Python

You can concatenate multiple shell commands together using the subprocess module. Here is an example of how to concatenate multiple shell commands using the subprocess.run() function:

import subprocess

# Concatenate the commands using the shell operator "&&"
subprocess.run("command1 && command2 && command3", shell=True)

In this example, we use the subprocess.run() function to concatenate three shell commands command1, command2 and command3 using the shell operator &&. The shell=True argument is used to indicate that the command is a shell command.

It’s important to note that when using the shell=True argument, the command is passed directly to the shell and shell-specific features such as variable expansion and wildcard expansion will work as expected. This could lead to security issues if you are not careful when handling user-supplied input.

You can also use the ; operator to concatenate multiple shell commands:

import subprocess

# Concatenate the commands using the shell operator ";"
subprocess.run("command1; command2; command3", shell=True)

When concatenating shell commands, the return code of the command is the return code of the last command in the concatenation.

You can also concatenate multiple shell commands using subprocess.Popen() function as shown below:

import subprocess

# Start the first command
process1 = subprocess.Popen(["command1", "arg1", "arg2"])

# Start the second command
process2 = subprocess.Popen(["command2", "arg1", "arg2"])

# Start the third command
process3 = subprocess.Popen(["command3", "arg1", "arg2"])

# Wait for the first command to complete
process1.wait()

# Wait for the second command to complete
process2.wait()

# Wait for the third command to complete
process3.wait()

In this example, we use the subprocess.Popen() function to start three shell commands, command1, command2 and command3 in a sequential order. The Popen() function takes a list of arguments, where the first element is the command to run and the rest of the elements are its arguments.

Summary

In this tutorial we have provide in-depth information on running shell commands inside Python program. We have multiple modules available for this purpose such as subprocess, os and sh library but subprocess is considered the most recommended module to execute shell commands due to the kind of flexibility it provides.

We also covered various practical example scenarios which user face in their day to day work life when working with shell commands inside Python.

References

subprocess — Subprocess management — Python 3.11.1