Python Automation Must-Know: os, shutil, subprocess, requests, schedule, and json

media.tenor

Apart from automation libs like **Selenium**, **Playwright or BeautifulSoup **for the basic libs that we need to know are:

• os
• shutil
• subprocess
• requests
• schedule
• json

These 6 modules help you work with the file system, move or copy files, run system commands, interact with web APIs, schedule tasks, and handle data.

Let’s explore them with practical examples.

1. The os Module

The os module lets you interact with the operating system. You can work with directories, environment variables, and more.

Common Use Cases:

• Create or delete folders
• List files in a directory
• Rename or move files

Example:

import os

# Create a new folder (แต่สร้างได้แค่ level เดียวนะ)
os.mkdir("test_folder")

# Create folders หลายชั้นได้ในครั้งเดียว
# ใส่ exist_ok=True เพื่อป้องกันถ้ามี folder อยู่แล้วจะไม่พังนะ (มีอยู่แล้วก้อโอเค ไรเงี้ย...ไม่ต้องสร้างใหม่)
os.makedirs("uploads/images", exist_ok=True)

# Rename a file
os.rename("old_file.txt", "new_file.txt")

# List files in current directory
print(os.listdir("."))

# Remove a folder
os.rmdir("test_folder")

# Set path
full_path = os.path.join(settings.BASE_DIR, settings.MEDIA_ROOT, folder_relative_path)
json_file_path = os.path.join("certs", "google_cloud_key.json")

################################################################
# .env
# SECRET_KEY=my-secret

# เมื่อใช้งานร่วมกับ .env ผ่าน lib "python-dotenv" (ต้องใช้ร่วมกับ 2 libs นะ)
# main.py
from dotenv import load_dotenv
import os

load_dotenv()  # Load values from .env เข้า os.environ
print(os.environ.get('SECRET_KEY', '123456')  # Get value from .env (have default value too)
print(api_key = os.environ["API_KEY"])  # แบบนี้ถ้าไม่มี "API_KEY" จะ error นะ

2. The shutil Module

shutil is used for file and folder operations like copying, moving, and deleting.

Common Use Cases:

• Copy or move files
• Create backups
• Clean up temporary folders

Example:

import shutil

# Copy a file to backup
shutil.copy("data.txt", "backup/data_backup.txt")

# Move the backup to archive
shutil.move("backup/data_backup.txt", "archive/data_backup.txt")

# Delete archive folder
shutil.rmtree("archive")

3. The subprocess Module

The subprocess module lets you run system commands like you would in a terminal or command prompt.

Common Use Cases:

• Run shell or batch scripts
• Automate CLI tools
• Capture command output for logs

Example:

import subprocess

# Run a command and print output
result = subprocess.run(["echo", "Automation started"], capture_output=True, text=True)
print(result.stdout)

# Run shell command to list directory
subprocess.run("dir" if os.name == "nt" else "ls", shell=True)

# We can create function to run more comfortable.

def run_command(command, print_error=True):
    print(f'\nrunning command: {command}')
    process_output = subprocess.run(command, capture_output=True) # รันคำสั่ง + เก็บผลลัพธ์ทั้งหมด

    # convert result (stdout) + error (stderr) from Bytes to String UTF-8
    command_output = process_output.stdout.decode('utf-8')
    command_error_output = process_output.stderr.decode('utf-8')

    print(f'command return code {process_output.returncode}\n')
    print(command_output)
    if command_error_output and print_error:
        print(command_error_output)
        print('\n')

    if process_output.returncode != 0:
        return False, command_output, command_error_output
    return True, command_output, command_error_output

# Usage when using in AWS cli
# check if user already exists
result = run_command(
    ['aws', 'iam', 'get-user', '--user-name', SES_USER_NAME], print_error=False
)
if result[0]:
    print(f'user {SES_USER_NAME} already exists')
else:
    print(f'user {SES_USER_NAME} does not exist, creating ...')
    result = run_command(
        ['aws', 'iam', 'create-user', '--user-name', SES_USER_NAME]
    )
    if not result[0]:  # When creating not success.
        return None, None

    # Create access key for the user
    result = run_command(
        [
            'aws',
            'iam',
            'create-access-key',
            '--user-name',
            SES_USER_NAME,
        ]
    )
    if not result[0]:
        return None, None
...

4. The requests Module

requests lets you send HTTP requests. It's perfect for working with APIs or downloading data from the internet.

Common Use Cases:

• Download web content
• Interact with REST APIs
• Automate web data extraction

Example:

import requests

# Get GitHub user profile
response = requests.get("https://api.github.com/users/octocat")
if response.status_code == 200:
    data = response.json()  # แปลง response ให้เป็น json ไปเลย จะได้เป็น dict จัดการ data ได้ง่ายๆ
    print(data["login"], data["public_repos"])

5. The schedule Module

schedule helps you run tasks at specific intervals. It’s useful for creating automated jobs like reports or checks.

Common Use Cases:

• Daily or weekly task automation
• Run scripts at fixed times
• Background monitoring

Example:

import schedule
import time

# Define job function
def check_status():
    print("Checking system status...")

# Schedule it every hour
schedule.every().hour.do(check_status)

# Run scheduler loop
while True:
    schedule.run_pending()
    time.sleep(1)

6. The json Module

json allows you to parse and save data in JSON format. This is useful when working with APIs or config files.

Common Use Cases:

• Read/write settings files
• Save automation logs
• Handle API data
• Convert Python objects to JSON strings (for web or log output)

Example:

import json

# Example data
data = {"user": "admin", "timeout": 30}

#########################################################
# Write to a JSON file (using dump)
with open("config.json", "w") as f:
    json.dump(data, f)  # write python objects to a file

#########################################################
# Read from a JSON file
with open("config.json") as f:
    loaded_data = json.load(f)
    print("Loaded from file:", loaded_data)

    # Loaded from file: {'user': 'admin', 'timeout': 30}

#########################################################
# Convert to JSON string (using dumps)
json_string = json.dumps(data)
print("JSON string:", json_string)

# JSON string: {"user": "admin", "timeout": 30}

#########################################################
# Convert back from string to Python dict
data_from_string = json.loads(json_string)
print("Converted back to dict:", data_from_string)

# Converted back to dict: {'user': 'admin', 'timeout': 30}

จัดการกับไฟล์

• **dump** = writes Python objects (dict) to a file
• **load** = reads from file

จัดการกับ API response (จะมี s เพิ่มมาจ้า)

• **dumps (stringify) = แปลง obj to string (หรือ python dict to json นั่นแหละ เพื่อให้ส่งหรือบันทึกข้อมูลได้ง่าย) > `จำง่ายๆว่า dumps ให้เป็น string** เพือส่งเป็น payload`
• **loads** (parse) = แปลง string to object (หรือ json to python (dict) จ้า) > จำง่ายๆว่า loads ให้เป็น dict

Conclusion

With os, shutil, subprocess, requests, schedule, and json, you can build powerful and flexible Python automation scripts. These libraries cover file operations, web tasks, scheduling, and configuration handling.

See ya!!

If you think it’s useful for you, just clap your hands 👏 to be encouraged me.

GRASSROOT ENGINEER 😘