Skip to content

Readme.md

Latest commit

 

History

History
138 lines (105 loc) · 8.62 KB

Readme.md

File metadata and controls

138 lines (105 loc) · 8.62 KB
Raw

Python Automation for Security

Turn repetitive security workflows into reliable Python — file wrangling, log analysis, recon, OSINT, tool orchestration, scheduling, and remote automation.

Part of the [[Readme|Python for Security Professionals]] course.

Overview

This is the practical heart of the course: using Python as glue to automate real defensive and offensive tasks. You will build small, runnable scripts that walk file systems, mine logs for attacks, sweep a subnet, gather public OSINT, wrap tools like nmap and whois, run jobs unattended, and drive remote hosts over SSH — all with the standard library plus a few well-known packages.

Learning Objectives

  • Recognise when to automate and how to structure resilient, idempotent glue code.
  • Enumerate and transform files at scale with pathlib, os, and glob.
  • Extract attack signals from logs using re and collections.Counter.
  • Discover live hosts and open ports with socket and ipaddress.
  • Gather public information ethically with requests and BeautifulSoup.
  • Safely wrap external tools with subprocess (no shell=True injection).
  • Schedule scripts with cron, the schedule library, and drift-free sleep loops.
  • Automate remote command execution and file transfer with paramiko.
  • Speed up I/O-bound work with threads and processes.

Prerequisites

  • [[Scripts,-Modules,-Packages,-and-Libraries/Readme|Modules & Packages]] — subprocess, os, and the concurrency modules.
  • [[Error-and-Exception-Handling/Readme|Error & Exception Handling]] — automation must survive failures.
  • [[Input-Output-File-Handling/Readme|Input/Output File Handling]] — reading target lists and writing reports.
  • [[Advanced-Python-Modules/Readme|Advanced Python Modules]] — re and Counter for log analysis.

Topics Covered

  • [[Automating-Repetitive-Tasks|Automating Repetitive Tasks]] — the scripting philosophy and glue-code patterns
  • [[File-System-Automation|File System Automation]] — walk trees, batch-rename, find files by pattern
  • [[Log-Parsing-and-Analysis|Log Parsing and Analysis]] — mine auth/access logs for failed logins with re + Counter
  • [[Subprocess-for-System-Commands|Subprocess for System Commands]] — wrap nmap/whois and parse their output safely
  • [[Bash-Script-in-Python|Bash Script in Python]] — drive shell scripts from Python
  • [[Network-Reconnaissance-Scripts|Network Reconnaissance Scripts]] — host discovery and subnet sweeps with socket/ipaddress
  • [[Web-Scraping-for-OSINT|Web Scraping for OSINT]] — gather public info with requests + BeautifulSoup (ethically)
  • [[SSH-Automation|SSH Automation]] — remote commands and file pulls with paramiko
  • [[Scheduling-Automated-Tasks|Scheduling Automated Tasks]] — cron, the schedule library, and time.sleep loops
  • [[Threading-Module|Threading Module]] — concurrency for I/O-bound automation
  • [[Multiprocessing-Module|Multiprocessing Module]] — parallelism for CPU-bound work
  • [[concurrent.futures-(ThreadPoolExecutor)-Module|ThreadPoolExecutor]] — high-level thread pools for I/O-bound sweeps
  • [[concurrent.futures-(ProcessPoolExecutor)-Module|ProcessPoolExecutor]] — high-level process pools for CPU-bound work

Practical Exercises

  • Build a resilient batch runner that processes a target list and logs per-item success/failure.
  • Write a failed-login detector for your own auth.log and flag brute-force source IPs.
  • Sweep 127.0.0.0/30 for open ports, then start a listener and confirm it is detected.
  • Scrape https://example.com politely: check robots.txt, extract links, rate-limit.
  • Wrap nmap against 127.0.0.1 and parse open ports into a table; then schedule it via cron.
  • Automate SSH to localhost: run commands and pull a file back over SFTP.

Security Applications

The distinction between these four techniques matters, and choosing wrongly is the most common mistake in this module:

Technique Use when Module
Automation A task is repetitive and deterministic os, pathlib, shutil
Concurrency (threads) The bottleneck is waiting — network, disk threading, ThreadPoolExecutor
Multiprocessing The bottleneck is CPU — hashing, parsing, crypto multiprocessing, ProcessPoolExecutor
External command execution An existing tool already does the job well subprocess
  • File system automation — walking trees to find world-writable files, SUID binaries, or stray credential files.
  • Log parsing and analysis — extracting failed logins with re, counting sources with Counter, and correlating by timestamp.
  • Network reconnaissance — host discovery and subnet sweeps built on socket and ipaddress, parallelised with a thread pool.
  • Web scraping for OSINT — gathering public information with requests and BeautifulSoup, respecting robots.txt and rate limits.
  • Wrapping existing tools — driving nmap or whois through subprocess and parsing their structured output.
  • Scheduled monitoring — running an integrity check or certificate-expiry sweep on a cron schedule.

[!warning] Authorized reconnaissance only Every reconnaissance and scanning example in this module must only be pointed at systems you own or are explicitly authorized to test. Use 127.0.0.1, a lab VM, or a private test network. Concurrency multiplies the traffic you generate — rate-limit deliberately, and remember that automated scanning is both loud and easily logged.

Common Mistakes

  • shell=True with untrusted input — command injection. Always pass an argument list to subprocess.run().
  • Using threads for CPU-bound work — the GIL means no speedup; use processes.
  • Using processes for I/O-bound work — startup cost outweighs any gain.
  • No timeout on a subprocess or socket, so one hung target blocks the entire run.
  • Ignoring a subprocess exit code, treating a failed tool as a clean result.
  • Unbounded concurrency — thousands of workers exhaust file descriptors and hammer the target.
  • Scraping without rate limiting or ignoring robots.txt.
  • Missing the __main__ guard with multiprocessing, causing recursive process spawning.
  • Hardcoding credentials in an automation script instead of reading them from the environment.

Best Practices

  • Choose threads for waiting and processes for computing; measure with [[Timeit-Module|timeit]] rather than guessing.
  • Always pass a list to subprocess.run() and always set a timeout.
  • Bound worker counts explicitly and add a deliberate delay between requests.
  • Enforce an authorization check before any code that generates traffic.
  • Log every action with a timestamp so the run is auditable and reproducible.
  • Prefer a tool's machine-readable output (-oX, --json) over scraping human-readable text.
  • Make scheduled jobs idempotent so a re-run does not duplicate results.

Review Questions

  1. What is the difference between concurrency and parallelism, and which does threading provide in CPython?
  2. When should you use ProcessPoolExecutor rather than ThreadPoolExecutor?
  3. What security issue can occur if subprocess is called with shell=True and a user-supplied target?
  4. How would you troubleshoot an automation run that finishes far too quickly and reports nothing?
  5. Why must reconnaissance scripts state their authorization scope, and how would you enforce it in code?
  6. Write a Python example that scans a list of hosts concurrently while limiting the run to 20 workers.

Commands

# Install the third-party libraries used in this module
pip install requests beautifulsoup4 paramiko schedule

# System tools the subprocess/recon notes wrap
sudo apt install nmap whois

# Run a scheduled check every 5 minutes (user crontab)
# */5 * * * * /usr/bin/python3 /full/path/check.py >> /tmp/check.log 2>&1

References

Related Notes

  • [[concurrent.futures-(ThreadPoolExecutor)-Module|concurrent.futures — ThreadPoolExecutor]]
  • [[concurrent.futures-(ProcessPoolExecutor)-Module|concurrent.futures — ProcessPoolExecutor]]
  • [[Paramiko]] — SSH/SFTP library reference
  • [[Readme|Python for Security Professionals]]

Navigation

  • Previous: [[Advanced-Python-Data-Structures/Readme|12. Advanced Python Data Structures]]
  • Home: [[Readme|Course Home]]
  • Next: [[Security-Tool-Development/Readme|14. Security Tool Development]]