Readme.md

Advanced Python Data Structures

Advanced numeric, string, set, dictionary, and list techniques, plus generators, data classes, and property decorators — applied to security tooling.

Part of the [[Readme|Python for Security Professionals]] course.

Overview

This module moves past the basics into the data-structure and language features that make security scripts fast, memory-efficient, and maintainable. Each note ties a Python concept to a concrete offensive/defensive task — XOR obfuscation, scan diffing, log streaming, and modeling findings — with runnable examples and a short lab.

Learning Objectives

• Manipulate numbers at the bit level (XOR, masks, base conversion) for encoding and simple obfuscation.
• Move fluently between str and bytes, and shape text with format specs and translate.
• Use set algebra to diff hosts, ports, and wordlists between scans.
• Aggregate, group, and count parsed data with advanced dictionary idioms.
• Slice, sort, and flatten lists to rank and batch results.
• Stream multi-gigabyte logs and captures with generators in constant memory.
• Model hosts, findings, and credentials with data classes and validate them with property decorators.

Prerequisites

• [[Python-Objects-and-Data-Structure-Basics/Readme|Python Objects & Data Structure Basics]] — the types being extended here.
• [[Object-Oriented-Programming/Readme|Object-Oriented Programming]] — data classes and properties are class features.
• [[Methods-and-Functions/Readme|Methods & Functions]] — decorators and generators build on functions.

Topics Covered

• [[Advanced-Numbers-and-Math|Advanced Numbers and Math]] — math, bitwise ops, hex/bin, XOR for obfuscation
• [[Advanced-String-Operations|Advanced String Operations]] — encode/decode, bytes, format specs, translate
• [[Advanced-Set-Operations|Advanced Set Operations]] — union/intersection/difference for scan diffing
• [[Advanced-Dictionary-Techniques|Advanced Dictionary Techniques]] — comprehensions, merging, get/setdefault, grouping
• [[Advanced-List-Manipulation|Advanced List Manipulation]] — slicing, sorting with key, nested lists
• [[Generators-and-Iterators|Generators and Iterators]] — yield, generator expressions, memory-efficient streaming
• [[Data-Classes|Data Classes]] — @dataclass, field, frozen for modeling findings/hosts
• [[Property-Decorators|Property Decorators]] — @property, getters/setters, input validation

Practical Exercises

• Brute-force a single-byte XOR key and recover the plaintext.
• Diff two Nmap host lists with set operations to find new and disappeared hosts.
• Stream a large auth.log through a generator pipeline and tally failed logins with Counter.
• Model findings as frozen data classes, dedupe them in a set, and dump to JSON.

Security Applications

• Modelling findings — a @dataclass for Finding or Host gives you __init__, __repr__, and __eq__ for free, and frozen=True makes records hashable so duplicates collapse in a set.
• Validated attributes — @property setters reject an out-of-range port or a malformed IP at assignment time, so invalid objects never exist.
• Streaming large captures — generators process a multi-gigabyte log or pcap with constant memory, which is the difference between a tool that runs and one that is killed by the OOM killer.
• Diffing scan runs — set difference and intersection show exactly what appeared and disappeared between two enumeration passes.
• Bitwise operations — XOR, masks, and shifts for simple obfuscation exercises, flag fields, and subnet arithmetic.
• Grouping and merging results — dictionary comprehensions and | merging combine per-host results from several tools into one report.

Common Mistakes

• Mutable defaults in a dataclass — use field(default_factory=list), never = [].
• Consuming a generator twice — it is exhausted after the first pass; build a list if you need to reuse it.
• Forgetting frozen=True when you need a dataclass to be hashable.
• Property setters that recurse — assigning to self.x inside the setter for x instead of self._x.
• Building a list where a generator would do, defeating the memory benefit.
• Treating XOR as encryption — it is obfuscation, trivially reversible, and must never protect real data.
• Mutating a set while iterating over it.

Best Practices

• Use @dataclass for anything that mostly holds data; add frozen=True for value objects.
• Use field(default_factory=...) for every mutable default.
• Prefer generators for anything larger than memory, and document that the result is single-use.
• Put validation in a property setter so it happens exactly once.
• Use set operations rather than nested loops when comparing result sets.
• Choose the container by access pattern, and say why in a comment when it is not obvious.

Review Questions

1. What is the difference between a generator function and a generator expression?
2. When should you use a @dataclass rather than a plain class or a dictionary?
3. What security issue can occur if XOR is used to protect stored credentials?
4. How would you troubleshoot a generator that appears to produce results only the first time it is used?
5. Why does a dataclass need field(default_factory=list) instead of = []?
6. Write a Python example that streams a large log file and yields only lines matching a pattern.

Commands

# Explore these concepts interactively
python3 -q

# Time and measure memory of a streaming log parser
/usr/bin/time -v python3 stream_logs.py big.log

# Generate two host lists to diff with set operations
nmap -sn 10.0.0.0/24 -oG scan_old.gnmap
nmap -sn 10.0.0.0/24 -oG scan_new.gnmap

References

• Python docs — The Python Standard Library
• Python docs — Data structures tutorial
• Real Python — tutorials

Related Notes

• [[Object-Oriented-Programming/Readme|Object-Oriented Programming]] — classes underpin data classes and properties
• [[Readme|Python for Security Professionals]]

Navigation

• Previous: [[Advanced-Python-Modules/Readme|11. Advanced Python Modules]]
• Home: [[Readme|Course Home]]
• Next: [[Python-Automation-for-Security/Readme|13. Python Automation for Security]]