Lab File Integrity Monitor

Build a lightweight file integrity monitor (FIM) that baselines a directory with SHA-256 hashes and reports added, modified, and deleted files.

Warning

Monitor only directories you own or are authorized to watch. A FIM records file paths and hashes — keep the baseline database out of world-readable locations. Run this against a scratch folder, not /etc, for the lab. Only run these labs against systems you own or are explicitly authorized to test.

Objective

Walk a directory tree with pathlib.
Compute SHA-256 digests of file contents in fixed-size chunks.
Persist a baseline to disk as JSON.
Diff the current state against the baseline to detect change.

Learning Outcomes

After completing this lab you will be able to:

Walk a directory tree and hash every file without loading it entirely into memory.
Explain why SHA-256 is appropriate here and MD5 is not.
Persist and reload a baseline as JSON.
Classify differences as added, removed, or modified.
Describe the limits of a hash-based integrity monitor.

Prerequisites

[[Hashlib-Module|Hashlib Module]] — chunked hashing with update().
[[Pathlib-Module|Pathlib Module]] — walking a tree with rglob().
[[Working-with-CSV-and-JSON-Files|Working with CSV and JSON Files]] — persisting the baseline.
[[Input-Output-File-Handling/Readme|Input/Output File Handling]] — binary reads and context managers.

Lab Environment

Item	Detail
Python	3.8+
Modules	`hashlib`, `pathlib`, `json`, `argparse` (standard library)
Target	A scratch directory you create in Setup

Setup

mkdir -p ~/labs/fim && cd ~/labs/fim
python3 -m venv .venv
source .venv/bin/activate

# Create a small tree to watch:
mkdir -p watched/config watched/bin
echo "listen_port = 8080"      > watched/config/app.conf
echo "#!/bin/sh"               > watched/bin/start.sh
echo "hello world"             > watched/readme.txt

nano fim.py     # paste the Code section

Tasks

Hash one file. Write file_digest(path) that opens the file in binary mode and feeds fixed-size chunks to hashlib.sha256(). Verify the result matches sha256sum.
Walk the tree. Use Path.rglob("*") and filter to regular files with is_file().
Build a baseline. Produce a dictionary mapping relative path to digest, and write it as JSON.
Reload and compare. Read the baseline back and compute the current state.
Classify changes. Use set operations on the two key sets to find added and removed paths, then compare digests for the intersection to find modified files.
Report. Print each change with a clear marker and return a non-zero exit code when anything changed.

Complete Example Code

#!/usr/bin/env python3
"""Baseline a directory and detect integrity changes with SHA-256."""
import argparse
import hashlib
import json
from pathlib import Path

CHUNK = 65536  # 64 KiB


def sha256_of(path: Path) -> str:
    digest = hashlib.sha256()
    with path.open("rb") as handle:
        for block in iter(lambda: handle.read(CHUNK), b""):
            digest.update(block)
    return digest.hexdigest()


def snapshot(root: Path) -> dict[str, str]:
    """Map relative path -> sha256 for every file under root."""
    state: dict[str, str] = {}
    for path in sorted(root.rglob("*")):
        if path.is_file():
            state[str(path.relative_to(root))] = sha256_of(path)
    return state


def baseline(root: Path, db: Path) -> None:
    state = snapshot(root)
    db.write_text(json.dumps(state, indent=2))
    print(f"[+] Baselined {len(state)} files -> {db}")


def check(root: Path, db: Path) -> None:
    if not db.exists():
        print(f"[!] No baseline at {db}. Run with 'baseline' first.")
        return
    old = json.loads(db.read_text())
    new = snapshot(root)

    added = sorted(set(new) - set(old))
    removed = sorted(set(old) - set(new))
    changed = sorted(f for f in set(old) & set(new) if old[f] != new[f])

    if not (added or removed or changed):
        print("[+] OK — no changes since baseline.")
        return

    for f in added:
        print(f"[ADDED]    {f}")
    for f in removed:
        print(f"[DELETED]  {f}")
    for f in changed:
        print(f"[MODIFIED] {f}")


def main() -> None:
    parser = argparse.ArgumentParser(description="File integrity monitor.")
    parser.add_argument("action", choices=["baseline", "check"])
    parser.add_argument("root", help="Directory to monitor")
    parser.add_argument("--db", default="baseline.json", help="Baseline file")
    args = parser.parse_args()

    root, db = Path(args.root), Path(args.db)
    if args.action == "baseline":
        baseline(root, db)
    else:
        check(root, db)


if __name__ == "__main__":
    main()

Expected Output

$ python fim.py baseline watched
[+] Baselined 3 files -> baseline.json

# Now tamper with the tree:
$ echo "listen_port = 9999" > watched/config/app.conf   # modify
$ echo "id" >> watched/bin/start.sh                      # modify
$ rm watched/readme.txt                                  # delete
$ touch watched/config/new_secret.key                    # add

$ python fim.py check watched
[ADDED]    config/new_secret.key
[DELETED]  readme.txt
[MODIFIED] bin/start.sh
[MODIFIED] config/app.conf

Explanation

SHA-256 in chunks (iter(lambda: handle.read(CHUNK), b"")) hashes files of any size with a constant, small memory footprint — important for large binaries.
Relative paths as keys make the baseline portable: the same tree hashes identically regardless of where it lives on disk.
Set arithmetic gives the three change classes cheaply: new - old = added, old - new = deleted, and same-key-different-hash = modified.
The baseline is just JSON, so it is diff-able and human-readable — but in production it must be stored somewhere an attacker who compromises the host cannot silently rewrite (append-only log, remote store, or signed file).
A cryptographic hash detects content change but not metadata-only change (permissions, owner). Extend the record to catch those (see Challenges).

Validation

# 1. Create the baseline
python3 fim.py --baseline ~/labs/fim/watched

# 2. Re-run with nothing changed
python3 fim.py --check ~/labs/fim/watched

[*] 3 files checked, no changes detected

# 3. Modify, add, and delete a file, then re-check
echo "tampered" >> ~/labs/fim/watched/config.ini
touch ~/labs/fim/watched/new.txt
rm ~/labs/fim/watched/notes.txt
python3 fim.py --check ~/labs/fim/watched

[!] MODIFIED  config.ini
[+] ADDED     new.txt
[-] REMOVED   notes.txt

An unchanged run reports no differences.
A single appended byte is detected as MODIFIED.
Added and removed files are classified correctly.
The baseline JSON is human-readable and reloadable.
The exit code is non-zero when changes are found.

Challenges

Also record file size, mode, and mtime, and report permission/ownership changes.
Add a --exclude glob so noisy paths (logs, caches) are skipped.
Schedule check every 5 minutes with cron and email or log any drift.
Sign the baseline with an HMAC key so tampering with baseline.json itself is detectable.
Swap SHA-256 for BLAKE2b and benchmark the difference on a large tree with time.

Troubleshooting

Symptom	Likely cause	Fix
`PermissionError` while walking	Unreadable file in the tree	Wrap the hash call in `try`/`except PermissionError` and record it as unreadable
Every file reports MODIFIED	Absolute paths stored in the baseline, then the directory moved	Store paths relative to the watched root with `Path.relative_to()`
`IsADirectoryError`	Directories included in the walk	Filter with `if path.is_file()`
Baseline is empty	`glob()` used instead of `rglob()`	`glob()` does not recurse
Very slow on a large tree	Reading whole files into memory	Hash in chunks with `update()`
Digest differs from `sha256sum`	File opened in text mode	Open with `"rb"`

Security Notes

A hash monitor detects change, not intent. It tells you a file differs from the baseline; deciding whether that was a legitimate patch or an attacker is your job.
The baseline is the trust anchor. If an attacker can write to the baseline file, they can hide their changes. Store it off-host, or on read-only media, and protect its permissions.
Use SHA-256 or better. MD5 and SHA-1 have practical collision attacks, so a determined attacker could craft a modified file with a matching digest.
Hashes miss metadata changes. Permissions, ownership, and timestamps are not covered unless you record them too — worth adding for a real deployment.
Time-of-check to time-of-use. A file can change between hashing and reporting; this tool is a periodic monitor, not a real-time control.
Run this only against directories on systems you own or are authorized to monitor.

Cleanup

deactivate
rm -rf ~/labs/fim

Navigation Menu

Uh oh!

Lab File Integrity Monitor

Objective

Learning Outcomes

Prerequisites

Lab Environment

Setup

Tasks

Complete Example Code

Expected Output

Explanation

Validation

Challenges

Troubleshooting

Security Notes

Cleanup

Further Reading

Related

Uh oh!

FilesExpand file tree

Lab-File-Integrity-Monitor.md

Latest commit

History

Lab-File-Integrity-Monitor.md

File metadata and controls

Lab File Integrity Monitor

Objective

Learning Outcomes

Prerequisites

Lab Environment

Setup

Tasks

Complete Example Code

Expected Output

Explanation

Validation

Challenges

Troubleshooting

Security Notes

Cleanup

Further Reading

Related