File Integrity Monitor

A host-based integrity monitor that hashes a directory tree into a baseline, then detects added, removed, and modified files on later runs, using hashlib and json.

Project Overview

File Integrity Monitoring (FIM) is a core detective control: you record a cryptographic hash of every important file once, then periodically re-hash and compare. Any drift means a file changed — potentially a dropped web shell, a tampered binary, or an unauthorized config edit. This project walks a directory with pathlib, hashes each file with SHA-256 in streaming chunks, stores a JSON baseline, and diffs current state against it.

[!warning] Monitor systems you administer Run FIM on hosts and directories you own or manage. Store the baseline out-of-band (a file an attacker can also edit is not trustworthy).

Objectives

Build a cryptographic baseline of every file in a directory tree.
Detect added, removed, and modified files against that baseline.
Record metadata alongside content hashes.
Run repeatedly on a schedule with stable, comparable output.
Store the baseline so that tampering with it is detectable.

Requirements

Item	Detail
Python	3.9 or newer
Dependencies	`hashlib`, `pathlib`, `json`, `argparse` (standard library)
Privileges	Read access to every watched path; root for system directories
Storage	A location for the baseline — ideally off-host or read-only
Target	Directories on systems you own or are authorized to monitor

Architecture

CLI — argparse subcommands: baseline (create) and check (compare).
Walker — Path.rglob("*") enumerates files under the target root.
Hasher — reads each file in 64 KB chunks so large files don't blow up memory.
Store — baseline persisted as {relative_path: sha256} JSON.
Differ — set math over path keys plus hash comparison classifies added/removed/modified.

baseline: walk ─▶ hash each file ─▶ write baseline.json
check:    walk ─▶ hash each file ─▶ diff vs baseline ─▶ report added/removed/modified

Directory Structure

file-integrity-monitor/
├── fim.py              # CLI (baseline / check)
└── baseline.json       # generated hash database

Implementation Plan

Hash one file. Read in binary chunks and feed them to hashlib.sha256().
Walk the tree. Use Path.rglob("*"), filtering to regular files.
Record metadata. Capture size, mode, owner, and mtime alongside the digest.
Persist the baseline. Write JSON keyed by path relative to the watched root, so the baseline survives being moved.
Compare. Use set operations on the key sets for added and removed files, then compare digests for the intersection.
Handle exclusions. Support glob patterns so caches, logs, and temporary files do not generate constant noise.
Report and exit. Print a clear change summary and return non-zero when anything changed.
Schedule it. Run under cron or a systemd timer, appending results to a log.

Complete Example Implementation

#!/usr/bin/env python3
"""fim.py - File Integrity Monitor: hash baseline + change detection.

Monitor directories you own or administer.
"""
import argparse
import hashlib
import json
import sys
from pathlib import Path

CHUNK = 65536   # 64 KB streaming read


def hash_file(path: Path) -> str:
    """Return the SHA-256 hex digest of a file, read in chunks."""
    h = hashlib.sha256()
    with path.open("rb") as fh:
        for block in iter(lambda: fh.read(CHUNK), b""):
            h.update(block)
    return h.hexdigest()


def snapshot(root: Path) -> dict[str, str]:
    """Map each file (relative path) under root to its SHA-256."""
    result: dict[str, str] = {}
    for path in sorted(root.rglob("*")):
        if path.is_file():
            rel = str(path.relative_to(root))
            try:
                result[rel] = hash_file(path)
            except OSError as exc:
                print(f"[!] skip {rel}: {exc}", file=sys.stderr)
    return result


def cmd_baseline(root: Path, db: Path) -> int:
    data = snapshot(root)
    db.write_text(json.dumps(data, indent=2))
    print(f"[+] Baseline of {len(data)} file(s) written to {db}")
    return 0


def cmd_check(root: Path, db: Path) -> int:
    if not db.exists():
        print(f"[!] No baseline at {db}; run 'baseline' first.", file=sys.stderr)
        return 2
    old = json.loads(db.read_text())
    new = snapshot(root)

    old_keys, new_keys = set(old), set(new)
    added = new_keys - old_keys
    removed = old_keys - new_keys
    modified = {k for k in old_keys & new_keys if old[k] != new[k]}

    for path in sorted(added):
        print(f"[+] ADDED    {path}")
    for path in sorted(removed):
        print(f"[-] REMOVED  {path}")
    for path in sorted(modified):
        print(f"[~] MODIFIED {path}")

    total = len(added) + len(removed) + len(modified)
    print(f"\n[*] {total} change(s) detected.")
    return 1 if total else 0


def main() -> int:
    parser = argparse.ArgumentParser(description="File Integrity Monitor.")
    parser.add_argument("command", choices=["baseline", "check"])
    parser.add_argument("root", help="directory to monitor")
    parser.add_argument("--db", default="baseline.json", help="baseline file")
    args = parser.parse_args()

    root, db = Path(args.root), Path(args.db)
    if not root.is_dir():
        print(f"[!] Not a directory: {root}", file=sys.stderr)
        return 2
    return cmd_baseline(root, db) if args.command == "baseline" else cmd_check(root, db)


if __name__ == "__main__":
    raise SystemExit(main())

python fim.py baseline /etc/myapp --db baseline.json
# ...later, after a change...
python fim.py check /etc/myapp --db baseline.json

Output

[~] MODIFIED config.yaml
[+] ADDED    uploads/shell.php
[-] REMOVED  old.conf

[*] 3 change(s) detected.

Usage

# Create the baseline
python3 fim.py baseline /etc --output /var/lib/fim/etc.json

# Check against it
python3 fim.py check /etc --baseline /var/lib/fim/etc.json

# With exclusions
python3 fim.py baseline /var/www --exclude '*.log' --exclude 'cache/*'

# JSON report for a monitoring pipeline
python3 fim.py check /etc --baseline /var/lib/fim/etc.json --json

Example Output

$ python3 fim.py baseline ./watched --output baseline.json
[*] Hashed 3 files in ./watched
[*] Baseline written to baseline.json

$ python3 fim.py check ./watched --baseline baseline.json
[*] 3 files checked, no changes detected

$ echo "tampered" >> ./watched/config.ini
$ touch ./watched/new.txt
$ rm ./watched/notes.txt
$ python3 fim.py check ./watched --baseline baseline.json

[!] MODIFIED  config.ini      (size 128 -> 137)
[+] ADDED     new.txt
[-] REMOVED   notes.txt

[*] 3 changes detected
$ echo $?
1

Error Handling

Condition	Exception	Response
Unreadable file	`PermissionError`	Record as unreadable and continue; report the count separately
File deleted mid-walk	`FileNotFoundError`	Skip and note it — the tree is live
Broken symlink	`OSError`	Record the link itself rather than following it
Baseline missing	`FileNotFoundError`	Explain that a baseline must be created first; exit 2
Baseline corrupt	`json.JSONDecodeError`	Refuse to run rather than reporting everything as changed; exit 2
Watched path missing	`FileNotFoundError`	Report and exit 2
Changes detected	—	Not an error, but exit 1 so schedulers can alert

Security Considerations

[!warning] Authorized use only Monitor only directories on systems you own or are explicitly authorized to monitor.

The baseline is the trust anchor. An attacker who can write to it can hide their changes entirely. Store it off-host, on read-only media, or sign it — a baseline sitting world-writable beside the watched tree provides no assurance.
Use SHA-256 or better. MD5 and SHA-1 have practical collision attacks, so a modified file could be crafted to match its recorded digest.
Content hashes miss metadata changes. A permission change from 0644 to 4755 is a privilege-escalation vector that a content digest alone will not catch — record mode and ownership too.
This is a periodic monitor, not a real-time control. A file changed and changed back between runs is invisible; consider auditd or inotify where that matters.
Time-of-check to time-of-use. A file can change between hashing and reporting.
Reports are sensitive — they map the filesystem and reveal which paths are monitored.

Testing

import json

import fim


def test_detects_modification(tmp_path):
    watched = tmp_path / "watched"
    watched.mkdir()
    target = watched / "a.txt"
    target.write_text("original", encoding="utf-8")

    baseline = fim.build_baseline(watched)
    assert fim.compare(watched, baseline) == {"added": [], "removed": [], "modified": []}

    target.write_text("tampered", encoding="utf-8")
    result = fim.compare(watched, baseline)
    assert result["modified"] == ["a.txt"]


def test_detects_add_and_remove(tmp_path):
    watched = tmp_path / "watched"
    watched.mkdir()
    (watched / "a.txt").write_text("a", encoding="utf-8")

    baseline = fim.build_baseline(watched)
    (watched / "b.txt").write_text("b", encoding="utf-8")
    (watched / "a.txt").unlink()

    result = fim.compare(watched, baseline)
    assert result["added"] == ["b.txt"]
    assert result["removed"] == ["a.txt"]


def test_baseline_roundtrips(tmp_path):
    watched = tmp_path / "w"
    watched.mkdir()
    (watched / "x").write_text("x", encoding="utf-8")
    path = tmp_path / "base.json"
    fim.save(fim.build_baseline(watched), path)
    assert fim.load(path) == fim.build_baseline(watched)

Cover: no change, modification, addition, removal, an unreadable file, and a corrupt baseline.

Extension Ideas

Metadata tracking — also record size, mode, owner, and mtime to catch permission changes.
HMAC baseline — sign the baseline with an HMAC key so tampering is detectable.
Scheduling & alerting — run under cron/systemd-timer and email/syslog on drift.
Ignore lists — glob-based excludes for logs and caches that legitimately change.
Watch mode — use watchdog for real-time filesystem event monitoring instead of polling.

Troubleshooting

Symptom	Likely cause	Fix
Everything reports MODIFIED	Absolute paths stored, then the tree moved	Key the baseline on paths relative to the watched root
Constant noise from the same files	Logs, caches, and temporary files included	Add exclusion patterns
Very slow on a large tree	Whole files read into memory	Hash in chunks with `update()`
`PermissionError` aborts the run	One unreadable file raising	Catch per file and continue
Digest differs from `sha256sum`	File opened in text mode	Open with `"rb"`
Baseline empty	`glob()` used instead of `rglob()`	`glob()` does not recurse

References

[[Hash-Cracker]]
[[Log-Analyzer]]
[[Password-Generator]]
[[Hashlib-Module]]
[[Mini-Projects/Readme|Mini-Projects]] — module index
[[Readme|Python for Security Professionals]] — course home

Navigation Menu

Uh oh!

File Integrity Monitor

Project Overview

Objectives

Requirements

Architecture

Directory Structure

Implementation Plan

Complete Example Implementation

Output

Usage

Example Output

Error Handling

Security Considerations

Testing

Extension Ideas

Troubleshooting

References

Related

Uh oh!

FilesExpand file tree

File-Integrity-Monitor.md

Latest commit

History

File-Integrity-Monitor.md

File metadata and controls

File Integrity Monitor

Project Overview

Objectives

Requirements

Architecture

Directory Structure

Implementation Plan

Complete Example Implementation

Output

Usage

Example Output

Error Handling

Security Considerations

Testing

Extension Ideas

Troubleshooting

References

Related