Skip to content
BranchesTags

Repository files navigation

    ____              __ __ _ ____           
   / __ \__  ______  / //_/(_) / /__  _____
  / / / / / / / __ \/ ,<  / / / / _ \/ ___/
 / /_/ / /_/ / /_/ / /| |/ / / /  __/ /    
/_____/\__,_/ .___/_/ |_/_/_/_/\___/_/     
           /_/

High-performance duplicate file finder

Find and remove duplicate files with BLAKE3 hashing, resumable scans, and 100% test coverage.

CI PyPI Python License Coverage

Features • Installation • Quick Start • Documentation • Benchmarks

Why dupkiller?

dupkiller is designed for real-world large-scale deduplication:

  • Well-tested — 100% test coverage, 336 tests
  • Resumable — Interrupt and resume from exact stage
  • Memory-efficient — O(1) memory for 1TB+ datasets via keyset pagination
  • HDD-aware — Auto-detects HDDs, optimizes I/O patterns
  • Comprehensive — Multiple deletion strategies, multiple export formats
  • Production-ready — Type-safe, zero linting errors, CI/CD included

Compared to older tools (fdupes, rdfind), dupkiller offers modern architecture and better performance on large datasets.

Features

Core Capabilities

  • File hashing — BLAKE3 (3 GB/s) + blake2b fallback
  • Smart pre-filtering — Partial sampling before full hash
  • Resumable scans — SQLite checkpointing, resume on Ctrl+C
  • Multiple strategies — Keep oldest/newest/shortest/longest/lexicographic
  • Dry-run mode — Preview deletions safely
  • Export formats — JSONL, CSV, TXT, HTML reports

Performance & Reliability

  • Keyset pagination — O(1) memory regardless of dataset size
  • Streaming output — No in-memory buffering
  • Bounded queues — Prevents OOM on millions of files
  • PID locking — Prevents concurrent scans
  • Crash-safe — Survive interruptions gracefully

HDD Optimization

  • Auto-detection — Identifies mechanical drives
  • Inode sorting — Minimizes random seeks
  • Adaptive concurrency — Reduces parallelism on saturated I/O
  • I/O throttling — Optional --max-throughput limiting

Code Quality

  • 100% test coverage — 336 tests, all public APIs covered
  • Type-safe — Strict mypy with zero violations
  • CI/CD — Automated tests on Linux and macOS
  • Zero linting errors — Ruff clean
  • Bilingual docs — English and Russian

Installation

From PyPI (recommended)

pip install dupkiller

With BLAKE3 support (5× faster hashing)

pip install dupkiller blake3

From source

git clone https://github.com/deadboizxc/dupkiller.git
cd dupkiller
pip install -e .

Requirements

  • Python 3.11+
  • Linux, macOS (Windows: partial support)

Quick Start

Find duplicates

# Scan directory and save results
dupkiller scan /path/to/directory --output duplicates.jsonl
Example output 
Scanning /data/photos...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:42
Scanned: 127,543 files (2.3 TB)
Candidates: 8,234 files (450 GB)
Hashing: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:02:15
Found 412 duplicate groups
Reclaimable: 89.3 GB

Results written to: duplicates.jsonl

List duplicates

# View duplicate groups from last scan
dupkiller list

# Or view from specific output file
dupkiller list --output duplicates.jsonl
Example output 
┏━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Hash          ┃ Size     ┃ Count  ┃ Reclaimable   ┃
┡━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ a3f5e9d2...   │ 2.3 GB   │ 5      │ 9.2 GB        │
│ b8c4d1a7...   │ 450 MB   │ 3      │ 900 MB        │
│ c9e2f8b3...   │ 128 MB   │ 12     │ 1.4 GB        │
└───────────────┴──────────┴────────┴───────────────┘

Group 1: a3f5e9d2... (2.3 GB × 5 copies)
  /data/photos/2023/IMG_5678.CR2
  /data/photos/backup/IMG_5678.CR2
  /data/photos/archive/2023-12-01/IMG_5678.CR2
  ...

Delete duplicates

# Preview deletions (dry-run)
dupkiller delete --keep oldest --dry-run

# Actually delete (keep oldest file in each group)
dupkiller delete --keep oldest

# Or work with specific output file
dupkiller delete --output duplicates.jsonl --keep oldest
Keep strategies
  • oldest — Keep file with earliest modification time
  • newest — Keep most recently modified file
  • shortest-name — Keep file with shortest path (often canonical)
  • longest-name — Keep file with longest path (detailed naming)
  • lexicographic — Keep first in alphabetical order (deterministic)

Advanced usage

# Scan with resume support
dupkiller scan /data --session /tmp/scan.db --output dups.jsonl

# Resume interrupted scan
dupkiller scan /data --session /tmp/scan.db --output dups.jsonl  # same command

# Fine-tune concurrency (instead of generic --workers)
dupkiller scan /data --threads 8 --processes 16

# Exclude patterns
dupkiller scan /data --exclude '*.tmp' --exclude '__pycache__'

# Size filters
dupkiller scan /data --min-size 1MB --max-size 10GB

# Throttle I/O (useful for production servers)
dupkiller scan /data --max-throughput 100MB

# Force HDD-optimized mode
dupkiller scan /mnt/hdd --hdd --ionice

# Export to CSV or HTML
dupkiller export --format csv --output report.csv
dupkiller export --format html --output report.html

# Or export from specific JSONL file
dupkiller export --jsonl-input duplicates.jsonl --format csv --output report.csv

Cache management

# View cache statistics
dupkiller cache stats

# Clean up cache entries for deleted files
dupkiller cache clean

# Preview what would be cleaned (dry-run)
dupkiller cache clean --dry-run

# Clean in batches with custom size
dupkiller cache clean --batch-size 5000

Scan statistics and comparison

# View statistics from last scan
dupkiller stats

# List all available scans
dupkiller stats --list

# Compare current scan with an older scan
dupkiller stats --compare-with 42

Interactive deletion

# Delete with confirmation for each group
dupkiller delete --interactive --keep newest

# Delete only groups wasting more than 1 GB
dupkiller delete --min-waste 1073741824 --keep oldest

Performance Benchmarks

Throughput (single file hashing)

Algorithm Throughput Notes
BLAKE3 (with SIMD) 1-3 GB/s Requires blake3 package, CPU-dependent
BLAKE2b (fallback) 400-600 MB/s Built-in, no dependencies

Real-world scan speed depends on:

  • Disk I/O (NVMe: ~50K files/sec, HDD: ~5K files/sec)
  • File size distribution (many small files = slower)
  • Partial hashing optimization (skips 90%+ of full reads)

Note: Benchmarks vary by hardware. Your mileage may vary.

Scan Performance (example datasets)

Dataset Type Files Total Size Typical Scan Time*
Photo library 50K 500 GB 5-15 minutes
Code repositories 2M 50 GB 10-30 minutes
Video archive 5K 8 TB 30-90 minutes

*Actual time depends on hardware, duplicate ratio, and whether BLAKE3 is installed.

Memory Usage

dupkiller uses constant memory regardless of dataset size:

  • 100K files → ~50 MB RAM
  • 1M files → ~55 MB RAM
  • 10M files → ~60 MB RAM

Other tools often use O(n) memory, leading to OOM on large datasets.

Use Cases

1. Deduplicate Photo Library

# Find duplicate photos (ignoring EXIF differences)
dupkiller scan ~/Photos --output photo-dups.jsonl

# Keep newest version (likely edited)
dupkiller delete photo-dups.jsonl --keep newest

Result: Saved 45 GB by removing duplicate RAW files from backups.

2. Clean Up Backup Directories

# Scan multiple backup locations
dupkiller scan /backup/daily /backup/weekly /backup/monthly \
  --output backup-dups.jsonl

# Keep oldest (original) file
dupkiller delete backup-dups.jsonl --keep oldest

Result: Saved 1.2 TB by removing incremental backup duplicates.

3. CI/CD Artifact Cleanup

# Find duplicate build artifacts
dupkiller scan /var/lib/jenkins/artifacts --min-size 10MB \
  --output artifacts-dups.jsonl

# Keep shortest name (canonical artifact path)
dupkiller delete artifacts-dups.jsonl --keep shortest-name --dry-run

Result: Identified 12 GB of duplicate JARs and tarballs.

4. Resume Long-Running Scan

# Start scan (may take hours on 10TB dataset)
dupkiller scan /mnt/storage --session /tmp/scan.db --output dups.jsonl

# ... interrupted by Ctrl+C or system restart ...

# Resume from exact point (no re-scanning)
dupkiller scan /mnt/storage --session /tmp/scan.db --output dups.jsonl

Result: Scanned 10 TB across 3 sessions without re-work.

Architecture

dupkiller uses a 5-stage pipeline for efficient duplicate detection:

┌─────────────┐
│  1. SCAN    │  Recursive file discovery
└──────┬──────┘  → FileInfo(path, size, mtime, inode)
       │
       ▼
┌─────────────┐
│ 2. PARTIAL  │  Hash head+mid+tail (512KB each)
│   HASHING   │  → Quick pre-filter before full read
└──────┬──────┘
       │
       ▼
┌─────────────┐
│  3. FULL    │  Hash entire file contents
│   HASHING   │  → Only for partial hash collisions
└──────┬──────┘
       │
       ▼
┌─────────────┐
│ 4. GROUPING │  Group by content hash
└──────┬──────┘  → {hash: [file1, file2, ...]}
       │
       ▼
┌─────────────┐
│ 5. OUTPUT   │  Emit duplicate groups
└─────────────┘  → JSONL / CSV / HTML

Key Components

  • Scanner — Recursive file discovery with exclude patterns
  • HashCache — SQLite database with keyset pagination
  • ScanSession — Resumable state machine with PID locking
  • BoundedExecutor — Memory-safe worker pools
  • OutputHandler — Streaming JSONL writer with deduplication

Documentation

API Reference

from dupkiller.pipeline import run_pipeline
from dupkiller.cache import HashCache

# Programmatic scanning
cache = HashCache("~/.dupkiller/cache.db")
run_pipeline(
    roots=["/data"],
    cache=cache,
    output_jsonl="duplicates.jsonl",
    min_size=1024*1024,  # 1 MB
    workers=16,
)

Development

Setup

# Clone repository
git clone https://github.com/deadboizxc/dupkiller.git
cd dupkiller

# Install in development mode
pip install -e .[dev]

Testing

# Run tests with coverage
pytest tests/ --cov=dupkiller --cov-report=term-missing

# Run linter
ruff check dupkiller tests

# Run type checker
mypy dupkiller

Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

Quick checklist:

  • Yes Add tests for new features (maintain 100% coverage)
  • Yes Run ruff format and ruff check --fix
  • Yes Run mypy dupkiller (must pass with no errors)
  • Yes Update CHANGELOG.md
  • Yes Add docstrings for public APIs

Comparison with Other Tools

vs. fdupes

  • Yes 5× faster with BLAKE3 vs MD5
  • Yes Resumable scans (fdupes restarts from scratch)
  • Yes Memory-safe on large datasets (fdupes can OOM)
  • Yes Better UX with progress bars and rich output

vs. rdfind

  • Yes Parallel hashing (rdfind is single-threaded)
  • Yes Modern hashing (BLAKE3 vs SHA-1)
  • Yes Resumable scans
  • Yes Better testing (100% vs ~60% coverage)

vs. jdupes

  • Yes More keep strategies (5 vs 3)
  • Yes Export formats (JSONL, CSV, HTML)
  • Yes HDD optimization (inode sorting, adaptive concurrency)
  • Yes Type-safe (strict mypy vs untyped C)

Troubleshooting

"Out of memory" errors

dupkiller uses constant memory, but if you still encounter OOM:

# Reduce worker count
dupkiller scan /data --workers 4

# Reduce max throughput
dupkiller scan /data --max-throughput 50MB

Slow scanning on HDD

dupkiller auto-detects HDDs and optimizes for them, but you can help:

# Reduce concurrency for HDDs
dupkiller scan /mnt/hdd --workers 2

# Use ionice (Linux only)
ionice -c3 dupkiller scan /mnt/hdd

Resume not working

Check that you're using the same --session path:

# Yes Correct
dupkiller scan /data --session /tmp/scan.db
dupkiller scan /data --session /tmp/scan.db  # same path

# No Wrong (creates new session)
dupkiller scan /data --session /tmp/scan.db
dupkiller scan /data --session /tmp/scan2.db  # different path

License

Apache 2.0 License — see LICENSE for details.

Acknowledgments

Built with:

  • BLAKE3 — Modern cryptographic hashing
  • Click — Composable CLI framework
  • Rich — Beautiful terminal output
  • pytest — Test framework

Inspired by: fdupes, rdfind, jdupes

Star History

If dupkiller saved you disk space, consider giving it a star!

About

No description, website, or topics provided.

Resources

Readme
Apache-2.0 license

Contributing

Contributing
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages