Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 34 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ jobs:
run: offloader info

- name: Lint
run: ruff check src tests
run: ruff check src tests scripts build/windows

- name: Test
run: pytest -q --cov=offloader --cov-report=term --cov-report=xml
Expand Down Expand Up @@ -121,7 +121,40 @@ jobs:
python -m pip install --upgrade pip build twine
python -m build
python -m twine check dist/*
- name: Check that the source archive contains the Windows builder
run: >-
python -c "import glob, tarfile;
names = tarfile.open(glob.glob('dist/*.tar.gz')[0]).getnames();
assert all(any(n.endswith('/build/windows/' + f) for n in names)
for f in ('build.py', 'sign.py', 'offloader.spec', 'installer.nsi'))"
- name: Install and check the built wheel outside the checkout
run: python scripts/check_wheel.py dist/*.whl
- uses: actions/upload-artifact@v4
with:
name: dist
path: dist/

windows-bundle:
name: Windows unsigned desktop and installer
runs-on: windows-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: pip
- name: Install packaging dependencies
run: python -m pip install -e ".[gui]" -r requirements-build.txt
- name: Install NSIS
run: choco install nsis --version=3.12.0 -y --no-progress
- name: Build unsigned artifacts and check the frozen applications
run: python build/windows/build.py --clean --no-sign
- uses: actions/upload-artifact@v4
with:
name: offloader-windows-unsigned
path: |
dist/windows/Offloader-*.exe
dist/windows/Offloader-*.zip
dist/windows/Offloader-*-inventory.json
dist/windows/SHA256SUMS.txt
if-no-files-found: error
7 changes: 6 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,9 +1,14 @@
__pycache__/
*.py[cod]
*.egg-info/
build/
/build/*
!/build/windows/
/build/windows/__pycache__/
/.pyinstaller/
/.python-build/
dist/
.venv/
.venv-build/
venv/
.pytest_cache/
.coverage
Expand Down
186 changes: 186 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,186 @@
# Offloader agent guide

Offloader is a Python verified-copy tool for large one-way transfers, with a
CLI and a PySide6 desktop app. Camera-card offload is its primary workflow;
the data profile supports arbitrary files without media dependencies.

**Someone may reformat a camera card because Offloader said "Verified".**
Protecting the source, existing good copies, and the accuracy of that verdict
takes priority over features, throughput, and presentation.

## Start here

- Read [CONTRIBUTING.md](CONTRIBUTING.md) for development conventions.
- Before changing copy, verify, cleanup, or destination handling, read
[docs/data-safety.md](docs/data-safety.md). Its known limits are part of the
product's contract, not guarantees that have already been implemented.
- Use [ROADMAP.md](ROADMAP.md) for feature priorities and
[docs/release-plan.md](docs/release-plan.md) for the proposed Windows beta.
The release plan describes future work; verify the checkout before claiming
an installer, signing pipeline, or release gate exists.
- Inspect the working tree before edits. Preserve unrelated user changes.
Keep this guide concise and link to detailed documentation rather than
copying it wholesale.

## Working with Owen

- Minimize typing and manual effort. Complete authorized, reversible work
without repeatedly asking for confirmation. Ask only when a decision
materially changes the result; prefer clickable choices when available.
- Keep replies concise. Never use em dashes in written output.
- Put text intended for copying into fenced code blocks, one per paste target,
with labels and commentary outside the block.
- Use `rg` for content searches and `rg --files` for file discovery. Do not
use `grep`, `findstr`, or `Select-String`.
- Delegate bounded searches or implementation only when it saves net effort;
prefer an appropriate cheaper model and concise reports. Keep design
decisions and final diff review in the main agent. Pass search rules on.
- Do not steal focus, move the cursor, or launch interactive GUI verification
without authorization. Prefer headless checks. Never terminate user apps by
image name; target only a specific process launched for the task.
- Never add AI authorship, co-author trailers, or AI-session links to commits,
PRs, issues, documentation, or code comments.

## Code map

All module paths below are relative to `src/offloader/`.

| Area | Files and responsibility |
| --- | --- |
| Entry points | `cli.py`, `__main__.py`, `gui/app.py`; root `run.py` prefers this checkout's `src/`, launching the GUI without arguments and forwarding arguments to the CLI |
| Copy and control | `engine.py`: scanning, destination validation, fan-out, staging, verification, progress, `JobControl`, and CLI control-file support |
| Integrity | `hashers.py`, `integrity.py`, `verify.py`, `retry.py`, `longpath.py` |
| Shared data | `models.py`; keep engine and model code independent of Qt |
| Reports | `reports/` for PDF, CSV, MHL 1.1, and HTML; `ascmhl.py` for ASC MHL histories and directory hashes |
| Media | `probe.py`, `thumbs.py`, `braw.py`, `ixml.py`, `companions.py` |
| Timeline import | `timeline.py`: optional OpenTimelineIO integration, media resolution, and ambiguity handling |
| Desktop | `gui/main_window.py`, `gui/worker.py`, `gui/queue_view.py`, mode/editor widgets, and `gui/drives.py` |
| Persistent state | `config.py`, `presets.py`, `history.py`; `volumes.py` discovers storage and `naming.py` handles naming |

Python 3.10+ is supported. Core dependencies are xxhash and ReportLab; PySide6
is the GUI extra. Timeline dependencies are separate extras. ffmpeg/ffprobe
on PATH enable media metadata and thumbnails but are not required for copying
and verification. Do not make a missing media tool block a valid transfer.

## Safety invariants

- Keep destination validation in the engine so CLI, GUI, and library callers
share it. Never weaken source-overlap, duplicate-target, self-copy, or
flattened-name collision checks to make a workflow pass.
- Write to staged `.offloader-partial` files. Preserve verification before
promotion in verified modes and atomic replacement of final paths. Failure
or cancellation must not truncate an existing good copy or leave incomplete
data under a plausible final name. Cleanup must target only owned staging
files, never source media or an unrelated existing destination.
- Preserve the distinction between `none`, `source-only`, and `full`
verification. Full verification rereads the destination with cache-eviction
handling; failed eviction must remain visible. Do not claim this proves
physical persistence past a drive/controller cache.
- A skipped file is not newly checksum-verified. `skip_existing` currently
compares size. Warnings, missing files, failed destinations, and report
failures must not disappear into a blanket success verdict.
- Keep manifests portable and independently usable at each destination.
Preserve relative paths and per-destination results. Hex digests compare
case-insensitively; C4 identifiers compare exactly.
- Keep pause, resume, and cancellation responsive during copying and retry
waits. Malformed or temporarily unreadable control-file contents mean no
change of state, not cancellation. Use the existing `JobControl` machinery.
- Treat filenames, removable-media metadata, manifests, and configuration as
untrusted input. Keep parsers bounded, reject unsafe XML constructs, escape
HTML output, and guard against traversal and collisions.
- Timeline resolution must refuse ambiguous matches rather than choosing an
arbitrary same-named file. Read [docs/timeline.md](docs/timeline.md) before
changing its source/destination rules, which differ from card offload.
- Keep configuration under `config.py`'s per-user directory. Preserve atomic
writes and unreadable-config fallback. Tests must use temporary state rather
than modifying the user's presets or history.
- Do not represent known gaps as implemented protections: concurrent instances
are not coordinated, size-only skip is not a checksum check, and directory
structure hashes are not yet rechecked. Consult the current safety document
for the full list before changing claims.

## Development and validation

Install development dependencies, preferably in an existing project virtual
environment or a new local one:

```powershell
python -m pip install -e ".[dev]"
```

Inspect environment capabilities without launching the GUI:

```powershell
python run.py info
```

Lint:

```powershell
python -m ruff check src tests
```

Run the suite:

```powershell
python -m pytest -q
```

Run the deeper property checks when relevant to safety or parser changes:

```powershell
python -m pytest tests/test_fuzz.py tests/test_fuzz_edges.py tests/test_edge_cases.py --fuzz -q
```

Build Python distributions when packaging changes:

```powershell
python -m build
```

The build command requires the `build` package, which CI installs separately.
GUI tests set Qt's offscreen platform before importing PySide6; CI also sets
`QT_QPA_PLATFORM=offscreen`. Do not launch the desktop just to run tests.

Changes to copy, verify, or delete paths require a regression test that fails
without the fix. Start with affected tests, then run lint and the full suite
for code changes. Use temporary fixtures, injected failures, and disposable
media copies. Never test destructive behavior against original footage.
Documentation-only changes need link/content and diff checks, not a test run.

Relevant suites include `test_data_safety.py`, `test_engine.py`,
`test_verify.py`, `test_retry.py`, and `test_control.py`. Parser, report,
timeline, and GUI suites live alongside them in `tests/`. Prefer independent
reference outputs for formats and checksums over tests that mirror the writer.
See CONTRIBUTING for existing synthetic BRAW/BWF fixtures and failure helpers.

CI configuration is in [ci.yml](.github/workflows/ci.yml): cross-platform
tests, a Python minimum-version job, ffmpeg coverage, property-test soak, and
wheel/sdist validation. A configured job is not evidence of a passing run.
Report what was actually executed and any skips or environment limitations.

## Implementation, documentation, and releases

- Match surrounding code, use type hints on new public functions, and explain
non-obvious constraints in comments. Keep Qt imports inside the GUI layer
and long-running work outside the GUI thread.
- Update the relevant `docs/` file when behavior changes. Report geometry,
media parsing, ASC MHL, timeline resolution, and performance each have their
own reference documents linked from README. Update published test counts
only from actual results, not estimates.
- Measure performance before claiming improvement; follow
[docs/performance.md](docs/performance.md) and state cache/durability effects.
- `src/offloader/_version.py` is the version source. Setuptools reads its
literal through dynamic metadata; the package re-exports it and the Windows
bundle uses it for executable metadata. Do not add another version literal
or bump the version merely for documentation work.
- Alpha-OSK is a release-process reference, not a runtime dependency. Do not
copy its product IDs, signer configuration, update endpoint, elevation
behavior, or repository targets into Offloader.
- Follow the release plan's artifact and clean-machine gates when packaging
lands. Installation or updates must not force-stop an active transfer.
Preparing a release is distinct from publishing or sending announcements;
perform external actions only within the user's authorized scope.
- Treat data loss, false verification, and silent omissions as security issues.
Follow [SECURITY.md](SECURITY.md) for private reporting; do not publish
sensitive reproductions or contact anyone without authorization.
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,16 @@ project uses [semantic versioning][semver].

### Added

- Windows desktop/CLI bundles and an NSIS installer, with pinned dependencies,
embedded version metadata, signing by default, explicit unsigned CI builds,
source/file inventories, checksums, and headless artifact checks. Installation
uses an application lifetime lock and inventoried files for replacement,
rollback, and uninstall; configuration/history are preserved. Real signing
and clean-machine installation qualification remain pending.
- One release version source for Python package metadata, the application,
reports, and Windows executable metadata. CI also installs the built wheel
outside the checkout to check its CLI and version consistency.

- **Pause, resume and cancel from the command line.** `JobControl` has existed
since the desktop app needed transport buttons, and is checked once per 8 MiB
chunk, but the CLI never passed one — so a job started in a terminal could
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ pip install -e ".[dev]"
`ffmpeg` and `ffprobe` on `PATH` are optional — the suite runs without them.

```sh
pytest # 482 tests, about 20s
pytest # run the full suite
pytest --fuzz # property tests at 3000 examples each, about 3 min
ruff check src tests
pytest --cov=offloader --cov-report=term-missing
Expand Down
5 changes: 5 additions & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
include AGENTS.md CONTRIBUTING.md SECURITY.md ROADMAP.md CHANGELOG.md run.py
include requirements-build.txt
recursive-include build/windows *.py *.spec *.nsi
recursive-include scripts *.py
recursive-include docs *.md
10 changes: 7 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -514,6 +514,8 @@ general-purpose tool reports a filename, a size, and a placeholder icon.
| Document | What is in it |
| --- | --- |
| [`ROADMAP.md`](ROADMAP.md) | What is next, why, and what this will not become |
| [`docs/release-plan.md`](docs/release-plan.md) | Windows beta release sequence, packaging, signing, acceptance gates, and recovery |
| [`docs/build-windows.md`](docs/build-windows.md) | Build, sign, and check Windows desktop bundles and installers |
| [`docs/data-safety.md`](docs/data-safety.md) | Threat model: what is guaranteed, what is not, and the bugs behind each guarantee |
| [`docs/report-layout.md`](docs/report-layout.md) | Every coordinate of the PDF, measured off the reference report |
| [`docs/performance.md`](docs/performance.md) | Why not robocopy, with benchmarks and the confounds that made the first run worthless |
Expand Down Expand Up @@ -552,18 +554,20 @@ what makes the report layer testable without moving bytes.

```sh
pip install -e ".[dev]"
pytest # 482 tests, ~20s
pytest # run the full suite
pytest --fuzz # same suite, 3000 examples per property (~3 min)
ruff check src tests
pytest --cov=offloader --cov-report=term-missing
```

482 tests at 83% line coverage. They cover formatting against the reference's
727 tests passed with 5 skipped and 85% line coverage on Windows/Python 3.12
in the latest local run. They cover formatting against the reference's
exact strings, checksum vectors and streaming equivalence, copy/verify
behaviour including simulated destination corruption, pause/resume/cancel
concurrency, retry discrimination, BRAW container parsing, ffprobe parsing,
preset and history persistence, card detection, PDF geometry read back with
PyMuPDF, the CLI, and the GUI.
PyMuPDF, the CLI, the GUI, and Windows installation ownership, rollback,
locking, build provenance, and signing failure handling.

The GUI tests run on Qt's offscreen platform and drive the real queue
controller — the worker thread actually copies files — so they cover the wiring
Expand Down
5 changes: 5 additions & 0 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,11 @@ remains the fallback for everything the chunk retry cannot reach.

## Next

The proposed first packaged Windows beta is tracked in
[`docs/release-plan.md`](docs/release-plan.md), with packaging, signing,
integrity gates, a private pilot, and recovery steps. It adapts Alpha-OSK's
release process to Offloader's data-safety requirements.

### Verify what is already written

`offloader verify` checks file hashes. ASC MHL also records **directory content
Expand Down
Loading
Loading