[BobbyZhouZijian]

Motivation

Processors currently expose different policy and trace sample/batch types, and report assembly adds another chain of public wrappers. Unify the boundary as TrainingBatch.items: tuple[TrainDataItem, ...], so one batch can contain ATIF trajectories, Harbor tasks, or both. This is a maintainer-requested follow-up to #445.

Related: #16 covers broader declarative group rules and introspection; this change preserves recipe-defined readiness rules. Open PR #444 also modifies the CORAL processor and its tests, so those files will need coordination when merging.

Changes

• Remove PolicySample, TraceSample, and their three batch subclasses. TrainDataItem is TrajectoryItem | TaskItem; trajectories contain ATIF documents exclusively.
• Capture provider messages, tool calls, observations and multimodal content as ATIF. Preserve original exchanges, feedback and exact rollout tensors in extra.reef; policy backends read that extension, while harness methods read trajectories or recorded exchanges.
• Migrate all shipped consumers, recipes, tutorials and tests. Preserve masks, log probabilities, top-K capture, teacher candidates, runtime-load spans, advantages and scheduling.
• Remove ReportSample, the reported processor's Candidate, and BatchUnit, plus the TTTD/CORAL row wrappers. Recipes return training items directly; one private _PendingReport owns buffering and consumption state. Optional grouping(context) supplies collection-group/retry coordinates, and decide_group receives items directly.
• Preserve complete-step barriers, first-report slot deduplication, reservations, manual/hybrid modes, retention and acknowledgement of selected rows omitted from training output. Add contract tests for these boundaries and update API/developer documentation.

Compatibility and operational impact

This changes Python extension contracts: custom processors and consumers must migrate from the removed types to training items, and grouped report processors must supply grouping(context). Collection groups remain separate from training comparison groups. Old pickled batches containing removed classes are incompatible; service and worker code must be upgraded together. Existing raw records continue to replay through the processors.

Current built-in backends explicitly reject task items until a recipe provides task rollout support. Text-only ATIF does not imply generic tokenization: algorithms requiring captured tokens or log probabilities fail explicitly when they are missing. No new runtime dependency is added.

Verification

• pre-commit run --all-files: passed.
• npm run check:docs in docs/site: passed.
• Changed Python files parse with Python 3.10 syntax.
• Focused processor/recipe/trainer suites: 182 passed; expanded reported-feedback contract suite: 24 passed.
• GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_SYSTEM=/dev/null .venv/bin/python -m pytest tests/ -n 4 -q --tb=short --durations=10: 3436 passed, 48 skipped, 1 failed. The failure is test_bridge_catalogs_paired_checkpoint_metrics_and_blocks_before_second_optimizer: the local filesystem reservation blocks the first job. Reproduced on clean main 142e0781.
• .venv/bin/python -m mypy: the same four existing errors in unchanged lora_transport.py and checkpoint.py; no new errors.
• Six serialized SFT/SAO/OpenClawRL/TTTD/grouped-PG preparation and multi-epoch scheduling cases match main 142e0781 exactly.
• JSON-roundtrip and provider/tool/multimodal conversion contracts passed. Five emitted/imported documents validated against Harbor 0.20.0's ATIF schema in an isolated installation.
• No real GPU training run was performed.

AI assistance

OpenAI Codex implemented the refactor, migrated consumers and documentation, added contract tests, and ran the checks above under maintainer direction. The human contributor remains responsible for reviewing and understanding the changes.

Checklist

• The change is focused and contains no unrelated cleanup.
• Tests cover behavior changes, or this pull request does not change behavior.
• Public interface changes include contract tests, or no public interface changes are present.
• Affected user and developer documentation is updated, or no documentation update is required.
• Root README changes update both languages and README.i18n.yaml, or do not affect README content.
• An accepted RFC is linked, or this change does not require an RFC. (Maintainer-requested interface refactor; no separate RFC linked.)
• Relevant pre-commit and test checks pass. (Pre-commit passes; existing local pytest/mypy failures are detailed above.)
• Non-trivial AI assistance is disclosed above, or none was used.

Review and merge

The Merge Oncall is assigned automatically. See the
maintenance model
for review and merge requirements.

[github-actions]

Merge Oncall: @BobbyZhouZijian

The merge oncall will route reviews and keep this pull request moving toward a clear decision.