Skip to content

feat: add bounded cross-language pipelines - #15

Draft
reacher-z wants to merge 3 commits into
feat/durable-recoveryfrom
feat/pipeline-runtime
Draft

feat: add bounded cross-language pipelines#15
reacher-z wants to merge 3 commits into
feat/durable-recoveryfrom
feat/pipeline-runtime

Conversation

@reacher-z

Copy link
Copy Markdown
Owner

What

  • add lazy, bounded TypeScript and Python pipeline runtimes with ordered and completion-order delivery
  • support retry, timeout, stop, drop, dead-letter, cancellation, iterator cleanup, and a 2,048-stage safety ceiling
  • add eight shared cross-language conformance cases and package-level smoke coverage
  • make graph-policy extension handling lossless and hash-stable across TypeScript and Python
  • harden explicit-null validation so invalid configuration cannot silently inherit defaults
  • add the full-plan coverage audit, 107-task dependency registry, evidence gates, 30-minute progress scanner, implementation briefs, and launch/growth controls

Why

The previous runtime exposed parallel fan-out but did not provide the streaming, per-item stage composition required by the Graph Engineering plan. That left independent items synchronized behind barriers and left no shared contract for failure handling, ordering, or cancellation.

The implementation review also found three cross-cutting correctness risks: explicit null values could fall through to defaults in TypeScript, Python policy extensions could be consumed by Pydantic instead of preserved in the canonical hash, and an overflowing Python stage iterator was not guaranteed to close. This stack fixes those root causes rather than masking individual fixtures.

Impact

  • independent items can now flow through stages without a global barrier
  • concurrency and buffering remain bounded under slow-consumer backpressure
  • both SDKs produce matching results for the shared pipeline corpus
  • failed work is isolated according to an explicit policy instead of implicitly collapsing the run
  • delivery progress is now tracked against the complete master plan with auditable evidence requirements

Validation

  • TypeScript build, typecheck, lint, and all package tests: core 41, persistence 27, primitives 147, runtime 115, patterns 93, MCP 15, CLI 131
  • Python: 587 tests plus 2 subtests, Ruff, and strict mypy across 23 source files
  • shared conformance: 14 graph fixtures; ready queue; invalid/cancellation; 8 barriers; 12 routers; persistence; durable recovery; 2 bidirectional interop cases; 8 pipeline cases
  • fixture validation: 26 JSON fixtures, 1 graph hash, 1 checkpoint, and 14 durable JSON records
  • seven npm package-content and clean-install smoke checks
  • Python wheel/sdist inspection: 28/30 expected archive entries
  • documentation link check: 193 local Markdown links
  • production dependency audit: no known vulnerabilities
  • progress scanner: 17 tests plus 2 subtests, Ruff clean, registry has 107 unique tasks with no dangling dependencies or cycles
  • git diff --check

Scope

This is a stacked draft targeting feat/durable-recovery. It deliberately excludes the active Builder/YAML and durable-redaction implementation lanes; those will land as independently reviewed milestones. No release or merge is performed by this pull request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant