[PRD] Deterministic, resilient, and efficient code review engine

[EmersonBraun]

PRD: Deterministic, Resilient, and Efficient Code Review Engine

Problem Statement

@agentskit/code-review is intended to become a free, open-source alternative to CodeRabbit, differentiated by the AgentsKit runtime, provider ecosystem, configurable review behavior, safe memory, and measurable continuous improvement.

The current release has strong fail-closed safeguards, immutable review evidence, idempotent GitHub comments, configuration validation, and a working single-PR review cycle. However, it is not yet reliable enough for unattended production operation:

• Multi-PR orchestration is still performed by an external LLM instead of one deterministic campaign command.
• The real cycle is controlled by a large procedural script while the existing state machine covers only part of the workflow.
• Successful review work can be discarded when another model execution in the same batch fails.
• Retries are based partly on aggregate stderr pattern matching instead of typed failures at the provider boundary.
• Large files or source-budget limits can block an entire pull request instead of producing complete changed-hunk coverage.
• Seven independent review lenses repeatedly send the same source, conventions, manifest, and schema to the model.
• The quality corpus and memory A/B evaluation are rerun for every pull request.
• The current quality matrix is created only for successful cycles, evaluates reliability as a single surviving run, and mixes synthetic corpus quality with real operational quality.
• The current memory path stores review conversations and source material, but does not provide a complete production feedback-to-learning lifecycle.
• GitHub-specific behavior is spread across source loading, reporting, CLI orchestration, state inspection, and merge logic, while the public configuration already suggests future GitLab support.

Real campaign evidence from the current implementation demonstrated the gap: seven pull requests were discovered, only one reached the quality matrix, and none completed review, decision, and merge as one uninterrupted autonomous campaign. The successful small pull request used 14 initial lens executions and approximately 963,000 review tokens for two files and ten changed lines, plus roughly 435,000 additional tokens for the quality corpus and memory A/B checks.

The core problem is therefore not another prompt adjustment. The product needs a deterministic, recoverable, provider-neutral engine with honest campaign-level quality measurement and strict token, time, and memory budgets.

Solution

Rebuild the internal execution architecture around one deterministic campaign engine. Orca or any other scheduler will invoke one packaged command with one validated configuration file. The engine will own discovery, eligibility, preflight, contract locking, work planning, queues, checkpoints, retries, quality evaluation, publication, and merge policy. LLMs will be used only for semantic code analysis and skeptical verification.

The implementation will remain one repository and one npm package. It will reuse the existing AgentsKit AdapterFactory, runtime contracts, token budgeting, retriever, and memory contracts. It will not create separate libraries or a second model abstraction.

The target architecture has the following boundaries:

1. A deterministic campaign engine owns the complete lifecycle of every discovered change request.
2. A pure state transition function is the only authority allowed to change campaign and pull-request states.
3. A provider reliability layer wraps existing AgentsKit model adapters with timeout, cancellation, typed errors, retry, circuit breaking, capability discovery, and complete token accounting.
4. An internal SCM contract separates the core from GitHub. GitHub is the first implementation; GitLab is a future implementation of the same contract.
5. A context planner creates bounded, diff-first context packs and assigns deterministic risk levels.
6. One multidimensional review call evaluates all enabled lenses for a normal context pack. Additional specialized analysis is reserved for high-risk changes.
7. Skeptical verification batches findings and escalates the number of independent votes according to severity, confidence, and disagreement.
8. Operational state, reusable review cache, feedback history, and approved review knowledge are stored separately.
9. AgentsKit retrieval injects only the small set of approved rules relevant to the current repository, path, language, and review category.
10. Run-level and campaign-level quality matrices are produced for successful and blocked executions alike.

The implementation will proceed incrementally, with each increment small enough to ship as one focused pull request. Every phase must execute a complete improvement cycle: establish or run the baseline, validate coverage and quality, identify process failures, correct them, rerun the real validation, run formal gates, and update Changesets/changelog/version/release before merging to main.

User Stories

1. As a repository maintainer, I want to schedule one deterministic campaign command, so that an external LLM does not control review orchestration.
2. As a repository maintainer, I want every discovered pull request to reach an explicit terminal outcome, so that no pull request silently disappears from a campaign.
3. As a repository maintainer, I want a validated configuration file with autocomplete, so that I do not need to maintain long, error-prone flag lists.
4. As a repository maintainer, I want configuration errors detected before model execution, so that invalid setup consumes no review tokens.
5. As a repository maintainer, I want worktrees created only after deterministic preflight succeeds, so that failed candidates do not consume setup time or disk space.
6. As a repository maintainer, I want Dependabot and configured authors excluded deterministically, so that eligibility is reproducible.
7. As a repository maintainer, I want previously reviewed SHA and policy combinations skipped idempotently, so that repeated campaigns do not create noise.
8. As a repository maintainer, I want review evidence locked to the head SHA, base SHA, configuration, prompt, model, policy, and library version, so that stale evidence can never authorize publication or merge.
9. As a repository maintainer, I want a crash-safe state machine, so that a stopped campaign resumes from the last valid checkpoint.
10. As a repository maintainer, I want successful review units reused after a partial failure, so that one failed model call does not discard valid work.
11. As a repository maintainer, I want transient failures retried at the failed unit, so that whole batches are not restarted unnecessarily.
12. As a repository maintainer, I want permanent failures classified immediately, so that authentication and configuration problems fail fast.
13. As a repository maintainer, I want adaptive concurrency, so that local CLIs and remote APIs run within their reliable capacity.
14. As a repository maintainer, I want the provider circuit breaker to stop new fan-out after repeated failures, so that the campaign does not waste calls during an outage.
15. As a repository maintainer, I want every external mutation to be idempotent, so that retries cannot duplicate reviews, comments, labels, or merges.
16. As a repository maintainer, I want merge decisions enforced by deterministic policy, so that an LLM finding alone never bypasses repository safeguards.
17. As a repository maintainer, I want configurable check policies, so that repositories with required, reported, or explicitly named checks can be handled correctly.
18. As a reviewer, I want comments anchored to valid diff lines, so that feedback appears where the change was made.
19. As a reviewer, I want every comment to explain the reason, impact, correction instructions, and evidence, so that a human or coding agent can act on it.
20. As a reviewer, I want comment language and presentation style to be configurable, so that the product works across teams while retaining a strong default.
21. As a reviewer, I want a CodeRabbit-inspired default presentation without copying a proprietary format, so that the review is polished and useful.
22. As a developer, I want one multidimensional analysis instead of seven repeated analyses of the same source, so that reviews are faster and cheaper.
23. As a developer, I want high-risk files to receive additional specialized analysis, so that token optimization does not reduce security or correctness coverage.
24. As a developer, I want review depth based on deterministic risk signals, so that credentials, authorization, migrations, and public contracts receive stronger review than documentation.
25. As a developer, I want changed hunks reviewed even when a complete file is large, so that file-size limits do not make the entire pull request unreviewable.
26. As a developer, I want the engine to request only bounded additional context, so that missing symbol context can be resolved without exposing the whole repository.
27. As a developer, I want source and related tests grouped when they fit within a safe context budget, so that the model can understand behavior without repeated prompts.
28. As a developer, I want static tools to provide lint, typecheck, test, secret, patch, and check evidence before the LLM runs, so that the model does not simulate deterministic tooling.
29. As a developer, I want skeptical verification proportional to finding risk, so that low-value findings do not receive the same expensive voting as blockers.
30. As a developer, I want multiple findings verified in one bounded request, so that adversarial validation remains strong without excessive provider calls.
31. As a developer, I want a cached result reused when source, policy, prompt, model, and relevant knowledge are unchanged, so that small updates do not repeat unaffected work.
32. As a developer, I want campaign, pull-request, context-pack, analysis, and verification budgets, so that one large pull request cannot consume the entire campaign allowance.
33. As a developer, I want the engine to use AgentsKit token budgeting and counters, so that every request is planned before execution.
34. As a developer, I want separate input, cached input, output, reasoning, retry, and memory token metrics, so that optimization is based on real usage.
35. As a developer, I want stable and dynamic prompt sections separated, so that providers capable of prompt caching can reuse stable prefixes.
36. As a developer, I want model routing to remain optional, so that a simple single-model deployment stays supported.
37. As a developer, I want faster models usable for low-risk work and stronger models reserved for critical changes, so that cost can be optimized without lowering safety.
38. As a maintainer of the library, I want the quality corpus and memory A/B suite run per relevant version or policy fingerprint, so that they do not add hundreds of thousands of tokens to every pull request.
39. As a maintainer of the library, I want a fixed benchmark comparing releases on the same corpus and real canary, so that performance claims are reproducible.
40. As a maintainer of the library, I want the quality matrix generated for blocked runs, so that failed executions are not excluded from reliability measurement.
41. As a maintainer of the library, I want a run matrix and a campaign matrix, so that semantic review quality and operational reliability are measured separately.
42. As a maintainer of the library, I want regressions to block release when they breach policy, so that a score remaining above the minimum does not hide deterioration.
43. As a maintainer of the library, I want GitHub integration values measured rather than hardcoded, so that the matrix represents real publication and merge behavior.
44. As a maintainer of the library, I want real posted inline comments validated, so that synthetic fixtures cannot stand in for GitHub evidence.
45. As a maintainer of the library, I want complete campaign wall-clock measurement, so that preflight, provider work, corpus, memory validation, publishing, and merge time are visible.
46. As a maintainer of the library, I want retries and wasted calls reflected in efficiency scores, so that unstable executions cannot receive a perfect score.
47. As a maintainer of the library, I want fault injection for every important failure mode, so that recovery behavior is proven without spending tokens.
48. As a maintainer of the library, I want deterministic replay of state transitions, so that production failures can be reproduced offline.
49. As a maintainer of the library, I want model adapters to expose capabilities, so that the engine selects supported behavior without provider-name conditionals.
50. As a maintainer of the library, I want GitHub behavior behind an SCM adapter, so that GitLab can be added later without rewriting the engine.
51. As a maintainer of the library, I want the existing AgentsKit adapter contract preserved, so that the package does not create a competing model abstraction.
52. As a self-hosted operator, I want review knowledge stored locally by default, so that I retain control of repository learning data.
53. As a self-hosted operator, I want operational state, cache, feedback, and knowledge stored separately, so that each can have an appropriate lifecycle and security policy.
54. As a security-conscious operator, I want full source, secrets, prompts, and raw tool transcripts excluded from persistent knowledge, so that learning does not become a source-code archive.
55. As a maintainer, I want findings correlated with later accepted, rejected, fixed, unresolved, or obsolete outcomes, so that learning is based on evidence.
56. As a maintainer, I want repeated evidence converted into candidate rules, so that individual observations do not immediately influence every future review.
57. As a maintainer, I want candidate rules tested against a separate regression corpus, so that an apparently useful rule does not introduce false positives.
58. As a maintainer, I want human approval required for rule promotion by default, so that hostile pull-request content cannot poison review memory.
59. As a maintainer, I want each active rule to have provenance, scope, confidence, usage, and validation history, so that it can be audited and rolled back.
60. As a developer, I want only a few relevant approved rules retrieved for each context pack, so that memory improves detection without bloating prompts.
61. As a developer, I want rule retrieval filtered by repository, path, language, and category, so that unrelated learning does not influence a review.
62. As a maintainer, I want memory impact measured through detection lift, precision, token overhead, and false positives, so that learning must prove its value.
63. As a maintainer, I want rules to expire or require revalidation, so that obsolete conventions do not remain permanently active.
64. As an open-source contributor, I want new model providers added through existing AgentsKit contracts, so that provider expansion stays consistent.
65. As an open-source contributor, I want future SCM providers to implement one stable internal contract, so that platform expansion is predictable.
66. As an Orca operator, I want the automation to run every three hours with one command, so that scheduling is simple and low-noise.
67. As an Orca operator, I want notifications only for meaningful completion, failure, or required action, so that unchanged state does not create noise.
68. As a project owner, I want AgentsKitOS used only as an external validation target, so that hardening the review product does not modify the revenue application.
69. As a project owner, I want each implementation phase delivered as a complete improvement cycle, so that progress is measured in finished outcomes rather than micro-steps.
70. As a project owner, I want each increment small enough for one focused pull request, so that implementation can be reviewed, tested, and released safely.

Implementation Decisions

Scope and product boundaries

• All implementation remains in the AgentsKit-io/code-review repository and the single @agentskit/code-review package.
• AgentsKitOS is an external validation target only and must not be modified by this initiative.
• The package remains repository-native and does not add an embedded chat product or documentation application.
• Existing AgentsKit model, runtime, tool, token, retriever, and memory contracts are reused.
• No second model abstraction or separate supporting npm package will be introduced.

Deterministic core

• One pure transition reducer will be the only authority allowed to move campaign and pull-request state.
• State transitions, side effects, and persistence will remain separate.
• Every state declares allowed events, invariants, required artifacts, timeout behavior, retry policy, and terminal outcomes.
• Campaign terminal outcomes are complete, partial, blocked, or cancelled.
• Pull-request terminal outcomes are skipped, blocked, approved, changes requested, merge blocked, merged, or cancelled.
• Review contracts are immutable and include source identities, configuration and policy fingerprints, library and prompt versions, provider/model identity, enabled lenses, and budgets.
• State and evidence are written atomically and are safe to replay.
• Completed review units are checkpointed and reused after interruption.

Deterministic campaign runner

• One packaged command owns configuration validation, discovery, eligibility, preflight, planning, queues, execution, reporting, publication, and merge policy.
• Orca is a scheduler only and must not interpret failures or choose subsequent workflow steps.
• Every discovered pull request must be included in the campaign report with one terminal outcome.
• One pull-request failure must not prevent independent eligible pull requests from reaching terminal outcomes.
• Worktrees may be created only after deterministic preflight passes.

Failure handling and resilience

• Provider and external failures are classified into typed categories at the boundary where they occur.
• Authentication, authorization, configuration, unsupported source, budget, and stale-contract failures are terminal or require explicit replanning.
• Rate limits, timeouts, eligible server failures, and recoverable transport failures may be retried with bounded exponential backoff and jitter.
• Retry occurs at the smallest failed review unit rather than at whole-batch level.
• Adaptive concurrency starts conservatively for local CLIs and reduces concurrency after instability.
• A provider circuit breaker prevents further fan-out and permits one controlled half-open probe after cooldown.
• Every external mutation uses an idempotency fingerprint.

Provider architecture

• Existing AgentsKit AdapterFactory implementations remain the model-provider boundary.
• Provider registry entries expose structured-output, token-usage, cached-token, cancellation, prompt-cache, persistent-session, context-limit, and recommended-concurrency capabilities when known.
• A common reliability wrapper provides timeout, cancellation, typed error normalization, retry, circuit breaking, token accounting, and telemetry.
• Provider-specific behavior does not leak into the deterministic engine.

SCM architecture

• An internal SCM contract owns change discovery, metadata, diffs, review state, publication, merge readiness, and merge execution.
• GitHub is the first complete SCM implementation.
• The public configuration must not imply that GitLab is operational until a GitLab adapter exists and passes the same contract tests.
• GitLab support is a future adapter implementation, not a reason to duplicate the core now.

Review engine and context strategy

• Enabled lenses remain separate logical quality dimensions.
• A normal context pack is analyzed by one multidimensional structured call rather than one call per lens.
• High-risk changes may receive additional specialized correctness or security analysis.
• Review input is diff-first and is organized around changed hunks, bounded adjacent context, directly related symbols, imports, types, and relevant rules.
• Missing context is requested explicitly and fulfilled through bounded deterministic retrieval.
• Small related files may share a context pack when they fit within a strict token budget.
• Static lint, typecheck, tests, secret scanning, patch validation, source classification, matching-test detection, and repository check evidence are collected before model execution.
• Deterministic risk classification selects low, normal, high, or critical review depth.
• Skeptical verification batches findings and escalates votes according to severity, confidence, and disagreement.
• A failed verification can block clean approval or merge but cannot manufacture a verified finding or a clean review.

Token, performance, and cache strategy

• AgentsKit token budgeting and token-counter contracts are used to plan each request.
• Budgets exist at campaign, pull-request, context-pack, analysis, and verification levels.
• Stable prompt instructions and dynamic review context are separated to permit provider prompt caching.
• Token accounting preserves input, cached input, output, reasoning, memory, retry, and total usage when the provider reports it.
• Review units are cached by source, base, policy, prompt, model, and relevant-knowledge fingerprints.
• Unchanged review units are reused without model calls.
• Quality corpus and memory A/B evaluations run only when their version, model, prompt, lens, schema, corpus, or memory-strategy fingerprint changes.
• Optional model routing may use a faster model for lower-risk analysis and a stronger model for critical analysis or disputed findings, but a simple single-model deployment remains supported.

Memory and learning architecture

• Operational state, reusable review cache, feedback ledger, and approved knowledge are separate stores.
• Raw source, full prompts, secrets, and unnecessary tool transcripts are not stored as permanent review knowledge.
• Self-hosted structured storage is the default.
• Feedback is reconciled with later fixed, accepted, rejected, unresolved, or obsolete outcomes.
• Repeated evidence creates candidate rules, not immediately active rules.
• Candidate rules are evaluated against a separate regression corpus.
• Rule promotion requires human approval by default; automatic promotion is opt-in.
• Active rules contain scope, category, language, paths, confidence, provenance, evidence counts, validation history, and lifecycle metadata.
• AgentsKit Retriever is used to inject a small, bounded set of relevant rules per context pack.
• Retrieval is filtered by repository, path, language, and category.
• AgentsKit VectorMemory remains optional and should be adopted only when measured rule volume justifies semantic retrieval.
• Chat-memory virtualization and summarization are not a substitute for avoiding unnecessary storage.

Quality matrices

• A run matrix measures semantic and operational quality for one pull request.
• A campaign matrix measures discovery, eligibility, completion, terminal outcomes, retries, blocked runs, merges, cache usage, total tokens, and total wall-clock.
• Matrices are produced for blocked and successful executions.
• Synthetic corpus evidence and real campaign evidence remain explicitly separate.
• GitHub, comment, safety, and merge metrics are derived from real evidence rather than hardcoded success values.
• Retry and wasted-call costs affect efficiency scores.
• Material regressions against the approved baseline can block release even when the resulting score remains above the minimum.
• No quality area may be below 3 out of 4.

Comments, checks, and merge

• Comment language, renderer, inline behavior, summary behavior, reason, impact, instructions, evidence, and suggested patches remain configurable.
• The default renderer is polished and CodeRabbit-inspired without copying proprietary output.
• Real GitHub inline placement and publication are validated.
• Check policies support branch-protection required checks, all reported checks, explicitly named checks, and an explicitly unsafe none mode.
• Automatic merge requires complete evidence, current SHA, passing quality, clean review, no changes requested, passing configured checks, mergeability, and enabled policy.
• Force and administrative merge remain forbidden by default.

Delivery phases

1. Establish the current baseline, the quality-matrix v2 schema, and architectural decisions.
2. Add central state, event, outcome, and typed-failure contracts.
3. Implement and test the pure state reducer.
4. Add atomic persistence, checkpoints, leases, replay, and resume.
5. Add provider capabilities and common error normalization.
6. Add selective retries, adaptive concurrency, and circuit breaking.
7. Define the internal SCM contract.
8. Move GitHub behavior behind the SCM adapter.
9. Add deterministic campaign discovery, eligibility, and complete preflight.
10. Add campaign queues and terminal outcome reporting.
11. Generalize multidimensional batched review.
12. Add the diff-first context planner.
13. Add deterministic risk classification.
14. Add batched adaptive skeptical verification.
15. Add review-unit fingerprint caching.
16. Add hierarchical budgets and complete accounting.
17. Separate operational state, cache, feedback, and knowledge storage.
18. Add real feedback reconciliation.
19. Add scoped AgentsKit knowledge retrieval.
20. Produce run and campaign quality matrices.
21. Complete configurable comment, check, and merge policies.
22. Add deterministic fault-injection and replay harnesses.
23. Replace the Orca prompt workflow with the single campaign command.
24. Complete shadow validation, controlled merge validation, release, and production rollout.

Release and rollout

• Offline harness validation precedes any real GitHub mutation.
• Shadow mode discovers and reviews without publishing or merging.
• Advisory mode publishes reviews without merging.
• Controlled merge mode starts with one known canary.
• Production mode processes all eligible pull requests every three hours.
• Notifications are emitted only for meaningful completion, failure, or required human action.
• Each phase is released through Changesets, changelog updates, npm Trusted Publishing, GitHub Release, and merge to main after its acceptance gates pass.

Testing Decisions

Tests must prove externally observable contracts and safety invariants rather than private implementation structure. Pure state transitions may be tested directly because the transition contract is itself an external core invariant. Provider, SCM, storage, and time boundaries will use deterministic test doubles. Real GitHub and real model validation are reserved for controlled integration and canary gates.

State and recovery testing

• Exercise every allowed state transition.
• Reject every invalid transition.
• Crash and resume after every state and external side effect.
• Reject mismatched SHA, policy, configuration, model, prompt, and artifact fingerprints.
• Prove that completed units are not repeated after resume.
• Prove that terminal states are immutable.
• Prove that concurrent campaigns cannot acquire the same lease.

Failure and resilience testing

• Inject provider timeout, rate limit, eligible server failure, invalid structured output, authentication failure, process exit, cancellation, and circuit-open failures.
• Prove that only transient failures retry.
• Prove bounded retry counts, backoff, concurrency reduction, and half-open probing.
• Prove that a partial unit failure does not discard other completed units.
• Inject publication timeout and crash-after-publication to prove idempotency.
• Inject a head-SHA change to prove stale evidence cannot publish or merge.
• Inject budget exhaustion to prove execution stops without fail-open behavior.

Adapter contract testing

• Run the same SCM contract suite against the GitHub adapter and future SCM adapters.
• Run provider capability and error-normalization tests without credentials.
• Preserve existing provider-neutral CLI and adapter fixtures.
• Test real provider smoke checks only when explicitly enabled.

Review-quality testing

• Maintain a versioned corpus containing clean examples, known correctness defects, security defects, missing-test behavior, documentation-only changes, large files, multi-file relationships, and prompt-injection attempts.
• Measure recall, precision, duplicates, severity accuracy, actionability, and inline location accuracy.
• Keep memory-learning evaluation cases separate from rule-generation cases.
• Compare multidimensional analysis with the previous seven-call baseline.
• Prove that risk-based specialization does not reduce critical detection.
• Prove that bounded context expansion resolves missing context without exposing unrelated repository content.

Token and performance testing

• Record input, cached input, output, reasoning, memory, retry, and total token usage.
• Record complete wall-clock, queue time, preflight, provider time, verification, publication, and merge time.
• Measure tokens per diff token, calls per review unit, calls per verified finding, prompt overhead, cache hit rate, and retry waste.
• Require at least 75% lower total token usage on the approved canary compared with the 0.8.0 baseline.
• Require at least three times fewer model calls and three times lower wall-clock for a small pull request.
• Prove that an unchanged second execution makes no model call.
• Prove that corpus and memory A/B validation add no per-pull-request model calls.

Memory and learning testing

• Prove that operational state, cache, feedback, and knowledge have separate lifecycles.
• Prove that source, prompts, secrets, and raw transcripts are excluded from permanent knowledge.
• Test accepted, rejected, fixed, unresolved, and obsolete feedback reconciliation.
• Test rule deduplication, scoping, provenance, approval, rollback, expiry, and revalidation.
• Measure memory-induced detection lift, false-positive changes, and token overhead.
• Reject memory promotion when precision regresses or provenance is missing.
• Test malformed and poisoned memory inputs fail closed.

GitHub behavior testing

• Validate real inline comment locations and sides against a controlled pull request.
• Prove fallback behavior for findings outside the diff.
• Prove marker-based update behavior creates no duplicate review output.
• Test required, reported, named, and none check policies.
• Prove a clean, current, fully validated canary may merge.
• Prove findings, changed SHA, failed checks, changes requested, incomplete evidence, or disabled policy prevent merge.

Quality-gate testing

• Generate matrices for complete, partial, blocked, and cancelled runs.
• Include every discovered pull request in campaign reliability.
• Prevent hardcoded integration or safety success from satisfying a gate.
• Ensure retry and wasted-call counts reduce efficiency scores.
• Block releases when any required area is below 3 or an absolute gate fails.
• Block material regressions according to the approved comparison policy.

Formal package and release gates

• Run typecheck, build, complete tests, documentation gates, CLI smoke validation, and package dry-run.
• Validate the packed artifact rather than only source checkout behavior.
• Execute one real canary review with the exact package version intended for release.
• Update Changesets and changelog for every public behavior change.
• Publish through npm Trusted Publishing and create the corresponding GitHub Release.

Out of Scope

• Modifying AgentsKitOS as part of this implementation. It is only an external validation target.
• Creating additional npm packages or repositories for campaign, memory, adapters, or quality evaluation.
• Implementing a complete GitLab adapter in the first delivery. The architecture must permit it without claiming it already works.
• Introducing a mandatory vector database or remote memory service.
• Indexing or embedding the entire source repository by default.
• Running an LLM as the workflow orchestrator, state machine, retry controller, or merge-policy engine.
• Making persistent provider sessions mandatory before measured evidence shows they are needed and safe.
• Enabling automatic rule promotion by default.
• Enabling force or administrative merge by default.
• Requiring multi-model routing for basic operation.
• Embedding AgentsKit Chat or adding a separate documentation application.

Further Notes

Quality targets

• Every discovered pull request reaches a terminal outcome.
• Every eligible changed hunk is reviewed or explicitly classified before model execution.
• No incomplete review is published as complete.
• No external action is duplicated after retry or resume.
• Only failed transient units are retried.
• No quality-matrix area scores below 3 out of 4.
• Total canary token usage is reduced by at least 75% from the 0.8.0 baseline.
• Small-pull-request provider calls and wall-clock are reduced by at least three times.
• No expected-detection loss or material false-positive regression is accepted.
• Memory injects no more than the configured bounded relevant knowledge per context pack.
• No complete source file is stored as permanent review knowledge.
• GitHub comments and merge behavior are validated using real integration evidence.

Phase completion protocol

Every phase is a complete continuous-improvement cycle:

1. Execute the baseline or real review.
2. Validate result coverage and quality.
3. Identify process failures.
4. Correct all internal failures within the phase scope.
5. Rerun the same real validation.
6. Run all formal gates.
7. Update Changeset/changelog, version, publish, and merge to main.

The phase must not stop between internal micro-steps. It may stop only for a genuine external blocker requiring human action, such as interactive authentication or new authorization.

Global Definition of Done

This initiative is complete only when:

• Orca schedules one deterministic command.
• No LLM controls workflow state or side effects.
• Every discovered pull request reaches a terminal outcome.
• Every eligible changed hunk is covered.
• No partial review is accepted or published as complete.
• Crash and resume never duplicate an external operation.
• Retry is selective and preserves successful work.
• Every quality area is at least 3 out of 4.
• The approved token and wall-clock reduction targets are achieved.
• Corpus and memory A/B checks do not run per pull request.
• Memory retrieves only relevant, approved, bounded knowledge.
• Real feedback demonstrates learning without precision regression.
• Real GitHub inline comments pass location and idempotency validation.
• Merge occurs only through deterministic policy.
• The Changesets and Trusted Publishing release flow passes.
• A real multi-pull-request campaign completes without manual workflow supervision.

Quality-matrix study artifacts and real-run measurements must remain local and must not be committed to the product repository. Only schemas, fixtures, deterministic harnesses, documentation, and non-sensitive aggregate expectations belong in source control.

[EmersonBraun]

Approved minimum-release acceptance plan (feature freeze): G1 real-provider positive/clean/regression corpus with per-case quality >=3/4; G2 complete medium and large PR reviews with immutable coverage and bounded resource usage; G3 packaged CLI auth/timeout/interruption/resume/stale-SHA/idempotence drills; G4 reconciled whole-cycle usage including retries and learning A/B, fail-closed resource ceilings; G5 artifact-verified Orca scheduling, a genuinely clean revision-locked eligible merge, and native CI dispatch for the Changesets version PR followed by OIDC release. Fix only defects found by these gates, preserve configured safety policy, and keep AgentsKitOS implementation read-only. Tests precede release, with current-source ak-verify evidence. Private quality matrices remain local. This bounded acceptance work does not close the entire PRD.

[GoodJobwilliam]

The deterministic evidence boundary in this PRD looks like a useful way to keep the model reviewer from becoming the source of truth. I would make the preflight output one versioned envelope that carries the source/base/head identities, tool or rule ID, location, severity, provenance, and an explicit status of pass, fail, or unavailable, plus a fingerprint for deduplication.

That lets the campaign engine consume existing SARIF/check results without rerunning equivalent analyzers, merge equivalent evidence before publication, and keep a provider timeout or missing tool distinct from a clean review. For large changes, representing each changed hunk as covered, skipped, or unavailable with a reason may be safer than making the whole pull request look reviewed or failing the entire run. Selective retries can then target only unavailable evidence while immutable review records retain the exact inputs.

Would this envelope fit the quality-matrix and SCM boundaries you are already planning, or is the current contract intentionally keeping deterministic tool evidence outside the review result?