docs: DIA v2 backlog from live e2e test (10 items, 6 issues)

[mdvnavy]

Captures the DIA v2 backlog discovered during a 2026-06-14 end-to-end test of the deployed app. 10 items total, ordered by impact-per-effort. The 4 highest-value items are also opened as issues (#14, #15, #17, #18, #19, #20). No code changes. See docs/DIA-V2-BACKLOG.md for the full triage.

[Copilot]

Pull request overview

This pull request captures a DIA v2 backlog write-up, but also introduces a large set of implementation changes across the runtime server, agent wiring (ADK + MCP), evaluation harness, tests (unit + adversarial + e2e), deployment artifacts, and repo security/CI scaffolding.

Changes:

• Adds an eval harness (eval/run_eval.py + eval/golden.json) and an autoresearch program (program-dia.md) with extensive new fixtures/tests.
• Expands the demo app with new endpoints (/healthz, /api/agent, /api/agent/status), runtime sample loading, request-body hardening, and optional local OBS capture hooks.
• Introduces ADK MCP toolset wiring (Make + GCP) and adds Cloud Run / deploy / CI / gitleaks tooling and documentation.

Reviewed changes

Copilot reviewed 71 out of 75 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
.claude/launch.json	Adds local launch config for running the app.
.devcontainer/devcontainer.json	Updates forwarded port to 8080 for Codespaces.
.dockerignore	Defines container build exclusions (notably excludes tests/).
.env.example	Documents required/optional env vars (Gemini + MCP).
.githooks/pre-commit	Delegates pre-commit hook to the maintained shell script.
.githooks/pre-commit.ps1	Adds PowerShell mirror for gitleaks pre-commit scanning.
.githooks/pre-commit.sh	Adds staged + working-tree gitleaks scanning hook.
.github/workflows/ci.yml	Adds CI job for pytest + gitleaks secret scanning.
.gitleaks.toml	Adds gitleaks configuration and allowlist.
.gitignore	Ignores autoresearch scratch logs/TSV outputs.
CAPTURE_INTEGRATION.md	Documents OBS capture side integration and overlap notes.
Dockerfile	Adds container build for Cloud Run (Python 3.12-slim, port 8080).
README.md	Expands docs: endpoints, UI features, deploy instructions, MCP notes.
SECURITY.md	Adds security/secret-handling guidance and incident steps.
agent_runtime.py	Adds sync bridge to run one ADK agent turn per HTTP request + MCP selection.
app.py	Adds health/status endpoints, /api/agent, request-size checks, runtime sample lookup, optional OBS capture.
character.py	Adds build_agent() + optional MCP toolsets; updates .env loading and key detection.
client_discovery/config.py	Adds dotenv-backed config loader/validator for keys + OBS settings.
client_discovery/core.py	Adds Jules refinement + OBS helpers, improves parsing/table handling, adjusts scoring helpers, extends generate_documents signature.
client_discovery/service.py	Adds alternate service wrapper that runs deterministic + (optional) Gemini + Jules + OBS capture.
cloud-run.yaml	Adds a sample Cloud Run service manifest.
deploy-cloudrun.sh	Adds Cloud Run deploy script using Secret Manager mounts.
deploy.ps1	Adds Windows PowerShell deploy helper.
deploy.sh	Adds bash deploy helper with Artifact Registry + Cloud Build flow.
docs/2026-06-10_overnight-features_design.md	Documents UI feature design decisions/spec.
docs/2026-06-10_overnight-features_plan.md	Documents UI implementation plan checklist.
docs/DIA-V2-BACKLOG.md	Adds DIA v2 backlog captured from live e2e test.
docs/architecture.mmd	Adds Mermaid architecture diagram for agent wiring.
docs/architecture.svg	Adds rendered architecture diagram asset.
docs/autoresearch/.gitkeep	Keeps autoresearch docs directory tracked.
docs/superpowers/specs/2026-06-10-dia-autoresearch-design.md	Adds spec for autoresearch harness and guardrails.
eval/golden.json	Adds golden expectations for deterministic pipeline fixtures.
eval/run_eval.py	Adds deterministic eval harness and scoring rubric.
program-dia.md	Adds instructions for autoresearch experiment loop and constraints.
pytest.ini	Adds baseline_pin marker configuration.
requirements.txt	Updates dependencies (ADK/genai versions) and adds MCP/requests/OBS libs.
samples/sample_questionnaire.md	Adds runtime sample questionnaire for shipping in the image.
static/style.css	Adds theme system, a11y styles, and UI styling for new controls.
templates/index.html	Adds UI features (theme toggle, Ask DIA panel, draft editor, a11y roles).
tests/conftest.py	Sets default test env vars for test runs.
tests/e2e/__init__.py	Initializes e2e test package.
tests/e2e/conftest.py	Adds e2e fixtures/helpers and extensive dependency stubbing.
tests/e2e/fixtures/inputs/complete_intake.md	Adds e2e intake fixture input.
tests/e2e/fixtures/inputs/enterprise_scenario_intake.md	Adds e2e intake fixture input.
tests/e2e/fixtures/inputs/extreme_pain_points_intake.md	Adds e2e intake fixture input.
tests/e2e/fixtures/inputs/large_intake.md	Adds e2e intake fixture input.
tests/e2e/fixtures/inputs/malformed_tables_intake.md	Adds e2e intake fixture input.
tests/e2e/fixtures/inputs/minimal_intake.md	Adds e2e intake fixture input.
tests/e2e/fixtures/inputs/missing_fields_warning_intake.md	Adds e2e intake fixture input.
tests/e2e/fixtures/inputs/missing_pain_points_error_intake.md	Adds e2e intake fixture input.
tests/e2e/fixtures/inputs/no_tools_intake.md	Adds e2e intake fixture input.
tests/e2e/fixtures/inputs/regulated_compliance_intake.md	Adds e2e intake fixture input.
tests/e2e/fixtures/inputs/special_characters_intake.md	Adds e2e intake fixture input.
tests/e2e/fixtures/inputs/tech_person_only_intake.md	Adds e2e intake fixture input.
tests/e2e/fixtures/responses/jules_empty.json	Adds Jules response fixture.
tests/e2e/fixtures/responses/jules_malformed.json	Adds Jules response fixture.
tests/e2e/fixtures/responses/jules_success.json	Adds Jules response fixture.
tests/e2e/test_adversarial.py	Adds black-box adversarial e2e tests around HTTP behavior.
tests/test_adversarial.py	Adds adversarial unit tests for HTTP handling and server robustness.
tests/test_adversarial_coverage.py	Adds adversarial coverage tests around OBS/Jules/document generation.
tests/test_adversarial_coverage_challenger.py	Adds additional adversarial tests for parser/scoring/server edge cases.
tests/test_adversarial_coverage_challenger_p2.py	Adds concurrency and transport-fault tests plus type mismatch checks.
tests/test_agent_runtime_mcp_selection.py	Adds tests for MCP toolset selection logic.
tests/test_app.py	Refactors request helper and expands endpoint tests (health/sample/agent).
tests/test_character.py	Adds unit tests for character tool wrappers.
tests/test_config.py	Adds tests for config loading/validation and dotenv load gating.
tests/test_core.py	Expands core tests to cover Jules + OBS helper behavior.
tests/test_deploy_validation.py	Adds deploy script validation and startup resilience tests.
tests/test_empirical_challenger.py	Adds empirical HTTP edge-case tests (Content-Length variants).
tests/test_eval_harness.py	Adds build-time tests for eval harness behavior and baseline expectations.
tests/test_gcp_mcp.py	Adds tests for optional GCP MCP toolset wiring.
tests/test_make_mcp.py	Adds tests for optional Make MCP toolset wiring and log hygiene.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.