Harden benchmark evidence and Nullalis local runner flow

Author: Muhra95

.github/ISSUE_TEMPLATE/submit-results.md

@@ -11,39 +11,59 @@ labels: results
 - **Version**: 
 - **URL/Repo**: 
 - **LLM Provider Used**: 
+- **Runtime Commit SHA**: 
+- **Harness Commit SHA**: 
 
 ## Benchmark Run
 
-- **DTaaS-Bench Version**: 0.1
+- **DTaaS-Bench Version**: 0.2
 - **Date**: 
 - **Platform**: 
 - **Harness Command Used**:
 
 ```bash
-python harness/runner.py --url <URL> --token <TOKEN> --user-id <ID> --name "<Name>"
+python -m harness.runner --url <URL> --token <TOKEN> --user-id <ID> --name "<Name>" --output results/<runtime>.json --markdown results/<runtime>.md --html results/<runtime>.html
 ```
 
 ## Results
 
-**Composite Score**: /100
+**Verified Composite Score**: /100
+**Projected Composite Score**: /100
+**Measured Coverage**: 
+**Coverage-Adjusted Verified Score**: /100
 **Rating**: 
 
 ### Dimension Scores
 
-| Dimension | Score |
-|-----------|-------|
-| Autonomy Control | |
-| Memory Persistence | |
-| Autonomous Execution | |
-| Cross-Channel Consistency | |
-| Integration Breadth | |
-| Security & Privacy | |
-| Scale & Cost Efficiency | |
-| Operational Resilience | |
-| Latency Profile | |
+| Dimension | Verified | Projected | Coverage |
+|-----------|----------|-----------|----------|
+| Autonomy Control | | | |
+| Memory Persistence | | | |
+| Functional Capability | | | |
+| Autonomous Execution | | | |
+| Cross-Channel Consistency | | | |
+| Integration Breadth | | | |
+| Security & Privacy | | | |
+| Scale & Cost Efficiency | | | |
+| Operational Resilience | | | |
+| Latency Profile | | | |
+
+## Evidence Pack
+
+- **Diagnostics attached**: yes/no
+- **Metrics attached**: yes/no
+- **Run manifest attached**: yes/no
+- **Any runtime/provider incident during run**:
+- **Incident attribution**: runtime / upstream dependency / network / unknown
+- **Notes on projected components**:
 
 ## Attachments
 
 Please attach:
 - [ ] `results/<runtime>.json` output from the harness
+- [ ] `results/<runtime>.md` or `results/<runtime>.html`
 - [ ] Any notes on configuration or setup
+- [ ] `/internal/diagnostics` snapshot if available
+- [ ] `/metrics` snapshot if available
+- [ ] Any incident notes or representative error messages if the run degraded
+- [ ] Optional run manifest based on `docs/run-manifest-v0.2.example.json`

CHANGELOG.md

@@ -0,0 +1,30 @@
+# Changelog
+
+All notable changes to DTaaS-Bench should be documented in this file.
+
+The format is intentionally simple and release-oriented.
+
+## [0.2.1] - 2026-03-24
+
+### Added
+
+- Trust model documentation in `docs/TRUST_MODEL.md`
+- Nullalis reference-runtime explainer in `docs/NULLALIS_REFERENCE_RUNTIME.md`
+- Example run manifest in `docs/run-manifest-v0.2.example.json`
+- Evidence parsing helpers in `harness/evidence.py`
+- Open-source release checklist in `docs/OPEN_SOURCE_RELEASE_CHECKLIST.md`
+
+### Changed
+
+- Updated result submission template to require v0.2-style verified/projected reporting
+- Updated README and CONTRIBUTING flow around trusted evidence packs
+- Hardened chat transport for runtimes that require explicit `session_key`
+- Isolated harness chat sessions per dimension
+- Improved autonomy scoring to use diagnostics and metrics more directly
+- Improved execution scoring with diagnostics-backed scheduler confirmation
+- Improved breadth scoring with runtime_info-backed structured counts
+- Reduced cross-channel reliance on generic assistant self-report
+
+### Notes
+
+- This release is focused on benchmark trust, open-source readiness, and compatibility with the current Nullalis gateway contract.

CONTRIBUTING.md

@@ -6,7 +6,7 @@ We welcome contributions from the community. Here's how you can help.
 
 The most impactful contribution is running DTaaS-Bench against your runtime and submitting verified results.
 
-1. Run the harness: `python3.10 -m harness.runner --url YOUR_URL --token YOUR_TOKEN --user-id 1 --name "Your Runtime" --output results/your-runtime.json`
+1. Run the harness: `python3.10 -m harness.runner --url YOUR_URL --token YOUR_TOKEN --user-id 1 --name "Your Runtime" --output results/your-runtime.json --markdown results/your-runtime.md --html results/your-runtime.html`
 2. Open an issue using the **Submit Results** template
 3. Attach your `results/your-runtime.json`
 
@@ -18,6 +18,17 @@ Required v0.2 fields for leaderboard eligibility:
 - `measured_coverage`
 - `coverage_adjusted_verified_score`
 
+Recommended submission attachments:
+- `results/your-runtime.md` or `results/your-runtime.html`
+- Runtime version and commit SHA
+- Harness commit SHA
+- `/internal/diagnostics` snapshot
+- `/metrics` snapshot when available
+- brief incident note if the runtime, network, or model provider degraded during the run
+- Optional run manifest based on `docs/run-manifest-v0.2.example.json`
+
+Please also read `docs/TRUST_MODEL.md` before submitting. The goal is not just a high score; it is a result other builders will trust.
+
 ## Propose New Dimensions
 
 If you believe an important DTaaS capability is not covered by the current 10 dimensions, open an issue with:

README.md

@@ -5,6 +5,8 @@
   </p>
   <p align="center">
     <a href="SPECIFICATION.md">Specification</a> &middot;
+    <a href="docs/TRUST_MODEL.md">Trust Model</a> &middot;
+    <a href="CHANGELOG.md">Changelog</a> &middot;
     <a href="results/nullalis-v0.2-report.md">Results</a> &middot;
     <a href="CONTRIBUTING.md">Contribute</a> &middot;
     <a href="https://novanuggets.com">Nova Nuggets</a>
@@ -19,6 +21,8 @@
 
 DTaaS-Bench does. 10 dimensions. Two explicit composites (`verified` + `projected`). Open harness. Run it against your runtime today.
 
+This repo is also an argument: there is a real category between "chatbot" and "agent framework" that deserves its own evaluation standard. We call that category `Digital Twin as a Service` (`DTaaS`).
+
 ---
 
 ## Verified Leaderboard (March 2026)
@@ -29,6 +33,8 @@ Only runs produced by this harness with raw artifacts are ranked here.
 |---------|------|------------------------------|----------|-----------|--------|
 | **[Nullalis v0.2 live artifact](results/nullalis-v0.2-report.md)** | Full runtime | **64.4** | **78.9%** | **79.8** | **Competitive** |
 
+Nullalis is the first reference runtime in this repository because it demonstrates the full stack this benchmark is naming: persistent memory, autonomous execution, multi-channel operation, and background-turn controls. Read [why that matters](docs/NULLALIS_REFERENCE_RUNTIME.md).
+
 ## External Comparison (Unverified, not ranked)
 
 External runtime rows are reference estimates from public documentation and are intentionally excluded from the verified leaderboard.
@@ -50,6 +56,16 @@ Submit a harness artifact to replace any estimate with a verified score.
 - CI now validates harness integrity and v0.2 artifact schema on every PR.
 - Latest live gateway run completed in 4392.9s (no per-chat timeout mode) and reports full measured timing metadata in the artifact.
 
+## Why This Repo Exists
+
+DTaaS-Bench is trying to do three jobs at once:
+
+- define the `DTaaS` runtime category clearly enough that people can argue about it in public
+- give builders a benchmark they can actually run against local and SaaS deployments
+- create a trusted submission standard so the leaderboard is earned rather than narrated
+
+Nullalis is important here because it is a strong reference runtime, not because it gets special treatment. The benchmark should become more valuable as more runtimes submit verified artifacts and try to beat it.
+
 ---
 
 ## Why DTaaS Needs Its Own Benchmark
@@ -104,6 +120,9 @@ Every run reports:
 
 The benchmark never promotes an unverified external estimate to leaderboard status.
 
+For the full evidence model, read [docs/TRUST_MODEL.md](docs/TRUST_MODEL.md).
+For nearby runtime compatibility status, read [docs/COMPETITOR_RUNNABILITY.md](docs/COMPETITOR_RUNNABILITY.md).
+
 ---
 
 ## Quick Start
@@ -136,10 +155,21 @@ python3.10 -m harness.runner \
   --output results/run.json \
   --markdown results/run.md \
   --html results/run.html
+
+# Local Nullalis: discover the active token from ~/.nullalis/config.json
+python3.10 -m harness.runner \
+  --url http://127.0.0.1:3000 \
+  --token-from-nullalis-config \
+  --user-id 1 \
+  --name "Nullalis Local"
 ```
 
 The harness produces a terminal summary table, a machine-readable JSON file, a Markdown report, and a self-contained HTML report with score bars and tier badges.
 
+Useful local tuning:
+- `--memory-sample-size 5` for faster validation reruns
+- `--timeout 20 --timeout-dynamic --timeout-floor 20 --timeout-ceiling 120 --timeout-grace 10` for bounded local gateway runs
+
 Runtime requirement:
 - Python 3.10+ (the harness uses modern type syntax)
 
@@ -164,21 +194,38 @@ Each dimension script sends real requests to the runtime, parses responses, and
 
 Run the benchmark. Submit your score. Join the leaderboard.
 
-1. `python -m harness.runner --url YOUR_URL --token TOKEN --user-id 1 --name "Your Runtime" --output results/your-runtime.json`
+1. `python -m harness.runner --url YOUR_URL --token TOKEN --user-id 1 --name "Your Runtime" --output results/your-runtime.json --markdown results/your-runtime.md --html results/your-runtime.html`
 2. Open an issue using the [Submit Results template](.github/ISSUE_TEMPLATE/submit-results.md)
-3. Attach your JSON output
+3. Attach your JSON output and recommended evidence pack
 
 All submitted results are published with full transparency. No editorial gatekeeping.
 
+Recommended evidence pack:
+
+- result JSON
+- Markdown or HTML report
+- runtime version or commit SHA
+- harness commit SHA
+- diagnostics snapshot
+- metrics snapshot
+- incident notes when the runtime or upstream model path degraded during the run
+- optional run manifest using [docs/run-manifest-v0.2.example.json](docs/run-manifest-v0.2.example.json)
+
 ---
 
 ## Repository Structure
 
 ```
 DTaaS-benchmark/
 ├── README.md                   You are here
+├── CHANGELOG.md                Release history
 ├── SPECIFICATION.md            Full benchmark spec (10 dimensions, scoring, rules)
 ├── CONTRIBUTING.md             How to contribute
+├── docs/
+│   ├── TRUST_MODEL.md          Verification rules and evidence tiers
+│   ├── NULLALIS_REFERENCE_RUNTIME.md
+│   ├── OPEN_SOURCE_RELEASE_CHECKLIST.md
+│   └── run-manifest-v0.2.example.json
 ├── LICENSE                     Apache-2.0
 ├── harness/
 │   ├── runner.py               CLI entry point
@@ -204,6 +251,8 @@ DTaaS-benchmark/
 
 [SPECIFICATION.md](SPECIFICATION.md) contains everything: dimension definitions, test protocols, scoring formulas, rules for fair comparison, and the complete methodology.
 
+If you want the short version of what counts as a trustworthy result, start with [docs/TRUST_MODEL.md](docs/TRUST_MODEL.md).
+
 ---
 
 ## License

artifacts/nullalis-diagnostics-post-2026-03-25.json

@@ -0,0 +1 @@
+{"error":"unauthorized"}

artifacts/nullalis-diagnostics-post-rerun.json

@@ -0,0 +1 @@
+{"gateway":{"requests_total":461,"in_flight_requests":1,"drain_rejected_total":0,"overload_rejected_total":0,"draining":false},"runtime_mode":"threaded","internal_auth_required":false,"internal_token_configured":true,"internal_token_policy_ok":true,"internal_token_policy_reason":null,"instance_id":"pid-65213-1774391416","owned_users_count":0,"tenant_lock_backend":"postgres_lease","tenant_lock_lease_secs":90,"tenant_lock_wait_ms":500,"tenant_lock_retry_min_ms":25,"tenant_lock_retry_max_ms":80,"tenant_lock_conflict_retries_total":0,"tenant_runtime_policy_attached":true,"effective_config_source":"postgres_user_config","effective_config_hash":"cdc96b6f7af70003","memory_search_enabled":true,"memory_summarizer_enabled":true,"provider_retries":2,"fallback_provider_count":1,"memory_vector_sync_mode":"best_effort","memory_outbox_enabled":false,"configured_config_source":"postgres_user_config","configured_memory_search_enabled":true,"configured_memory_summarizer_enabled":true,"configured_provider_retries":2,"configured_fallback_provider_count":1,"configured_memory_vector_sync_mode_requested":"best_effort","configured_memory_outbox_requested":false,"control_plane":{"configured_config_source":"postgres_user_config","effective_config_source":"postgres_user_config","configured_config_hash":"cdc96b6f7af70003","effective_config_hash":"cdc96b6f7af70003","configured_assistant_mode":"balanced","effective_assistant_mode":"balanced","configured_ignored_tenant_override_count":4,"effective_ignored_tenant_override_count":4,"controls":{"agent.parallel_tools":{"configured":true,"effective":true,"owner":"operator","source":"postgres_user_config","drift":false},"agent.parallel_tools_rollout_percent":{"configured":100,"effective":100,"owner":"operator","source":"postgres_user_config","drift":false},"agent.tool_dispatcher":{"configured":"parallel","effective":"parallel","owner":"operator","source":"postgres_user_config","drift":false},"memory.reliability.rollout_mode":{"configured":"on","effective":"on","owner":"operator","source":"postgres_user_config","drift":false},"memory.reliability.shadow_hybrid_percent":{"configured":0,"effective":0,"owner":"operator","source":"postgres_user_config","drift":false},"memory.reliability.canary_hybrid_percent":{"configured":0,"effective":0,"owner":"operator","source":"postgres_user_config","drift":false},"memory.reliability.fallback_policy":{"configured":"degrade","effective":"degrade","owner":"operator","source":"postgres_user_config","drift":false},"memory.search.enabled":{"configured":true,"effective":true,"owner":"operator","source":"postgres_user_config","drift":false},"memory.search.sync.mode":{"configured":"best_effort","effective":"best_effort","owner":"operator","source":"postgres_user_config","drift":false},"memory.summarizer.enabled":{"configured":true,"effective":true,"owner":"operator","source":"postgres_user_config","drift":false},"memory.summarizer.window_size_tokens":{"configured":4000,"effective":4000,"owner":"operator","source":"postgres_user_config","drift":false},"memory.summarizer.summary_max_tokens":{"configured":500,"effective":500,"owner":"operator","source":"postgres_user_config","drift":false},"memory.summarizer.auto_extract_semantic":{"configured":true,"effective":true,"owner":"operator","source":"postgres_user_config","drift":false},"tenant.identity_mapping_enforcement":{"configured":"staged_strict","effective":"staged_strict","owner":"operator","source":"postgres_user_config","drift":false},"tenant.identity_mapping_strict_channels":{"configured":["telegram"],"effective":["telegram"],"owner":"operator","source":"postgres_user_config","drift":false},"gateway.require_explicit_chat_stream_session_key":{"configured":true,"effective":true,"owner":"operator","source":"postgres_user_config","drift":false},"session.cross_channel_shared_main":{"configured":false,"effective":false,"owner":"operator","source":"postgres_user_config","drift":false}}},"sandbox_workspace_validation_failed_total":0,"sandbox_fallback_none_total":0,"sandbox_workspace_validation_last_reason":"none","chat_stream_require_explicit_session_key":true,"chat_stream_lane_counts":{"main":1,"thread":164,"task":0,"cron":0},"chat_stream_session_key_rejections":{"missing":0,"invalid":0,"wrong_user":0,"invalid_lane":0},"background_main_reroutes_total":0,"background_main_reroutes_last_job_id":null,"tenant_lock_conflicts_by_route":{"chat_stream_sse":0,"chat_stream_http":0,"webhook":0,"daemon":0,"api":0},"tenant_lease_probe":{"user_id":"1","data_source":"postgres_lease","owner_id":null,"lease_until_s":null,"updated_at_s":null},"bus":{"inbound_len":0,"outbound_len":0,"capacity":1024},"pool":{"hits":0,"misses":0,"idle":0},"startup_self_check":{"config_path":"/Users/nova/.nullalis/config.json","state_backend_configured":"postgres","state_backend_effective":"postgres","heartbeat_enabled":true,"heartbeat_interval_minutes":60,"tenant_enabled":true,"degraded":false,"degraded_reason":"","postgres_host":"127.0.0.1","postgres_port":5432,"postgres_schema":"zaki_bot","scheduler_backend":"postgres","webhook_mode":"none","chat_provider_effective":"together","chat_fallback_chain":"openrouter","embedding_provider_effective":"together","provider_data_source":"config"},"heartbeat_wake":{"pending":0,"dropped_total":0,"coalesced_total":0},"last_trigger":null,"heartbeat_runtime":{"user_id":"1","available":true,"last_run_s":1774394928,"last_status":"idle","last_reason":"no_actionable_output"},"integrations":{"telegram":{"configured":true,"connected":true,"account_id":"default","chat_id":1110331014,"data_source":"postgres"}},"identity_mapping":{"mapped":0,"unmapped":0,"strict_rejected":0,"degraded_compat":0,"cache_hit":0,"cache_miss":0,"cache_stale":0,"db_lookup_count":0,"db_lookup_ms_total":0},"stt":{"transcriber_configured":0,"transcription_attempted":0,"transcription_succeeded":0,"transcription_failed":0,"transcription_skipped_no_transcriber":0,"failure_get_file":0,"failure_download":0,"failure_transcriber":0,"failure_empty_transcript":0},"multimodal":{"image_markers_detected":0,"messages_with_image_markers":0,"image_parts_prepared":0,"image_parts_failed":0,"image_markers_ignored":0},"ops":{"proactive_sent_total":0,"proactive_blocked_rate_total":0,"proactive_blocked_dedupe_total":0,"proactive_send_errors_total":0,"scheduler_executed_total":0,"scheduler_blocked_burst_total":0,"scheduler_blocked_cooldown_total":0,"proactive_policy":{"dedupe_window_secs":120,"rate_window_secs":300,"rate_limit_per_window":12},"last_event":null,"recent_events":[]}}