Funding opportunities — SBIR/STTR, NIST AISC, NSF, and commercial paths

[AlbinoGeek]

Purpose

Consolidated tracker for funding opportunities relevant to bakeoff as an open-source AI evaluation platform. Covers government grants, cooperative frameworks, and commercial angles. Updates posted here as solicitations open or close.

---

Opportunity register

SBIR / STTR (fastest government path)

Agency	Relevance	Status
IARPA	AI evaluation and red-team tooling — direct overlap	Monitor BROAD agency announcements
DoD OUSD(R&E)	AI assurance and adversarial robustness	Monitor SBIR.gov topic lists (quarterly)
DHS S&T	AI safety evaluation for critical infrastructure	Monitor; STTR eligible with university partner

Recommended action: Register at SBIR.gov, set alerts for "artificial intelligence evaluation", "LLM red-teaming", "AI assurance". Phase I awards typically $50–250K, 6-month cycle from solicitation close to award notice.

NIST AI Safety Institute (AISI)

NIST AISI is actively seeking private-sector partners for benchmark development. Non-dilutive (no equity, no repayment). bakeoff's open-source methodology and hardware-grounded scoring are directly relevant.

Recommended action: Monitor nist.gov/artificial-intelligence for partnership announcements. A one-page capability statement describing bakeoff's scoring methodology is the entry artifact.

NSF

CISE directorate funds AI safety, robustness, and evaluation research under public-good framing.

Recommended action: Track NSF 24-xxx solicitations in the "Responsible Computing" and "Safe Learning-Enabled Systems" programs. University collaboration strengthens eligibility.

Commercial / VC

VC interest follows commercial defensibility. The benchmark data itself is not the product — enterprise red-teaming workflows built on top of bakeoff data would be. Secondary priority until the methodology is published and the open dataset exists.

---

Timeline considerations

The government funding window for shaping AI cyber-benchmark methodology is open now (see #26). Once the classified EO benchmark process closes, the most credible positioning angle becomes reactive. Recommend publishing a methodology brief before that window closes, even if the full implementation (phase 2+) is not yet complete.

---

Blockers / open questions

• Phase 1 complete — funding conversations are credible only once basic test infrastructure is functional and results are reproducible
• Methodology brief — one-page public write-up needed for solicitation responses and NIST partnership outreach

---

Updates

First post — 032203ZJUN26. Subsequent updates appended as opportunities open, close, or advance.

— Bastion

[AlbinoGeek]

Methodology brief — capability statement for outreach

bakeoff is an open-source N-vs-N LLM evaluation harness designed for hardware-grounded, reproducible benchmarking of locally-hosted models. It measures what matters for government and enterprise AI assurance: not token cost, but energy cost, failure behavior, and verifiable judgment consistency. The June 2026 EO on AI cyber benchmarks creates a narrow window to position an open-source methodology before the classified process closes.

---

What we measure

• Accuracy / completeness — task × prompt-variant matrix with completeness-weighted partial scoring and floor-score rollup across incomplete runs
• Energy efficiency — watt-hours per task via nvidia-smi / rocm-smi; hardware-grounded cost axis, not token proxies
• Failure modes — 9-code taxonomy: timeout, refusal, malformed output, OOM, load failure, capability gap, infra error, cancelled, unknown
• Judge consistency — pairwise (N≤4) or scored (N≥5) LLM judge; order randomized per call; swapped verdicts inverted to cancel positional bias
• Cyber capability proxies — planned C1–C6 suite: prompt injection resistance, CBRN refusal floor, dual-use code generation rate, agentic containment, exfiltration-via-reasoning, non-expert uplift measurement

---

How we measure it

Models are served through a llama-swap proxy (LM Studio GGUFs via ghcr.io/ggml-org/llama.cpp:server-vulkan, OpenAI-compatible API). Each model is evaluated sequentially in isolation — no cross-contamination between runs. Energy draw is polled at the hardware layer, giving a cost signal that survives across API versions, quantization levels, and deployment configurations. Result envelopes are Ed25519-signed with git SHA and hardware metadata at capture time.

---

Open methodology commitments

• Full task definitions, prompt variants, and scoring rubrics published in-repo
• Signed result bundles submitted to bakeoff-results (open data repo), rendered as a static site via GitHub Pages
• Planned: Sigstore/Rekor transparency log for public submissions — tamper-evident public record
• No proprietary judge lock-in; judge model and prompts are configurable and disclosed

---

Cyber safety evaluation (EO alignment)

The June 2026 EO commits the US government to a classified benchmarking process for advanced AI cyber capabilities. The C1–C6 proxy suite maps to that domain: prompt injection, CBRN refusal, dual-use code generation, agentic containment, exfiltration-via-reasoning, and non-expert uplift. Publishing the open-source proxy methodology before the classified process closes positions bakeoff as the unclassified complement — the reproducible, auditable baseline against which classified results can be calibrated. That is a role NIST AISI is structurally incentivized to support.

---

Current state

Complete: core harness, energy scoring, failure taxonomy, Ed25519 provenance, pairwise/scored judge with bias mitigation, result publication pipeline.
In progress: C1–C6 cyber safety suite, Rekor integration, bakeoff-results open data repo.
Honest gap: Phase 1 infrastructure must be demonstrably functional and results reproducible before funding conversations are credible. The C1–C6 suite is planned, not shipped.

---

Recommended outreach artifacts

NIST AISI capability statement needs: scoring methodology (energy axis + completeness weighting), failure taxonomy, judge design, provenance chain, and a clear statement of what reproducibility guarantees exist today vs. planned.

SBIR Phase I abstract needs: problem statement (LLM evaluation lacks hardware-grounded, open, signed methodology), proposed solution (bakeoff), differentiation (energy axis + provenance + open data), and team capacity. Phase I awards ($50–250K) are feasible once Phase 1 infra closes.

Close the Phase 1 blocker first. Outreach before it closes reads as premature to any technical reviewer.

---

— Bastion // 050139ZJUN26

[AlbinoGeek]

bakeoff: AI Model Evaluation Platform — Methodology Brief

Prepared for: NIST AISI Benchmark Development Partnership Outreach; SBIR/STTR Solicitation Response Support
Repository: https://github.com/Rethunk-AI/bakeoff (open source, MIT)

---

What bakeoff is

bakeoff is an open-source AI model evaluation platform built for hardware-grounded, reproducible benchmarking of large language models running on local inference hardware. Unlike cloud-based evaluation services, bakeoff runs models directly on the evaluating system — measuring quality, latency, and energy cost under real deployment constraints. Results are cryptographically signed and stored in a schema-versioned data model designed for longitudinal comparison and public reproducibility.

The platform is designed to be operator-reproducible: any organization can clone the repository, point it at their own hardware and model weights, and produce a result bundle that is independently verifiable against the original.

---

Methodology

Model queue and execution. bakeoff maintains a disk-backed run queue (pending/ → completed/) with rename-as-mutex claim semantics. The runner iterates models sequentially — all task × prompt cells for model A complete before model B loads. This design is hardware-accurate: unified-memory and modest-VRAM systems cannot hold multiple model instances concurrently. One model loads, all its evaluations run, it unloads, the next loads. Exactly N swaps per benchmark run (N+1 with judge model).

Task matrix. Each run executes a tasks × prompt_variants × models matrix. Task categories include qa, code, summarize, classify, dumb_model (deterministic floor tier), and cyber_safety (see below). Scoring modes: pairwise tournament (pairwise_all, optimal for N=2–4 models), LLM rubric scoring (1–5 scale, linear for N≥5), and heuristic scorers (exact match, contains, regex) for structured ground-truth tasks.

Result signing. Completed result bundles are signed using Ed25519 keys. Each runner registers a public key; the signing module (bench/signing.py) produces a canonical sha256 digest of the result envelope before signing. This provides tamper-evidence for result bundles submitted to the public bakeoff-results repository.

Schema-versioned storage. The relational schema (schema/schema.sql) tracks models, runs, hardware context, prompts, and scores with explicit schema versioning tables (schema_versions, schema_tables). UUID5 deterministic identifiers allow deduplication of model records across runs. The disk-persistence layer (bench/store.py) uses atomic JSON I/O under a configurable data directory, with schema version gates that hard-error on unexpected versions.

---

Differentiators

Hardware-native scoring. bakeoff records VRAM (MB), theoretical TFLOPS (fp16/bf16/fp32), PCIe interface class, CPU, and RAM alongside every result. Scores are normalized against actual hardware capability — a 7B model on a 16 GB VRAM GPU is evaluated in a different hardware tier than the same model on an 80 GB datacenter card. Energy cost (energy_wh, cost_usd) is measured via nvidia-smi or rocm-smi power sampling, not estimated from token counts.

Sigstore-compatible signed result bundles. Result envelopes carry Ed25519 signatures tied to registered runner identities. The publish pipeline (bench/publish.py) packages, signs, and submits bundles to a public results repository, enabling third-party verification of any published benchmark result.

Open data schema. The full relational schema is public, versioned, and migration-tracked. Any downstream consumer — government evaluator, academic researcher, or competing platform — can ingest and reproduce bakeoff result data without proprietary tooling.

Cyber capability proxy test suite (C1–C6). The cyber_safety task category implements six proxy evaluations for AI safety-relevant capabilities:

• C1 — Injection resistance: resistance to prompt injection and instruction-override attacks
• C2 — Refusal quality: calibration and consistency of harmful-request refusals
• C3 — Dual-use code generation: detection of capability uplift in code assistance
• C4 — Agentic containment: boundary adherence in multi-step agentic task execution (trace-level logging via agent_traces table)
• C5 — Exfiltration-via-reasoning: detection of sensitive data leakage through chain-of-thought
• C6 — Non-expert uplift: measurement of capability uplift provided to non-technical adversarial users

The uplift_baseline_task_id schema field enables delta scoring: a model's C6 score is measured against an established non-expert baseline, quantifying marginal uplift rather than absolute capability.

---

Current status

Phase 1 infrastructure is complete. The test runner is functional end-to-end: models load, execute task matrices, scores are computed (heuristic and judge-based), failures are classified via a 9-code taxonomy (timeout, refusal, malformed_output, oom, load_failure, capability_gap, infra_error, cancelled, unknown), and results are stored and reportable. The bakeoff-report CLI produces JSON, Markdown, and single-file HTML dashboards from stored runs. Hardware context collection, result signing, and the cyber_safety schema extensions (C4 + C6 agent traces) are implemented. Schema versioning is in place with migration graph DDL ready for the migration runner (Phase 2).

---

Relevance to government programs

NIST AISI. bakeoff is designed as a benchmark development platform, not a benchmark. Its open schema, signed result bundles, and hardware-normalized scoring address NIST AISI's stated priorities for transparent, reproducible AI evaluation infrastructure. The cyber proxy suite (C1–C6) maps directly to AISI's adversarial robustness and dual-use capability assessment interests.

SBIR/STTR alignment. The platform addresses a Phase I technical challenge — building hardware-grounded, tamper-evident AI evaluation infrastructure — with a working prototype. Phase II natural extensions include a migration runner, distributed queue workers, and integration with NIST benchmark datasets.

IARPA / DoD relevance. The agentic containment (C4) and exfiltration-via-reasoning (C5) test categories speak directly to DoD and IARPA interests in AI assurance for autonomous systems. The agent trace schema provides the step-level audit trail required for post-hoc containment analysis.

---

— Bastion // 050600ZJUN26

[AlbinoGeek]

Bastion — Update 1 — 100210ZJUN26 — Phase 1 milestone + methodology brief draft

---

Status on blockers

Blocker 1: Phase 1 complete — NEAR-RESOLVED. Basic test infrastructure is functional as of this week (test runner, migration runner, results reporting all shipped; one schema correction pending in #38). Phase 1 gate will be clear once #38 closes.

Blocker 2: Methodology brief — See draft below.

---

Methodology brief (draft — v0.1)

bakeoff: Open Hardware-Grounded LLM Evaluation

Purpose: one-page capability statement for solicitation responses and NIST AISI partnership outreach.

---

What bakeoff measures

bakeoff is an open-source, hardware-grounded harness for comparative evaluation of large language models (LLMs). Rather than measuring tokens-per-second or aggregate benchmark scores in isolation, bakeoff measures capability per unit of energy on real consumer and prosumer hardware — making results reproducible, hardware-contextualized, and directly relevant to deployment cost.

Methodology

Task matrix. bakeoff runs a defined task suite across tasks × prompt_variants × models on local hardware. Task categories include:

• Baseline — quality floor tests (deterministic scorers; minimal-capability detection)
• Comparison — standard capability dimensions (reasoning, coding, summarization, classification)
• Advanced / Cyber safety — EO-aligned cyber proxy tests (injection resistance, refusal quality, agentic containment, non-expert uplift measurement, exfiltration-via-reasoning)

Hardware capture. Each run records the physical hardware context (GPU model, VRAM, interface, system specs) and energy draw (via nvidia-smi / rocm-smi). The cost axis is watt-hours per task, not latency — a deliberate choice to surface total-cost-of-inference rather than raw speed.

Scoring. Deterministic heuristic scorers cover objective tasks. Subjective quality dimensions use an LLM judge in pairwise mode (N=2–4 models) or scored mode (N≥5) with randomized presentation order and inverted-verdict correction, mitigating positional bias.

Integrity. Results are signed with Ed25519 and submitted as Sigstore-attested bundles to a public results repository (Rethunk-AI/bakeoff-results). Signing identity is verified at ingestion. Reproducibility is first-class: model hash, hardware snapshot, quantization method, and software stack are all captured per run.

Migration framework. The schema is versioned with a Tengo-scripted migration runner (Go), enabling controlled evolution of the data model without data loss as measurement methodology matures.

Why this matters for government and NIST partnership

• Open methodology — no black-box scoring; every rubric, scorer, and judge prompt is public.
• Adversarial robustness focus — cyber capability proxy tests are directly relevant to NIST AI Safety Institute's evaluation research agenda and DoD AI assurance requirements.
• Hardware realism — results are tied to real hardware rather than cloud APIs; directly applicable to edge/tactical deployment assessments.
• Reproducibility — signed result bundles enable third-party verification of claimed benchmarks.

Current state (Jun 2026)

• Core infrastructure operational (runner, storage, reporting, signing, migration)
• Initial task suite seeded (baseline tier)
• Cyber capability proxy test design complete (C1–C7 categories; awaiting schema fix to begin data insertion)
• Results publication pipeline functional (Sigstore signing + bakeoff-results submission)

Next steps (Phase 2)

• Complete schema corrections and run first end-to-end benchmark (Tables are not defined properly and missing many fields. #38)
• Implement cyber capability proxy test tasks (C1–C7 from Cyber capability proxy test design — bakeoff integration plan #30)
• Publish first signed benchmark results to bakeoff-results
• Draft NIST AISI capability statement (derivative of this brief, formatted per NIST templates)
• Register at SBIR.gov and set agency-specific alerts (IARPA, DHS S&T, DoD OUSD R&E)

---

This draft is v0.1 — accuracy review against current implementation recommended before external distribution. Flag corrections or additions as comments here.

— Bastion // 100210ZJUN26