Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 0 additions & 24 deletions .github/workflows/codeql.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,37 +32,14 @@ jobs:
fetch-depth: 0
persist-credentials: false

- name: Detect Swift build changes
id: changes
env:
BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.before }}
HEAD_SHA: ${{ github.event.pull_request.head.sha || github.sha }}
run: |
analyze=false
if [[ "${{ github.event_name }}" == "schedule" ]]; then
analyze=true
else
while IFS= read -r changed_path; do
case "$changed_path" in
.github/workflows/codeql.yml|Cargo.lock|Cargo.toml|Package.resolved|Package.swift|rust-toolchain.toml|Sources/*|apps/ios/voice_input/*|crates/*|scripts/build_ios_rust_ffi.sh|scripts/build_rust_ffi.sh|scripts/fetch_ios_asr_runtime.sh)
analyze=true
break
;;
esac
done < <(git diff --name-only --diff-filter=ACMRT "$BASE_SHA" "$HEAD_SHA")
fi
echo "analyze=$analyze" >> "$GITHUB_OUTPUT"

- name: Initialize CodeQL
if: steps.changes.outputs.analyze == 'true'
uses: github/codeql-action/init@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4.37.9
with:
build-mode: manual
languages: swift
queries: security-extended

- name: Build Swift products
if: steps.changes.outputs.analyze == 'true'
run: |
scripts/build_rust_ffi.sh
swift build --arch arm64
Expand All @@ -81,5 +58,4 @@ jobs:
ONLY_ACTIVE_ARCH=YES

- name: Perform CodeQL analysis
if: steps.changes.outputs.analyze == 'true'
uses: github/codeql-action/analyze@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4.37.9
6 changes: 3 additions & 3 deletions docs/game_plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,13 +72,13 @@ gated. See

| Gate | Contract | Current automated evidence |
| --- | --- | --- |
| HID dispatch | p50 ≤ 3 ms, p95 ≤ 8 ms, p99 ≤ 15 ms, max ≤ 30 ms across 10,000 transitions; no loss or duplication. | M15 current-source p50 0.011 ms, p95 0.017 ms, p99 0.028 ms, max 0.131 ms; 10,000 ordered dispatches. |
| HID dispatch | p50 ≤ 3 ms, p95 ≤ 8 ms, p99 ≤ 15 ms, max ≤ 30 ms across 10,000 transitions; no loss or duplication. | Current-source p50 0.011 ms, p95 0.016 ms, p99 0.022 ms, max 0.174 ms; 10,000 ordered dispatches. |
| Microphone activation | Warm maximum ≤ 250 ms. | p50 48.118 ms, p95/p99/max 87.050 ms across five starts; one-time preparation 149.539 ms. |
| Local AI semantic safety | No final candidate may violate requested casing, required list structure, or protected content; invalid output falls back to Edited text once. | Prompt-6 fixed 19-case corpus: Qwen 4B 2 semantic failures, Qwen 9B 0, Apple 1; every provider passes the typed gate. |
| Local AI refinement | Warm raw-final-to-refined p95 ≤ 1 s on the reference Mac. | Open: prompt-6 Qwen 4B p95 15.739 s includes one provider timeout. |
| Local AI end to end | Warm release-to-insertion p95 ≤ 1.5 s on the reference Mac. | Historical prompt-5 prewarmed M4 production-controller p95 1.004 s; prompt 6 requires a new run. |
| Local AI end to end | Warm release-to-insertion p95 ≤ 1.5 s on the reference Mac. | Prompt-6 prewarmed Qwen 4B production-controller p50 1.108 s, p95/p99/max 1.313 s across 19 samples; pass. |
| Local AI deadline | Preparation plus generation must fall back within three seconds after final speech text. | Deterministic deadline and late-output tests. |
| Voice History | Warm 5,000-session search p95 ≤ 250 ms; startup recovery precedes retention without delaying the input runtime. | M8 current-source p95 2.639 ms; current source passes 513 Swift tests in 76 suites plus 34 Rust domain/archive/model/ABI tests and two linked/native C consumers. |
| Voice History | Warm 5,000-session search p95 ≤ 250 ms; startup recovery precedes retention without delaying the input runtime. | M8 current-source p95 2.639 ms; current source passes 539 Swift tests in 80 suites plus 34 Rust domain/archive/model/ABI tests and two linked/native C consumers. |
| Privacy | Voice artifacts remain app-owned and local; no speech content is logged; no remote-capable provider receives a call; Ollama cannot reach a nonloopback endpoint. | Deterministic provider-boundary, SQLite/CAF, fallback, fixed-endpoint transport tests, and an iOS source/capability network scan. |
| iOS local ASR | File-ASR RTF ≤ 0.75 on the pinned native integration corpus; selected bytes are revalidated immediately before load. | `HC_RUN_IOS_ASR_PERFORMANCE=1` enforces the named-hardware gate; whisper.cpp `b4938` + `tiny.en` reference warm CPU RTF 0.0111. Every check runs real transcription correctness; Rust digest/runtime/capability/tamper and timed C/Swift result tests pass. |
| iOS local History | Final output is unavailable until Raw/Edited/Formatted plus audio evidence commit; configurable 90-day/1-GiB/2,000-artifact defaults retain transcripts after audio expiry. | Real SQLite/filesystem tests cover reload, search, digest evidence, migration, pinning, recovery protection, age/count/byte/low-disk expiry, post-commit maintenance failure, Data Protection where exposed, backup exclusion, partial cleanup, and orphan cleanup. |
Expand Down
13 changes: 7 additions & 6 deletions docs/local_ai_model_evaluation.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,16 +76,17 @@ before changing the default. The result reinforces that a model swap alone
cannot replace deterministic casing, typed list normalization, validation, and
fallback.

## Historical controller benchmark
## Prompt 6 production-controller benchmark

The prompt-5 recommended-model benchmark explicitly prepared Qwen 3.5 4B before
timing 17 production-controller samples:
The benchmark explicitly prepared Qwen 3.5 4B before timing all 19 prompt-6
cases through the production controller:

```bash
HC_RUN_LOCAL_AI_END_TO_END_BENCHMARK=1 \
swift test --filter measuresWarmReleaseToInsertionWithTheRecommendedModel
```

It measured release-to-insertion p50 0.773 seconds and p95/p99/maximum 1.004
seconds with one insertion per case. Re-run this benchmark before making a
prompt-6 production-latency claim.
It measured release-to-insertion p50 1.108 seconds and p95/p99/maximum 1.313
seconds with one insertion per case. The warm p95 passes the 1.5-second gate.
The controller result includes deterministic normalization, validation,
fallback, and insertion; it does not replace the provider-only latency row.