diff --git a/.github/workflows/pages.yml b/.github/workflows/pages.yml new file mode 100644 index 0000000..a789a13 --- /dev/null +++ b/.github/workflows/pages.yml @@ -0,0 +1,39 @@ +# Publishes the static Bluefy page after a push to main. +# It does not run on pull requests, does not sign an app, and does not +# change the phone runtime. GitHub Pages still has to be switched to +# GitHub Actions by a person before a deploy can succeed. + +name: Pages + +on: + push: + branches: [main] + +permissions: + contents: read + pages: write + id-token: write + +concurrency: + group: pages + cancel-in-progress: true + +jobs: + deploy: + if: github.ref == 'refs/heads/main' && github.event_name == 'push' + name: Publish web page + runs-on: ubuntu-latest + environment: + name: github-pages + url: ${{ steps.deployment.outputs.page_url }} + timeout-minutes: 10 + steps: + - name: Checkout + uses: actions/checkout@v4 + - name: Upload web folder + uses: actions/upload-pages-artifact@v5 + with: + path: web + - name: Deploy + id: deployment + uses: actions/deploy-pages@v5 diff --git a/README.md b/README.md index f9da7d0..c6c8ff8 100644 --- a/README.md +++ b/README.md @@ -9,4 +9,6 @@ Shared instructions for the owner, one teammate, Cursor, and Claude Code: [docs/ Supervised iPhone 16 and UNO R4 WiFi cane demo: [docs/PROTOTYPE.md](docs/PROTOTYPE.md). +Bluefy web page, still processed on the phone: [web/index.html](web/index.html). Walkthrough: [docs/WEB_DEMO.md](docs/WEB_DEMO.md). + Host logic tests, with no phone or board required: `make -C prototype test`. diff --git a/docs/COLLABORATION.md b/docs/COLLABORATION.md index ce35679..c1df786 100644 --- a/docs/COLLABORATION.md +++ b/docs/COLLABORATION.md @@ -58,7 +58,7 @@ Keep each pull request to one task. A reviewer should be able to check the diff GitHub fills in [`.github/PULL_REQUEST_TEMPLATE.md`](../.github/PULL_REQUEST_TEMPLATE.md) for new pull requests once the template is on the default branch, `main`. -In the pull request, name the review steps and the command you ran. One job validates these workflow files and runs the host cane-logic tests. Another job compiles the unsigned iOS app and the UNO R4 WiFi sketch. Those jobs do not sign, install, or flash anything, and they do not certify the demo as safe. Runtime video stays on the phone. +In the pull request, name the review steps and the command you ran. One job validates these workflow files and runs the host cane-logic tests. Another job compiles the unsigned iOS app and the UNO R4 WiFi sketch. A Pages workflow can publish the static web folder only from `main` after a person selects GitHub Actions as the Pages source. Those jobs do not sign, install, or flash the phone app, and they do not certify the demo as safe. Runtime video stays on the phone. The author opens the pull request and stops. A human merges. Prefer approval from a human who did not author the change. When the other person is unavailable, the repository owner may merge and records that on the pull request. An agent is not an approver. @@ -135,6 +135,7 @@ Issue and pull request template behavior is documented in [About issue and pull - [`scripts/check_workflow.py`](../scripts/check_workflow.py) checks these files, local links, the Claude Code import, and the workflow syntax this repo can parse without added packages. - [`.github/workflows/workflow-check.yml`](../.github/workflows/workflow-check.yml) runs that script, then `make -C prototype test`, with Python 3 and a C compiler already on the GitHub-hosted runner and GitHub's `actions/checkout` action at major version 4. That workflow installs nothing and reads no secrets. - [`.github/workflows/prototype-build.yml`](../.github/workflows/prototype-build.yml) compiles the unsigned iPhone app on a GitHub-hosted Mac and compiles the UNO R4 WiFi sketch on Linux. It reads repository contents only. It does not sign or install the app. +- [`.github/workflows/pages.yml`](../.github/workflows/pages.yml) publishes `web/` with GitHub's Pages actions after a push to `main`. It does not run for pull requests. After changing any of these files, run this from the repository root: diff --git a/docs/PROTOTYPE.md b/docs/PROTOTYPE.md index 03fcf0f..aa0ca6c 100644 --- a/docs/PROTOTYPE.md +++ b/docs/PROTOTYPE.md @@ -6,6 +6,8 @@ The phone warns with uncertainty. It does not invent a safe path. A quiet hazard At runtime the path is the iPhone 16, then Bluetooth LE, then the UNO R4 WiFi, then the four actuator drivers. The phone processes the camera and motion on the device. No laptop and no cloud service infer a hazard. GitHub Actions can compile the unsigned app and the sketch. That compile job is not part of the running demo. +The Bluefy page in [docs/WEB_DEMO.md](WEB_DEMO.md) is the path that can run without a Mac. It keeps video on the phone. Its hazard uses a separate image-heuristic packet. The native camera packet still cannot carry an elevated hazard. + ## Fixed demo requirements - The phone is body-mounted. This repository does not design the mount. diff --git a/docs/WEB_DEMO.md b/docs/WEB_DEMO.md new file mode 100644 index 0000000..0204e6b --- /dev/null +++ b/docs/WEB_DEMO.md @@ -0,0 +1,43 @@ +# Bluefy page + +The page is [web/index.html](../web/index.html). It is the phone runtime for this increment. A standard iPhone 16, with Bluefy installed by you, opens the page. The page uses the rear camera, runs COCO-SSD in the browser, and writes the existing 18-byte Bluetooth packet to the UNO R4 WiFi. A laptop is not part of that runtime. + +This repository has not run that combination on a phone. A green test log is not a Bluefy result. + +## What the packet means + +The native camera path still cannot send an elevated hazard. The web page uses a separate source, image heuristic, value 3. + +- D2 pulses only for an elevated claim. From the web page that means a supported box is in the center of the picture, or you turned on the manual D2 switch. It is not a measured collision. +- D3 pulses when the camera, model, or page is not ready. Quiet D3 is not an all-clear. +- D4 pulses only for a manual left or right label. That label is not a clear direction. +- D5 is driven by the board. A short blip means packets are arriving. A fast pattern means the 500 ms watchdog expired. The page tries to send every 200 ms while it is connected. + +The image-heuristic packet is rejected if it contains a distance or a route. The model does not report meters, speed, or time to contact. If the model fails, the page shows the error and does not invent a warning from brightness. + +## Software already in the repo + +- Sketch: [prototype/firmware/uno_r4_wifi/uno_r4_wifi.ino](../prototype/firmware/uno_r4_wifi/uno_r4_wifi.ino) +- The compile job that passed on the previous pull request used Arduino CLI 1.5.1, board package `arduino:renesas_uno` 1.6.0, and ArduinoBLE 2.1.0. This page keeps those versions. +- Pick the board Arduino UNO R4 WiFi. The sketch calls `Serial.begin(115200)` and prints `Cane demo controller`, `Advertising CaneDemo as a BLE peripheral.`, `accepted source=`, and `rejected packet`. +- Pinned browser downloads, only when you tap Load object model: TensorFlow.js 4.22.0 and COCO-SSD 2.2.3 from jsDelivr, both Apache-2.0. The weights are the library's `lite_mobilenet_v2` files at `https://storage.googleapis.com/tfjs-models/savedmodel/ssdlite_mobilenet_v2/model.json`. The camera picture is not sent anywhere. + +## What only you can do + +1. Merge the pull request. +2. In the repository Settings, open Pages, and set the source to GitHub Actions. The workflow [pages.yml](../.github/workflows/pages.yml) does not run for a pull request, and it cannot switch that setting. Until that source is selected, there is no published page. `https://pschmeln.github.io/agent-workflow/` returned 404 when this branch was prepared. +3. After the Pages job on `main` succeeds, open the URL printed by that job in Bluefy on the iPhone 16. Do not type a guessed address. +4. Install Bluefy yourself. Flash the sketch. Wire four external drivers to D2, D3, D4, and D5. Do not power actuators from the header pins. +5. With a sighted spotter, use the page sections one at a time, then the combined switch. Background the app and confirm D5 changes to the fast pattern. Do not cover anyone's eyes. + +## Actuator check + +| What you do | What should move | +| --- | --- | +| Phone not connected | D5 fast, D2 D3 D4 off | +| Connected, no manual switch and no combined warning | D3 unavailable pattern, D5 slow blip | +| Manual D2 | D2 pulses, serial `source=simulation` | +| Manual D4 left or right label | D4 pulses. The label is not a safe turn | +| Combined warning on, supported object in the center box | D2 pulses, serial `source=image-heuristic` | +| Combined warning on, nothing in the center | D2 off. The page says this is not an all-clear | +| Leave Bluefy or disconnect | D5 fast within about a second | diff --git a/prototype/Makefile b/prototype/Makefile index 8b14259..78b86cf 100644 --- a/prototype/Makefile +++ b/prototype/Makefile @@ -3,12 +3,18 @@ CFLAGS ?= -std=c11 -Wall -Wextra -Werror -I firmware/uno_r4_wifi .PHONY: test clean -test: build/test_cane +test: build/test_cane build/print_fixture ./build/test_cane + ./build/print_fixture > build/packets.json + node --test ../web/logic.test.mjs build/test_cane: firmware/uno_r4_wifi/cane_logic.c firmware/uno_r4_wifi/cane_logic.h tests/test_cane.c mkdir -p build $(CC) $(CFLAGS) -o $@ firmware/uno_r4_wifi/cane_logic.c tests/test_cane.c -lm +build/print_fixture: firmware/uno_r4_wifi/cane_logic.c firmware/uno_r4_wifi/cane_logic.h tests/print_fixture.c + mkdir -p build + $(CC) $(CFLAGS) -o $@ firmware/uno_r4_wifi/cane_logic.c tests/print_fixture.c -lm + clean: rm -rf build diff --git a/prototype/firmware/uno_r4_wifi/cane_logic.c b/prototype/firmware/uno_r4_wifi/cane_logic.c index c00a0cc..2ab6760 100644 --- a/prototype/firmware/uno_r4_wifi/cane_logic.c +++ b/prototype/firmware/uno_r4_wifi/cane_logic.c @@ -89,6 +89,19 @@ void cane_assess(const cane_input_t *in, cane_assessment_t *out) { return; } + if (in->source == CANE_SOURCE_IMAGE_HEURISTIC) { + clear_plane(out); + out->route_cue = CANE_ROUTE_NONE; + if (!in->image_pipeline_ready) { + out->system_unavailable = 1; + out->hazard = CANE_HAZARD_NO_CLAIM; + return; + } + out->system_unavailable = 0; + out->hazard = in->image_heuristic_elevated ? CANE_HAZARD_ELEVATED : CANE_HAZARD_NO_CLAIM; + return; + } + if (in->source == CANE_SOURCE_SIMULATION) { out->hazard = in->simulation_elevated_hazard ? CANE_HAZARD_ELEVATED : CANE_HAZARD_NO_CLAIM; out->system_unavailable = in->simulation_unavailable ? 1 : 0; @@ -148,7 +161,8 @@ void cane_command_from_assessment( } int cane_command_admissible(const cane_command_t *command) { - if (command->source != CANE_SOURCE_CAMERA && command->source != CANE_SOURCE_SIMULATION) { + if (command->source != CANE_SOURCE_CAMERA && command->source != CANE_SOURCE_SIMULATION && + command->source != CANE_SOURCE_IMAGE_HEURISTIC) { return 0; } if (command->hazard != CANE_HAZARD_NO_CLAIM && command->hazard != CANE_HAZARD_ELEVATED) { @@ -169,6 +183,11 @@ int cane_command_admissible(const cane_command_t *command) { return 0; } } + if (command->source == CANE_SOURCE_IMAGE_HEURISTIC) { + if (command->route_cue != CANE_ROUTE_NONE || command->plane_hit_cm != CANE_PLANE_CM_NONE) { + return 0; + } + } return 1; } diff --git a/prototype/firmware/uno_r4_wifi/cane_logic.h b/prototype/firmware/uno_r4_wifi/cane_logic.h index 4c7e607..0e94211 100644 --- a/prototype/firmware/uno_r4_wifi/cane_logic.h +++ b/prototype/firmware/uno_r4_wifi/cane_logic.h @@ -39,7 +39,8 @@ extern "C" { enum { CANE_SOURCE_CAMERA = 1, - CANE_SOURCE_SIMULATION = 2 + CANE_SOURCE_SIMULATION = 2, + CANE_SOURCE_IMAGE_HEURISTIC = 3 }; enum { @@ -84,6 +85,8 @@ typedef struct { int simulation_elevated_hazard; int simulation_unavailable; int route_cue; + int image_heuristic_elevated; + int image_pipeline_ready; } cane_input_t; typedef struct { diff --git a/prototype/firmware/uno_r4_wifi/uno_r4_wifi.ino b/prototype/firmware/uno_r4_wifi/uno_r4_wifi.ino index 6b538d9..638495a 100644 --- a/prototype/firmware/uno_r4_wifi/uno_r4_wifi.ino +++ b/prototype/firmware/uno_r4_wifi/uno_r4_wifi.ino @@ -35,9 +35,24 @@ static void set_pin_low(int pin) { digitalWrite(pin, LOW); } +static const char *source_name(int source) { + if (source == CANE_SOURCE_CAMERA) { + return "camera"; + } + if (source == CANE_SOURCE_SIMULATION) { + return "simulation"; + } + if (source == CANE_SOURCE_IMAGE_HEURISTIC) { + return "image-heuristic"; + } + return "unknown"; +} + static void print_outputs(uint32_t now_ms, const cane_outputs_t *out) { Serial.print("ms="); Serial.print(now_ms); + Serial.print(" source="); + Serial.print(source_name(link.command.source)); Serial.print(" link="); Serial.print(out->link_state == CANE_LINK_OUTAGE ? "outage" : "heartbeat"); Serial.print(" pins hazard="); @@ -112,6 +127,13 @@ void loop() { Serial.print("ms="); Serial.print(now_ms); Serial.println(" rejected packet"); + } else { + Serial.print("ms="); + Serial.print(now_ms); + Serial.print(" accepted source="); + Serial.print(source_name(link.command.source)); + Serial.print(" seq="); + Serial.println(link.command.sequence); } } } diff --git a/prototype/ios/CaneDemo/CaneDemo/DemoController.swift b/prototype/ios/CaneDemo/CaneDemo/DemoController.swift index 8849c3a..2a3bfbd 100644 --- a/prototype/ios/CaneDemo/CaneDemo/DemoController.swift +++ b/prototype/ios/CaneDemo/CaneDemo/DemoController.swift @@ -135,7 +135,9 @@ final class DemoController: NSObject, ObservableObject, ARSessionDelegate, CBCen plane_hit_meters: 0, simulation_elevated_hazard: 0, simulation_unavailable: 0, - route_cue: Int32(CANE_ROUTE_NONE) + route_cue: Int32(CANE_ROUTE_NONE), + image_heuristic_elevated: 0, + image_pipeline_ready: 0 ) if sourceIsSimulation { input.simulation_elevated_hazard = Int32(simulatedHazard ? 1 : 0) diff --git a/prototype/tests/print_fixture.c b/prototype/tests/print_fixture.c new file mode 100644 index 0000000..dd64388 --- /dev/null +++ b/prototype/tests/print_fixture.c @@ -0,0 +1,46 @@ +#include "cane_logic.h" + +#include +#include + +static void emit(const char *name, const cane_command_t *command, int last) { + uint8_t bytes[CANE_PACKET_BYTES]; + int ok = cane_command_encode(command, bytes); + size_t i; + printf(" {\"name\":\"%s\",\"ok\":%d,\"bytes\":[", name, ok); + if (ok) { + for (i = 0; i < CANE_PACKET_BYTES; i++) { + printf("%s%u", i == 0 ? "" : ",", bytes[i]); + } + } + printf("]}%s\n", last ? "" : ","); +} + +static cane_command_t command_for(int source, int hazard, int unavailable, int route, uint16_t plane_cm) { + cane_command_t command; + memset(&command, 0, sizeof(command)); + command.source = source; + command.hazard = hazard; + command.system_unavailable = unavailable; + command.route_cue = route; + command.tracking = CANE_TRACKING_NORMAL; + command.plane_hit_cm = plane_cm; + command.sequence = 9; + command.time_ms_low = 200; + return command; +} + +int main(void) { + cane_command_t simulation = command_for( + CANE_SOURCE_SIMULATION, CANE_HAZARD_ELEVATED, 0, CANE_ROUTE_DEMO_LEFT, CANE_PLANE_CM_NONE); + cane_command_t image = command_for( + CANE_SOURCE_IMAGE_HEURISTIC, CANE_HAZARD_ELEVATED, 0, CANE_ROUTE_NONE, CANE_PLANE_CM_NONE); + cane_command_t camera = command_for( + CANE_SOURCE_CAMERA, CANE_HAZARD_NO_CLAIM, 1, CANE_ROUTE_NONE, CANE_PLANE_CM_NONE); + printf("[\n"); + emit("simulation", &simulation, 0); + emit("image", &image, 0); + emit("camera", &camera, 1); + printf("]\n"); + return 0; +} diff --git a/prototype/tests/test_cane.c b/prototype/tests/test_cane.c index cbafadc..9e320e4 100644 --- a/prototype/tests/test_cane.c +++ b/prototype/tests/test_cane.c @@ -283,6 +283,47 @@ static void test_assessment_command_is_sendable(void) { expect(packet[4] == CANE_ROUTE_NONE, "packet route is none"); } +static void test_image_heuristic_is_not_a_distance(void) { + cane_input_t input = camera_ready(); + cane_assessment_t assessment; + cane_command_t command; + uint8_t packet[CANE_PACKET_BYTES]; + + input.image_heuristic_elevated = 1; + cane_assess(&input, &assessment); + expect(assessment.source == CANE_SOURCE_CAMERA, "camera source ignores the image flag"); + expect(assessment.hazard == CANE_HAZARD_NO_CLAIM, "camera source still cannot raise a hazard"); + + input = camera_ready(); + input.source = CANE_SOURCE_IMAGE_HEURISTIC; + input.image_pipeline_ready = 1; + input.image_heuristic_elevated = 1; + input.has_plane_hit = 1; + input.plane_hit_meters = 0.4f; + input.route_cue = CANE_ROUTE_DEMO_RIGHT; + cane_assess(&input, &assessment); + expect(assessment.hazard == CANE_HAZARD_ELEVATED, "fresh image heuristic can raise a hazard"); + expect(assessment.system_unavailable == 0, "fresh image pipeline is available"); + expect(assessment.route_cue == CANE_ROUTE_NONE, "image heuristic drops a route cue"); + expect(assessment.plane_hit_valid == 0, "image heuristic drops a plane distance"); + cane_command_from_assessment(&assessment, 4, 50, &command); + expect(cane_command_encode(&command, packet) == 1, "image heuristic packet encodes"); + expect(packet[1] == CANE_SOURCE_IMAGE_HEURISTIC, "packet source is image heuristic"); + expect(packet[14] == 0xFF && packet[15] == 0xFF, "image heuristic packet has no distance"); + + input.image_pipeline_ready = 0; + cane_assess(&input, &assessment); + expect(assessment.hazard == CANE_HAZARD_NO_CLAIM, "stale image pipeline clears the hazard"); + expect(assessment.system_unavailable == 1, "stale image pipeline is unavailable"); + + fill_command(&command, CANE_SOURCE_IMAGE_HEURISTIC, CANE_HAZARD_ELEVATED, CANE_ROUTE_DEMO_LEFT); + command.plane_hit_cm = CANE_PLANE_CM_NONE; + expect(cane_command_encode(&command, packet) == 0, "image heuristic route is rejected"); + fill_command(&command, CANE_SOURCE_IMAGE_HEURISTIC, CANE_HAZARD_ELEVATED, CANE_ROUTE_NONE); + command.plane_hit_cm = 30; + expect(cane_command_encode(&command, packet) == 0, "image heuristic distance is rejected"); +} + int main(void) { test_actuator_pins(); test_crc_vector(); @@ -296,6 +337,7 @@ int main(void) { test_patterns(); test_status_roundtrip(); test_assessment_command_is_sendable(); + test_image_heuristic_is_not_a_distance(); if (failures != 0) { fprintf(stderr, "%d failure(s)\n", failures); return 1; diff --git a/scripts/check_workflow.py b/scripts/check_workflow.py index 549712e..ff2653e 100644 --- a/scripts/check_workflow.py +++ b/scripts/check_workflow.py @@ -19,12 +19,14 @@ ".github/PULL_REQUEST_TEMPLATE.md", ".github/ISSUE_TEMPLATE/task.md", "docs/PROTOTYPE.md", + "docs/WEB_DEMO.md", ) REQUIRED_FILES = MARKDOWN_FILES + ( ".gitignore", ".github/workflows/workflow-check.yml", ".github/workflows/prototype-build.yml", + ".github/workflows/pages.yml", "scripts/check_workflow.py", ) @@ -266,6 +268,27 @@ def validate_build_workflow(text: str, report: Report) -> None: report.error("prototype build workflow must check out once per job and use no other action") +def validate_pages_workflow(text: str, report: Report) -> None: + forbidden = ("pull_request_target", "secrets.", "workflow_run:", "pull_request:") + for token in forbidden: + if token in text: + report.error(f"pages workflow must not contain {token}") + required = ( + "branches: [main]", + "contents: read", + "pages: write", + "id-token: write", + "actions/checkout@v4", + "actions/upload-pages-artifact@v5", + "actions/deploy-pages@v5", + "path: web", + "refs/heads/main", + ) + for item in required: + if item not in text: + report.error(f"pages workflow is missing {item}") + + def split_front_matter(text: str) -> tuple[str, str]: if not text.startswith("---\n"): raise ValueError("issue template is missing opening front matter") @@ -519,6 +542,9 @@ def main(argv: list[str]) -> int: build_workflow = ROOT / ".github/workflows/prototype-build.yml" if build_workflow.is_file(): validate_build_workflow(build_workflow.read_text(encoding="utf-8"), report) + pages_workflow = ROOT / ".github/workflows/pages.yml" + if pages_workflow.is_file(): + validate_pages_workflow(pages_workflow.read_text(encoding="utf-8"), report) issue = ROOT / ".github/ISSUE_TEMPLATE/task.md" if issue.is_file(): validate_issue_template(issue.read_text(encoding="utf-8"), report) diff --git a/web/app.js b/web/app.js new file mode 100644 index 0000000..cbc187e --- /dev/null +++ b/web/app.js @@ -0,0 +1,393 @@ +import { + COMMAND_UUID, + HAZARD_ELEVATED, + HEARTBEAT_MS, + SERVICE_UUID, + SOURCE_IMAGE_HEURISTIC, + SOURCE_SIMULATION, + STATUS_UUID, + createWriteQueue, + decodeStatus, + encodeCommand, +} from "./protocol.js"; +import { claimForMode } from "./state.js"; +import { centerZoneHit, updateTracks } from "./tracking.js"; + +const TFJS_URL = "https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@4.22.0/dist/tf.min.js"; +const COCO_URL = "https://cdn.jsdelivr.net/npm/@tensorflow-models/coco-ssd@2.2.3/dist/coco-ssd.min.js"; +const MODEL_BASE = "lite_mobilenet_v2"; + +const diagnostics = document.querySelector("#diagnostics"); +const scriptState = document.querySelector("#script-state"); +const orientationText = document.querySelector("#orientation"); +const cameraStatus = document.querySelector("#camera-status"); +const modelStatus = document.querySelector("#model-status"); +const detectStatus = document.querySelector("#detect-status"); +const bleStatus = document.querySelector("#ble-status"); +const boardStatus = document.querySelector("#board-status"); +const claimStatus = document.querySelector("#claim-status"); +const video = document.querySelector("#video"); +const overlay = document.querySelector("#overlay"); +const manualMode = document.querySelector("#manual-mode"); +const manualHazard = document.querySelector("#manual-hazard"); +const manualUnavailable = document.querySelector("#manual-unavailable"); +const manualRoute = document.querySelector("#manual-route"); +const detectEnable = document.querySelector("#detect-enable"); + +const state = { + cameraOn: false, + detecting: false, + model: null, + modelReady: false, + modelError: false, + lastFrameMs: null, + tracks: [], + zoneHit: null, + device: null, + server: null, + commandChar: null, + queue: null, + reconnectsLeft: 0, + sequence: 0, + stream: null, +}; + +scriptState.textContent = "Page script started. Camera inference on Bluefy has not been tried from this repository."; + +function setBox(element, text, kind) { + element.textContent = text; + element.className = kind; +} + +function diagnosticsText() { + const lines = [ + window.isSecureContext + ? "This page is a secure context." + : "This page is not a secure context. Camera and Bluetooth need HTTPS.", + navigator.bluetooth + ? "Web Bluetooth is present. That does not prove Bluefy can use it together with the camera." + : "Web Bluetooth is missing. Bluefy is expected to provide it; this page cannot add it.", + navigator.mediaDevices && navigator.mediaDevices.getUserMedia + ? "Camera API is present." + : "Camera API is missing.", + ]; + setBox(diagnostics, lines.join(" "), window.isSecureContext ? "ok" : "bad"); +} + +function sourceName(source) { + if (source === SOURCE_SIMULATION) { + return "manual simulation"; + } + if (source === SOURCE_IMAGE_HEURISTIC) { + return "image heuristic"; + } + return "other"; +} + +function currentClaim() { + const frameAgeMs = state.lastFrameMs === null ? null : performance.now() - state.lastFrameMs; + const mode = detectEnable.checked ? "detect" : manualMode.checked ? "manual" : "idle"; + return claimForMode({ + mode, + tabHidden: document.visibilityState === "hidden", + cameraOn: state.cameraOn, + modelReady: state.modelReady, + modelError: state.modelError, + frameAgeMs, + detectionEnabled: detectEnable.checked, + zoneHit: Boolean(state.zoneHit), + hazard: manualHazard.checked, + unavailable: manualUnavailable.checked, + route: Number(manualRoute.value), + }); +} + +function describeClaim(claim) { + const hazard = claim.hazard === HAZARD_ELEVATED ? "elevated image or manual hazard" : "no hazard claim"; + const availability = claim.unavailable ? "unavailable" : "pipeline running"; + const hit = state.zoneHit + ? `Center box: ${state.zoneHit.className} score ${state.zoneHit.score.toFixed(2)} id ${state.zoneHit.trackId}.` + : "No supported class in the center zone."; + claimStatus.textContent = `${sourceName(claim.source)}. ${hazard}. ${availability}. ${hit} This is not an all-clear.`; +} + +async function sendClaim() { + const claim = currentClaim(); + describeClaim(claim); + if (!state.queue || !state.commandChar) { + return; + } + state.sequence = (state.sequence + 1) >>> 0; + const bytes = encodeCommand({ + source: claim.source, + hazard: claim.hazard, + unavailable: claim.unavailable, + route: claim.route, + tracking: claim.tracking, + sceneDepth: 0, + sequence: state.sequence, + timeMsLow: Date.now() & 0xffff, + planeCm: claim.planeCm, + }); + if (!bytes) { + setBox(bleStatus, "The shared packet rules refused this claim. Nothing was sent.", "bad"); + return; + } + try { + await state.queue.enqueue(bytes); + } catch (error) { + setBox(bleStatus, `Bluetooth write failed: ${error.message}`, "bad"); + } +} + +function drawTracks() { + const width = video.videoWidth; + const height = video.videoHeight; + if (!width || !height) { + return; + } + overlay.width = width; + overlay.height = height; + const context = overlay.getContext("2d"); + context.clearRect(0, 0, width, height); + context.strokeStyle = "#ffe14a"; + context.lineWidth = 3; + context.font = "20px sans-serif"; + context.fillStyle = "#ffe14a"; + for (const track of state.tracks) { + context.strokeRect(track.bbox.x, track.bbox.y, track.bbox.w, track.bbox.h); + context.fillText(`${track.id} ${track.className} ${track.score.toFixed(2)}`, track.bbox.x, track.bbox.y + 22); + } + const x = width * 0.25; + const y = height * 0.25; + context.strokeStyle = "#ffffff"; + context.strokeRect(x, y, width * 0.5, height * 0.5); +} + +let detectTimer = 0; +async function detectOnce() { + if (!state.model || !state.cameraOn || state.modelError) { + return; + } + try { + const predictions = await state.model.detect(video, 20, 0.4); + const now = performance.now(); + const detections = predictions.map((prediction) => ({ + className: prediction.class, + score: prediction.score, + bbox: { + x: prediction.bbox[0], + y: prediction.bbox[1], + w: prediction.bbox[2], + h: prediction.bbox[3], + }, + })); + state.tracks = updateTracks(state.tracks, detections, now); + state.lastFrameMs = now; + state.zoneHit = centerZoneHit(state.tracks, video.videoWidth, video.videoHeight, now); + drawTracks(); + detectStatus.textContent = state.zoneHit + ? `Experimental center warning: ${state.zoneHit.className}, score ${state.zoneHit.score.toFixed(2)}, track ${state.zoneHit.trackId}. Not a distance.` + : "No supported class in the center zone. That is not an all-clear."; + } catch (error) { + state.modelError = true; + state.zoneHit = null; + setBox(modelStatus, `Model stopped: ${error.message}. No brightness substitute is used.`, "bad"); + } +} + +function detectionLoop(now) { + if (state.cameraOn && state.modelReady && !state.modelError && !state.detecting && now - detectTimer > 200) { + detectTimer = now; + state.detecting = true; + detectOnce().finally(() => { + state.detecting = false; + }); + } + if (state.cameraOn) { + requestAnimationFrame(detectionLoop); + } +} + +async function loadScript(url) { + await new Promise((resolve, reject) => { + const script = document.createElement("script"); + script.src = url; + script.onload = resolve; + script.onerror = () => reject(new Error(`Could not download ${url}`)); + document.head.appendChild(script); + }); +} + +document.querySelector("#camera-start").addEventListener("click", async () => { + try { + const stream = await navigator.mediaDevices.getUserMedia({ + audio: false, + video: { facingMode: { ideal: "environment" } }, + }); + state.stream = stream; + video.srcObject = stream; + state.cameraOn = true; + state.lastFrameMs = null; + setBox(cameraStatus, "Rear camera is on. The picture stays in this browser.", "ok"); + requestAnimationFrame(detectionLoop); + } catch (error) { + state.cameraOn = false; + setBox(cameraStatus, `Camera did not start: ${error.name || "Error"} ${error.message}`, "bad"); + } +}); + +document.querySelector("#camera-stop").addEventListener("click", () => { + state.cameraOn = false; + state.lastFrameMs = null; + state.tracks = []; + state.zoneHit = null; + if (state.stream) { + for (const track of state.stream.getTracks()) { + track.stop(); + } + } + video.srcObject = null; + setBox(cameraStatus, "Camera is off.", "warn"); + detectStatus.textContent = "Camera is off. No detection is not an all-clear."; +}); + +document.querySelector("#model-load").addEventListener("click", async () => { + state.modelError = false; + setBox(modelStatus, "Downloading TensorFlow.js and COCO-SSD onto this phone.", "warn"); + try { + if (!window.tf) { + await loadScript(TFJS_URL); + } + if (!window.cocoSsd) { + await loadScript(COCO_URL); + } + state.model = await window.cocoSsd.load({ base: MODEL_BASE }); + state.modelReady = true; + setBox( + modelStatus, + "Model loaded on this phone. Weights came from storage.googleapis.com. The picture is not uploaded.", + "ok" + ); + } catch (error) { + state.modelReady = false; + state.modelError = true; + setBox(modelStatus, `Model did not load: ${error.message}`, "bad"); + } +}); + +function manualEnabled(enabled) { + manualHazard.disabled = !enabled; + manualUnavailable.disabled = !enabled; + manualRoute.disabled = !enabled; +} + +manualMode.addEventListener("change", () => { + if (manualMode.checked) { + detectEnable.checked = false; + } + manualEnabled(manualMode.checked); +}); + +detectEnable.addEventListener("change", () => { + if (detectEnable.checked) { + manualMode.checked = false; + manualHazard.checked = false; + manualUnavailable.checked = false; + manualRoute.value = "0"; + } + manualEnabled(false); +}); + +async function connectGatt(device) { + const server = await device.gatt.connect(); + const service = await server.getPrimaryService(SERVICE_UUID); + const command = await service.getCharacteristic(COMMAND_UUID); + const status = await service.getCharacteristic(STATUS_UUID); + state.server = server; + state.commandChar = command; + state.queue = createWriteQueue(async (bytes) => { + if (command.properties.writeWithoutResponse) { + await command.writeValueWithoutResponse(bytes); + } else { + await command.writeValue(bytes); + } + }); + if (status.properties.notify) { + await status.startNotifications(); + status.addEventListener("characteristicvaluechanged", (event) => { + const decoded = decodeStatus(new Uint8Array(event.target.value.buffer)); + boardStatus.textContent = decoded + ? `Board ${decoded.linkState === 1 ? "link outage" : "link heartbeat"}, sequence ${decoded.sequence}, pins hazard ${decoded.hazardOn} unavailable ${decoded.unavailableOn} route ${decoded.routeOn} link ${decoded.linkOn}.` + : "Board status did not match the 8-byte status packet."; + }); + } + setBox(bleStatus, `Connected to ${device.name || "the board"}.`, "ok"); +} + +async function reconnect() { + if (!state.device || state.reconnectsLeft <= 0) { + setBox(bleStatus, "Bluetooth disconnected. Tap Connect to try again.", "bad"); + state.queue = null; + state.commandChar = null; + return; + } + state.reconnectsLeft -= 1; + setBox(bleStatus, "Bluetooth dropped. Trying to reconnect.", "warn"); + try { + await connectGatt(state.device); + state.reconnectsLeft = 3; + } catch (error) { + setTimeout(reconnect, 1000); + } +} + +document.querySelector("#ble-connect").addEventListener("click", async () => { + if (!navigator.bluetooth) { + setBox(bleStatus, "This page cannot see Web Bluetooth.", "bad"); + return; + } + try { + const device = await navigator.bluetooth.requestDevice({ + filters: [{ services: [SERVICE_UUID] }], + }); + state.device = device; + state.reconnectsLeft = 3; + device.addEventListener("gattserverdisconnected", () => { + state.queue = null; + state.commandChar = null; + reconnect(); + }); + await connectGatt(device); + } catch (error) { + setBox(bleStatus, `Bluetooth did not connect: ${error.message}`, "bad"); + } +}); + +document.querySelector("#rotation-button").addEventListener("click", async () => { + try { + if (typeof DeviceOrientationEvent !== "undefined" && typeof DeviceOrientationEvent.requestPermission === "function") { + const answer = await DeviceOrientationEvent.requestPermission(); + if (answer !== "granted") { + orientationText.textContent = "Rotation permission was not granted. Rotation is not position."; + return; + } + } + window.addEventListener("deviceorientation", (event) => { + orientationText.textContent = `Rotation only: alpha ${event.alpha}, beta ${event.beta}, gamma ${event.gamma}. This is not 6DoF and not a distance.`; + }, { once: false }); + } catch (error) { + orientationText.textContent = `Rotation is unavailable: ${error.message}. It is not used for a warning.`; + } +}); + +document.addEventListener("visibilitychange", () => { + if (document.visibilityState === "hidden") { + state.zoneHit = null; + detectStatus.textContent = "This page is in the background. The image warning is cleared. That is not an all-clear."; + } +}); + +setInterval(sendClaim, HEARTBEAT_MS); +manualEnabled(false); +diagnosticsText(); diff --git a/web/index.html b/web/index.html new file mode 100644 index 0000000..e67dc8b --- /dev/null +++ b/web/index.html @@ -0,0 +1,77 @@ + + + + + + Cane demo for Bluefy + + + +

Cane demo

+ +

This page is a bench demo for a standard iPhone 16 with the Bluefy app, plus an Arduino UNO R4 WiFi. Video stays on the phone. Nothing here is a measured distance, a collision alarm, or a safe direction. A quiet hazard actuator is not an all-clear. We have not run this page on a phone.

+ +

1. What this phone can see

+
Checking this page.
+

Rotation: not requested. Rotation is not position and not a distance.

+ + +

2. Camera only

+

Starts the rear camera. It does not load the model and does not talk to the board.

+ + +

Camera is off.

+
+ + +
+ +

3. Model only

+

Loads a COCO-SSD model in this browser and draws boxes. Boxes are 2D image positions. They are not meters. This does not send a hazard until section 5 is on.

+ +

Model is not loaded.

+

No detection yet. No detection is not an all-clear.

+ +

4. Bluetooth only

+

Connects to the board named CaneDemo. The switches below are a manual simulation. They are not the camera.

+ +

Bluetooth is not connected.

+

No board status yet.

+ + + + +

D4's left and right labels are not clear directions. D5 is the board's own link light: a slow blip while packets arrive, and a fast pattern when packets stop.

+ +

5. Camera, model, and Bluetooth together

+ +

Combined warning is off.

+

The warning can turn on only for a person, bicycle, car, motorcycle, bus, or truck whose box center is in the middle of the picture and whose score is at least 0.5. That is an image guess. It is not proof of an obstacle and it does not choose a direction.

+ +

What you do, in order

+
    +
  1. Flash prototype/firmware/uno_r4_wifi/uno_r4_wifi.ino to an Arduino UNO R4 WiFi. In Arduino IDE pick that board, install ArduinoBLE 2.1.0, and use the UNO R4 board package 1.6.0. The sketch opens serial at 115200 and prints Cane demo controller and Advertising CaneDemo as a BLE peripheral.
  2. +
  3. Wire four external drivers. D2 is hazard, D3 is unavailable, D4 is the route demo, D5 is the link. Do not power an actuator from a header pin.
  4. +
  5. Install Bluefy yourself from the App Store. Open this HTTPS page in Bluefy on the iPhone 16. A laptop is not the runtime.
  6. +
  7. Read section 1. Then try camera alone, model alone, and Bluetooth alone. Then turn on section 5.
  8. +
  9. With a sighted spotter, check the physical outputs listed at the bottom. Do not cover anyone's eyes. The cane you already use stays the cane.
  10. +
+ +

What a spotter can check

+ + + + + diff --git a/web/logic.test.mjs b/web/logic.test.mjs new file mode 100644 index 0000000..19d1a4d --- /dev/null +++ b/web/logic.test.mjs @@ -0,0 +1,225 @@ +import assert from "node:assert/strict"; +import { readFileSync } from "node:fs"; +import test from "node:test"; + +import { + HAZARD_ELEVATED, + HAZARD_NO_CLAIM, + PLANE_CM_NONE, + ROUTE_DEMO_LEFT, + ROUTE_NONE, + SOURCE_CAMERA, + SOURCE_IMAGE_HEURISTIC, + SOURCE_SIMULATION, + TRACKING_NORMAL, + commandAdmissible, + crc16, + createWriteQueue, + encodeCommand, +} from "./protocol.js"; +import { claimForMode, imageHeuristicClaim, pipelineReady } from "./state.js"; +import { centerZoneHit, iou, updateTracks } from "./tracking.js"; + +function box(x, y, w, h) { + return { x, y, w, h }; +} + +test("crc16 matches the CCITT-FALSE vector", () => { + const digits = new TextEncoder().encode("123456789"); + assert.equal(crc16(digits), 0x29b1); +}); + +test("javascript packets match the C encoder", () => { + const fixture = JSON.parse( + readFileSync(new URL("../prototype/build/packets.json", import.meta.url), "utf8") + ); + const shared = { + unavailable: 0, + tracking: TRACKING_NORMAL, + sceneDepth: 0, + sequence: 9, + timeMsLow: 200, + planeCm: PLANE_CM_NONE, + }; + const expected = { + simulation: { + ...shared, + source: SOURCE_SIMULATION, + hazard: HAZARD_ELEVATED, + route: ROUTE_DEMO_LEFT, + }, + image: { + ...shared, + source: SOURCE_IMAGE_HEURISTIC, + hazard: HAZARD_ELEVATED, + route: ROUTE_NONE, + }, + camera: { + ...shared, + source: SOURCE_CAMERA, + hazard: HAZARD_NO_CLAIM, + unavailable: 1, + route: ROUTE_NONE, + }, + }; + for (const row of fixture) { + const encoded = encodeCommand(expected[row.name]); + assert.ok(encoded, row.name); + assert.deepEqual(Array.from(encoded), row.bytes, row.name); + } +}); + +test("image heuristic cannot carry a distance or a route", () => { + assert.equal( + commandAdmissible({ + source: SOURCE_IMAGE_HEURISTIC, + hazard: HAZARD_ELEVATED, + unavailable: 0, + route: ROUTE_DEMO_LEFT, + tracking: TRACKING_NORMAL, + planeCm: PLANE_CM_NONE, + }), + false + ); + assert.equal( + encodeCommand({ + source: SOURCE_IMAGE_HEURISTIC, + hazard: HAZARD_ELEVATED, + unavailable: 0, + route: ROUTE_NONE, + tracking: TRACKING_NORMAL, + sceneDepth: 0, + sequence: 1, + timeMsLow: 1, + planeCm: 40, + }), + null + ); + assert.equal( + commandAdmissible({ + source: SOURCE_CAMERA, + hazard: HAZARD_ELEVATED, + unavailable: 0, + route: ROUTE_NONE, + tracking: TRACKING_NORMAL, + planeCm: PLANE_CM_NONE, + }), + false + ); +}); + +test("iou keeps an id and expiry drops it", () => { + const first = updateTracks( + [], + [{ className: "person", score: 0.9, bbox: box(0, 0, 10, 10) }], + 1000 + ); + const second = updateTracks( + first, + [{ className: "person", score: 0.8, bbox: box(1, 1, 10, 10) }], + 1100 + ); + assert.equal(second.length, 1); + assert.equal(second[0].id, first[0].id); + assert.ok(iou(box(0, 0, 10, 10), box(1, 1, 10, 10)) > 0.3); + const moved = updateTracks( + second, + [{ className: "person", score: 0.8, bbox: box(100, 100, 10, 10) }], + 1200 + ); + assert.notEqual(moved[0].id, second[0].id); + const expired = updateTracks(second, [], 2000); + assert.equal(expired.length, 0); +}); + +test("center warning is 2D and ignores unsupported classes", () => { + const tracks = [ + { id: 1, className: "kite", score: 0.99, bbox: box(40, 40, 20, 20), lastSeen: 0 }, + { id: 2, className: "person", score: 0.2, bbox: box(40, 40, 20, 20), lastSeen: 0 }, + { id: 3, className: "person", score: 0.8, bbox: box(0, 0, 10, 10), lastSeen: 0 }, + { id: 4, className: "car", score: 0.7, bbox: box(40, 40, 20, 20), lastSeen: 0 }, + ]; + const hit = centerZoneHit(tracks, 100, 100, 0); + assert.equal(hit.className, "car"); + assert.equal(hit.zone, "center"); + assert.equal(centerZoneHit([], 100, 100, 0), null); +}); + +test("faults clear an image hazard and stay distinct from manual input", () => { + const fresh = imageHeuristicClaim({ + cameraOn: true, + modelReady: true, + modelError: false, + frameAgeMs: 100, + tabHidden: false, + detectionEnabled: true, + zoneHit: true, + }); + assert.equal(fresh.source, SOURCE_IMAGE_HEURISTIC); + assert.equal(fresh.hazard, HAZARD_ELEVATED); + assert.equal(fresh.planeCm, PLANE_CM_NONE); + assert.equal(fresh.route, ROUTE_NONE); + + const stale = imageHeuristicClaim({ + cameraOn: true, + modelReady: true, + modelError: false, + frameAgeMs: 1000, + tabHidden: false, + detectionEnabled: true, + zoneHit: true, + }); + assert.equal(stale.hazard, HAZARD_NO_CLAIM); + assert.equal(stale.unavailable, 1); + assert.equal(pipelineReady({ + cameraOn: false, + modelReady: true, + modelError: false, + frameAgeMs: 10, + tabHidden: false, + }), false); + + const hiddenManual = claimForMode({ + mode: "manual", + tabHidden: true, + cameraOn: false, + modelReady: false, + modelError: false, + frameAgeMs: null, + detectionEnabled: false, + zoneHit: false, + hazard: true, + unavailable: false, + route: ROUTE_DEMO_LEFT, + }); + assert.equal(hiddenManual.source, SOURCE_IMAGE_HEURISTIC); + assert.equal(hiddenManual.hazard, HAZARD_NO_CLAIM); + assert.notEqual(hiddenManual.source, SOURCE_SIMULATION); + + const manual = claimForMode({ + mode: "manual", + tabHidden: false, + hazard: true, + unavailable: false, + route: ROUTE_DEMO_LEFT, + }); + assert.equal(manual.source, SOURCE_SIMULATION); + assert.equal(manual.hazard, HAZARD_ELEVATED); + assert.equal(manual.route, ROUTE_DEMO_LEFT); +}); + +test("writes stay in order", async () => { + const seen = []; + let active = 0; + let maxActive = 0; + const queue = createWriteQueue(async (value) => { + active += 1; + maxActive = Math.max(maxActive, active); + await new Promise((resolve) => setTimeout(resolve, 5)); + seen.push(value); + active -= 1; + }); + await Promise.all([queue.enqueue("a"), queue.enqueue("b"), queue.enqueue("c")]); + assert.deepEqual(seen, ["a", "b", "c"]); + assert.equal(maxActive, 1); +}); diff --git a/web/protocol.js b/web/protocol.js new file mode 100644 index 0000000..2c3483c --- /dev/null +++ b/web/protocol.js @@ -0,0 +1,141 @@ +export const SERVICE_UUID = "8f3a1c20-7b6e-4d5a-9c11-6e2f0a9b1001"; +export const COMMAND_UUID = "8f3a1c20-7b6e-4d5a-9c11-6e2f0a9b1002"; +export const STATUS_UUID = "8f3a1c20-7b6e-4d5a-9c11-6e2f0a9b1003"; + +export const PACKET_VERSION = 1; +export const PACKET_BYTES = 18; +export const PLANE_CM_NONE = 0xffff; +export const WATCHDOG_MS = 500; +export const HEARTBEAT_MS = 200; +export const STALE_MS = 750; + +export const SOURCE_CAMERA = 1; +export const SOURCE_SIMULATION = 2; +export const SOURCE_IMAGE_HEURISTIC = 3; + +export const HAZARD_NO_CLAIM = 0; +export const HAZARD_ELEVATED = 1; + +export const ROUTE_NONE = 0; +export const ROUTE_DEMO_LEFT = 1; +export const ROUTE_DEMO_RIGHT = 2; + +export const TRACKING_UNSUPPORTED = 0; +export const TRACKING_UNAVAILABLE = 1; +export const TRACKING_NORMAL = 3; + +export function crc16(bytes) { + let crc = 0xffff; + for (let i = 0; i < bytes.length; i += 1) { + crc ^= bytes[i] << 8; + for (let bit = 0; bit < 8; bit += 1) { + if ((crc & 0x8000) !== 0) { + crc = ((crc << 1) ^ 0x1021) & 0xffff; + } else { + crc = (crc << 1) & 0xffff; + } + } + } + return crc; +} + +export function commandAdmissible(command) { + if ( + command.source !== SOURCE_CAMERA && + command.source !== SOURCE_SIMULATION && + command.source !== SOURCE_IMAGE_HEURISTIC + ) { + return false; + } + if (command.hazard !== HAZARD_NO_CLAIM && command.hazard !== HAZARD_ELEVATED) { + return false; + } + if (command.unavailable !== 0 && command.unavailable !== 1) { + return false; + } + if ( + command.route !== ROUTE_NONE && + command.route !== ROUTE_DEMO_LEFT && + command.route !== ROUTE_DEMO_RIGHT + ) { + return false; + } + if (command.tracking < TRACKING_UNSUPPORTED || command.tracking > TRACKING_NORMAL) { + return false; + } + if (command.source === SOURCE_CAMERA) { + if (command.hazard !== HAZARD_NO_CLAIM || command.route !== ROUTE_NONE) { + return false; + } + } + if (command.source === SOURCE_IMAGE_HEURISTIC) { + if (command.route !== ROUTE_NONE || command.planeCm !== PLANE_CM_NONE) { + return false; + } + } + return true; +} + +function write16(bytes, offset, value) { + bytes[offset] = value & 0xff; + bytes[offset + 1] = (value >> 8) & 0xff; +} + +function write32(bytes, offset, value) { + bytes[offset] = value & 0xff; + bytes[offset + 1] = (value >> 8) & 0xff; + bytes[offset + 2] = (value >> 16) & 0xff; + bytes[offset + 3] = (value >> 24) & 0xff; +} + +export function encodeCommand(command) { + if (!commandAdmissible(command)) { + return null; + } + const bytes = new Uint8Array(PACKET_BYTES); + bytes[0] = PACKET_VERSION; + bytes[1] = command.source; + bytes[2] = command.hazard; + bytes[3] = command.unavailable; + bytes[4] = command.route; + bytes[5] = command.tracking; + bytes[6] = command.sceneDepth ? 1 : 0; + bytes[7] = 0; + write32(bytes, 8, command.sequence >>> 0); + write16(bytes, 12, command.timeMsLow & 0xffff); + write16(bytes, 14, command.planeCm & 0xffff); + const crc = crc16(bytes.subarray(0, 16)); + write16(bytes, 16, crc); + return bytes; +} + +export function decodeStatus(bytes) { + if (bytes.length !== 8 || bytes[0] !== PACKET_VERSION || bytes[3] !== 0) { + return null; + } + if (bytes[1] !== 0 && bytes[1] !== 1) { + return null; + } + return { + linkState: bytes[1], + hazardOn: (bytes[2] & 1) !== 0, + unavailableOn: (bytes[2] & 2) !== 0, + routeOn: (bytes[2] & 4) !== 0, + linkOn: (bytes[2] & 8) !== 0, + sequence: (bytes[4] | (bytes[5] << 8) | (bytes[6] << 16) | (bytes[7] << 24)) >>> 0, + }; +} + +export function createWriteQueue(writeFn) { + let chain = Promise.resolve(); + return { + enqueue(bytes) { + const run = chain.then(() => writeFn(bytes)); + chain = run.then( + () => undefined, + () => undefined + ); + return run; + }, + }; +} diff --git a/web/state.js b/web/state.js new file mode 100644 index 0000000..b1d35f4 --- /dev/null +++ b/web/state.js @@ -0,0 +1,59 @@ +import { + HAZARD_ELEVATED, + HAZARD_NO_CLAIM, + PLANE_CM_NONE, + ROUTE_NONE, + SOURCE_IMAGE_HEURISTIC, + SOURCE_SIMULATION, + STALE_MS, + TRACKING_NORMAL, + TRACKING_UNAVAILABLE, +} from "./protocol.js"; + +export function pipelineReady(input) { + if (!input.cameraOn || !input.modelReady || input.modelError || input.tabHidden) { + return false; + } + if (input.frameAgeMs === null || input.frameAgeMs === undefined || input.frameAgeMs > STALE_MS) { + return false; + } + return true; +} + +export function imageHeuristicClaim(input) { + const ready = pipelineReady(input); + return { + source: SOURCE_IMAGE_HEURISTIC, + hazard: ready && input.detectionEnabled && input.zoneHit ? HAZARD_ELEVATED : HAZARD_NO_CLAIM, + unavailable: ready ? 0 : 1, + route: ROUTE_NONE, + planeCm: PLANE_CM_NONE, + sceneDepth: 0, + tracking: ready ? TRACKING_NORMAL : TRACKING_UNAVAILABLE, + experimental: true, + }; +} + +export function manualClaim(input) { + let route = input.route; + if (route !== 0 && route !== 1 && route !== 2) { + route = ROUTE_NONE; + } + return { + source: SOURCE_SIMULATION, + hazard: input.hazard ? HAZARD_ELEVATED : HAZARD_NO_CLAIM, + unavailable: input.unavailable ? 1 : 0, + route, + planeCm: PLANE_CM_NONE, + sceneDepth: 0, + tracking: input.tabHidden ? TRACKING_UNAVAILABLE : TRACKING_NORMAL, + experimental: false, + }; +} + +export function claimForMode(input) { + if (input.mode === "manual" && !input.tabHidden) { + return manualClaim(input); + } + return imageHeuristicClaim(input); +} diff --git a/web/styles.css b/web/styles.css new file mode 100644 index 0000000..f2c601b --- /dev/null +++ b/web/styles.css @@ -0,0 +1,69 @@ +:root { + color: #111; + background: #f4f1ea; + font-family: Georgia, "Times New Roman", serif; + font-size: 18px; + line-height: 1.4; +} + +body { + margin: 0 auto; + max-width: 42rem; + padding: 1rem; +} + +h1, h2 { + line-height: 1.2; +} + +button, select { + font: inherit; + min-height: 3rem; + margin: 0.25rem 0.25rem 0.25rem 0; +} + +.banner, .warn, .ok, .bad { + padding: 0.75rem; + margin: 0.75rem 0; +} + +.banner, .warn { + background: #fff4cc; + border: 2px solid #111; +} + +.ok { + background: #e5f6e5; +} + +.bad { + background: #ffd9d9; +} + +.stage { + position: relative; + background: #111; + min-height: 12rem; +} + +video, canvas { + width: 100%; + height: auto; + display: block; +} + +canvas { + position: absolute; + inset: 0; + width: 100%; + height: 100%; +} + +section { + margin: 1.5rem 0; +} + +label { + display: block; + margin: 0.4rem 0; +} diff --git a/web/tracking.js b/web/tracking.js new file mode 100644 index 0000000..b551c4a --- /dev/null +++ b/web/tracking.js @@ -0,0 +1,94 @@ +export const CENTER_MARGIN = 0.25; +export const IOU_THRESHOLD = 0.3; +export const TRACK_MAX_AGE_MS = 500; +export const SUPPORTED_CLASSES = ["person", "bicycle", "car", "motorcycle", "bus", "truck"]; +export const MIN_SCORE = 0.5; + +export function iou(a, b) { + const ax2 = a.x + a.w; + const ay2 = a.y + a.h; + const bx2 = b.x + b.w; + const by2 = b.y + b.h; + const ix = Math.max(0, Math.min(ax2, bx2) - Math.max(a.x, b.x)); + const iy = Math.max(0, Math.min(ay2, by2) - Math.max(a.y, b.y)); + const inter = ix * iy; + const union = a.w * a.h + b.w * b.h - inter; + if (union <= 0) { + return 0; + } + return inter / union; +} + +export function updateTracks(previous, detections, nowMs) { + const used = new Set(); + const next = []; + let nextId = previous.reduce((max, track) => Math.max(max, track.id), 0); + const pool = previous.filter((track) => nowMs - track.lastSeen <= TRACK_MAX_AGE_MS); + for (const detection of detections) { + let best = null; + let bestScore = IOU_THRESHOLD; + for (const track of pool) { + if (used.has(track.id)) { + continue; + } + const score = iou(track.bbox, detection.bbox); + if (score >= bestScore) { + best = track; + bestScore = score; + } + } + if (best) { + used.add(best.id); + next.push({ + id: best.id, + className: detection.className, + score: detection.score, + bbox: detection.bbox, + lastSeen: nowMs, + }); + } else { + nextId += 1; + next.push({ + id: nextId, + className: detection.className, + score: detection.score, + bbox: detection.bbox, + lastSeen: nowMs, + }); + } + } + return next; +} + +export function centerZoneHit(tracks, width, height, nowMs) { + if (width <= 0 || height <= 0) { + return null; + } + const x0 = width * CENTER_MARGIN; + const x1 = width * (1 - CENTER_MARGIN); + const y0 = height * CENTER_MARGIN; + const y1 = height * (1 - CENTER_MARGIN); + let best = null; + for (const track of tracks) { + if (nowMs - track.lastSeen > TRACK_MAX_AGE_MS) { + continue; + } + if (!SUPPORTED_CLASSES.includes(track.className) || track.score < MIN_SCORE) { + continue; + } + const cx = track.bbox.x + track.bbox.w / 2; + const cy = track.bbox.y + track.bbox.h / 2; + if (cx < x0 || cx > x1 || cy < y0 || cy > y1) { + continue; + } + if (!best || track.score > best.score) { + best = { + trackId: track.id, + className: track.className, + score: track.score, + zone: "center", + }; + } + } + return best; +}