HoTH · Heroes of the Hay

حيّ — the social network for your neighbourhood, with civic reporting at its core

Snap a hazard. Talk for a minute, in Arabic or English. Your neighbours back it up and the crew fixes it, and the AI checks the repair.

🥈 2nd Place — Everything AI Abu Dhabi 2026

---

💡 The problem

Every neighbourhood has a hazard people walk past a hundred times: a playground spring rider with a snapped seat and sharp metal, an exposed cable, a pothole. Nobody reports it. The forms are long and English-first, the report disappears into a private queue, and nothing visibly happens.

✨ What HoTH does

HoTH turns one civic action into a neighbourhood moment. A report takes about a minute, by voice, in your own language. It then lives in public with your neighbours, who see it, confirm it, comment on it and celebrate the fix together.

📸 Live-camera-only trust	No gallery picker. Photos are taken live and stamped with GPS and time on the device. "Every other reporting app lets you upload a screenshot from Google. Ours physically can't."
🎙️ Voice-first, multilingual	Speak Khaleeji Arabic or English. An AI voice assistant asks at most two questions, reads the report back, and files a bilingual ticket.
🚨 Explained AI priority	Every ticket shows its level and why, e.g. "CRITICAL — child safety". Deterministic safety floors can only raise the model's call, so child-safety hazards are never ranked below CRITICAL.
🏘️ A social layer for the Hay	A neighbourhood feed and map, "I saw it too" confirmations, AI-moderated comments, and "Hazard near you" alerts within 1 km.
🛠️ Verified fixes, not closed tickets	The crew's repair photo is compared against the original by AI. Half-done jobs are rejected, with boxes drawn on what's still broken.
🏆 Reputation	Earn points for reports, confirmations and fixes: Neighbour → Guardian → Hero of the Hay.

📱 Screenshots

Feed	Live capture	AI analysis	Voice interview
Ticket	Crew queue	Fix rejected	Fixed ✨
Posted to the Hay	Hazard near you	Feed map

🔁 How it works

flowchart LR
  A[📸 Live capture<br/>GPS + time] --> B[👁️ Vision AI<br/>category, hazards,<br/>plausibility]
  B --> C[🎙️ Voice interview<br/>1–2 questions<br/>AR / EN]
  C --> D[🚨 Priority agent<br/>LLM + safety floors]
  D --> E[🎫 Ticket + feed<br/>faces & plates blurred]
  E --> F[🏘️ Neighbours<br/>confirm · comment · alerts]
  E --> G[🛠️ Crew queue<br/>ranked by priority]
  G --> H[🔍 AI fix check<br/>before vs after]
  H -->|accepted| I[✨ Fixed!<br/>points + level-up]
  H -->|partial| G

Loading

AI does four triage jobs, it isn't a chatbot:

1. Is it real? The vision model has to confirm the photo shows the issue. Photos of a screen or a print are caught.
2. Already reported? Reports in the same category within 75 m go to an AI same-issue judge, and matches merge into one stronger report.
3. How urgent? The LLM proposes a level, and deterministic floors can only raise it. The reasons are shown in English and Arabic.
4. What's missing? The voice interview asks only for what the photo didn't answer.

Danger first: for electrical or acute hazards the app skips the questions and shows "keep children away" with a Call 999 button.

🏗️ Architecture

Fully on-device, no backend. SwiftUI + SwiftData app, with all AI logic in a pure, tested Swift package.

flowchart LR
  subgraph iPhone["iPhone (SwiftUI + SwiftData)"]
    CAM[Live camera] --> BLUR[Apple Vision<br/>face + plate blur]
    CAM --> VA[VisionAnalyzer]
    VA --> PA[PriorityAgent]
    VOICE[Voice interview<br/>WebRTC] -->|tool calls| PA
    PA --> DB[(Reports, votes,<br/>comments, points)]
    FIX[FixVerifier] --> DB
    MOD[Comment moderation] --> DB
    KC[Keychain<br/>API key]
  end
  VA -->|strict JSON schema| V[OpenAI vision model]
  FIX --> V
  MOD --> T[OpenAI text model]
  KC -->|ephemeral secret| RT[OpenAI Realtime voice]
  VOICE <--> RT

Loading

Layer	Tech
App	SwiftUI, SwiftData, MapKit, Apple Vision (on-device face and licence-plate blur)
AI core	HoTHCore Swift package: OpenAIClient, VisionAnalyzer, PriorityAgent, Plausibility, Dedup, DuplicateJudge, FixVerifier, Rewards
Models	OpenAI vision (Responses API + strict JSON schemas), gpt-realtime voice over WebRTC, a small text model for the chat fallback and moderation, gpt-image for app art
Security	API key in the iOS Keychain; the voice session uses short-lived ephemeral client secrets
Eval	Python harness (tools/eval) that runs the app's exact prompts against labelled Commons and synthetic images
Pitch	Interactive React deck built with open-slide

🚀 Getting started

Requirements: macOS with Xcode 26, XcodeGen, and an OpenAI API key.

brew install xcodegen
make gen                    # generates ios/HoTH.xcodeproj from ios/project.yml
open ios/HoTH.xcodeproj

1. Set your signing team on the HoTH target.
2. Run it on an iPhone (live camera and voice) or in the simulator (Demo mode is on by default and uses a labelled sample set).
3. In the app, go to Settings → API key, paste your key, then tap Save to Keychain.

🧪 Tests & eval

cd ios/Packages/HoTHCore && swift test   # 75 unit tests on the AI logic
make test                                # 69 app unit tests + 6 UI tests (simulator)

OPENAI_API_KEY=... uv run --project tools/eval tools/eval/run_eval.py --subset smoke   # or --subset full

The tests cover the priority agent and its safety floors, plausibility checks, dedup, the fix check, schemas, the privacy blur, the realtime event parser, the alert policy, comment moderation, and end-to-end golden-path UI flows. The latest eval results are in tools/eval/results.md.

🎤 Pitch deck

cd pitch-deck && npm install && npm run dev   # → http://localhost:5173/s/hoth-pitch

Press F for fullscreen and P for presenter notes.

📂 Repository layout

ios/                 SwiftUI app, XcodeGen project spec, unit + UI tests
ios/Packages/HoTHCore  Pure Swift AI logic + 75 unit tests
tools/eval/          Python eval harness and labelled dataset
pitch-deck/          Interactive pitch deck (open-slide)
submission/          Screenshots, demo shot list, packaging script
HACKATHON_SUBMISSION.md  Full write-up: problem, AI design, evidence, business value, risks

🏅 Hackathon

HoTH was built by Team Fomy_Io for Everything AI Abu Dhabi 2026, where it won 🥈 2nd place. The original judge-facing write-up, with a claim → evidence index, pilot KPIs, model routing and risks, is in HACKATHON_SUBMISSION.md.

🙏 Credits

• App images: IMAGE_CREDITS.md
• Eval dataset (Wikimedia Commons + AI-generated): ATTRIBUTION.md
• Changes: CHANGELOG.md

Made with ❤️ in Abu Dhabi, for every Hay.