Agentic mission control for autonomous drone fleets. A multi-agent system that plans inspection flights over industrial sites, reads aerial imagery, answers operational-policy questions from a retrieval corpus, and commands physical aircraft — with a deterministic safety gate and a human authorisation step standing between the model and anything that moves.
Full stack: Python / FastAPI orchestration backend, Next.js / TypeScript operator console, streamed over SSE, traced end to end.
┌─────────────────────────────────────────────────────────────────────────┐
│ Next.js console · live agent trace · approval dialog · fleet map │
└───────────────────────────────┬─────────────────────────────────────────┘
│ SSE + WebSocket
┌───────────────────────────────▼─────────────────────────────────────────┐
│ FastAPI · orchestration graph │
│ │
│ recall ─→ route ─→ [ specialist ]* ─→ synthesize ─→ persist │
│ │ │
│ supervisor mission_planner · knowledge │
│ (routing only) perception · analytics │
│ │ │
│ ┌───────▼────────┐ │
│ │ SAFETY GATE │ deterministic, not a │
│ │ │ prompt │
│ └───────┬────────┘ │
│ allow ──────────┼────────── block │
│ │ │
│ require_approval │
│ │ │
│ ┌───────▼────────┐ │
│ │ HUMAN operator │ fails closed on timeout │
│ └───────┬────────┘ │
│ ▼ │
│ tools: fleet · airspace · weather · RAG · vision · sandboxed Python │
│ memory: episodic + working + semantic (SQLite) │
│ tracing: span tree · token/cost accounting · LangSmith projection │
└─────────────────────────────────────────────────────────────────────────┘
Every screenshot below is from the system running with no API key — the
deterministic provider drives it, so npm run screenshots reproduces these
exactly on any machine.
The planner has drafted a mission and assigned an aircraft. It cannot launch. The operator sees the exact call, the reasoning that produced it, and a countdown that denies the request if it expires.
Gusts at the refinery exceed the FB-Ranger-H6's envelope. The gate blocks it
rather than asking a human to approve something unsafe — note that no
authorisation was ever raised. The trace on the right shows the tool marked
CRITICAL then REFUSED, with the specific rule that fired.
Every graph node, agent, guardrail check, tool call and model call, positioned
by real start time. In a run that paused for an operator, the dominant span is
hitl.await_approval — the human, not the model.
Each passage shows its dense rank, its BM25 rank and the fused score, so a bad answer can be traced to a bad retrieval rather than blamed on the model.
Fleet view and a completed run
No API key is required. With no credential configured the system runs a deterministic simulated model provider — the graph, the tools, the safety gate and the approval queue are all the real implementations, and only token generation is stubbed.
docker compose up --buildConsole at http://localhost:3000, API at http://localhost:8000/docs.
To route the same graph through a real model, put a key in backend/.env
(already gitignored) and verify the live path before running anything larger.
Two options — Gemini's free tier (no credit card) or Claude:
# backend/.env — free option
AEROMIND_LLM_API_KEY=...
AEROMIND_LLM_BASE_URL=https://generativelanguage.googleapis.com/v1beta/openai/
AEROMIND_LLM_MODEL=gemini-3.5-flash-lite
# or
AEROMIND_ANTHROPIC_API_KEY=sk-ant-...
One OpenAICompatibleProvider covers Gemini, Groq, GitHub Models, OpenRouter
and Cerebras, so switching platform is two environment variables rather than a
new client. Then:
python -m scripts.verify_live --missionThat makes one cheap call to confirm auth and model access, then optionally runs
one full multi-agent turn, reporting tokens and dollars for both. A measured
first call beats discovering the cost afterwards — a full mission turn is
roughly $0.23 on claude-opus-5, $0.05 on claude-haiku-4-5, and $0 on
Gemini's free tier.
Running without Docker
cd backend
python -m venv .venv && .venv/Scripts/activate # Unix: source .venv/bin/activate
pip install -r requirements.txt
python -m scripts.generate_imagery
uvicorn app.main:app --reloadcd frontend
npm install
npm run devTwo services: the API on Fly.io, the console on Vercel.
# API
cd backend
fly launch --no-deploy --copy-config
fly secrets set AEROMIND_LLM_API_KEY=...
fly deploy
# Console
cd ../frontend
vercel --prod
# set NEXT_PUBLIC_AEROMIND_API=https://<your-app>.fly.dev in project settings
# and add that origin to AEROMIND_CORS_ORIGINS on the APIFly rather than a serverless host because this service holds connections open: the console streams the agent trace over SSE for the length of a turn, and a turn parked on a human approval produces no bytes at all for up to 90 seconds. That is exactly the traffic request/response platforms cut.
For the same reason the browser calls the API directly in production instead of going through Vercel's rewrite — every extra proxy hop is somewhere SSE events can sit and buffer. That is not theoretical: during development Next's default gzip did precisely this, and the approval dialog never appeared because the events were stuck in the compressor.
The deployed instance defaults to the simulator, on purpose. A public URL plus a free-tier key is a quota anyone can drain in two clicks. Live mode is a switch the owner flips:
fly secrets set AEROMIND_FORCE_SIMULATED=false # demo
fly secrets set AEROMIND_FORCE_SIMULATED=true # back to safeWhat the free tier actually allows (measured, not quoted)
Running this against Gemini's free tier turned up limits that are not published on the docs page — they are only visible in your own AI Studio dashboard:
| Limit | Value | Consequence |
|---|---|---|
| Requests per minute | 5 | One mission turn (~10 model calls) must pace itself |
| Requests per day | 20 on gemini-3.8-flash |
~2 mission runs per day |
| Transient 503s | frequent | "high demand" errors need their own retry path |
The daily cap is the binding one, and it is why the deployed demo runs the
simulator by default. gemini-3.5-flash-lite has a separate, larger budget and
is the configured default.
Three provider-specific things had to be built to make the free path work at
all, all of them in app/llm/client.py:
- A sliding-window pacer. Reacting to 429s is not enough at 5 rpm — a ten-call turn would spend most of its life in backoff. Requests are paced up front so a run is merely slow rather than a sawtooth of rejections.
- Two retry strategies, not one. A 429 carries the server's own
retryDelay, so the right move is to wait exactly that long. A 503 is momentary oversubscription, where short exponential backoff clears it. Disabling the SDK's blind retry to get the first right made the second mandatory. - Opaque provider metadata round-tripping. Gemini attaches an encrypted
thought_signatureto every function call and rejects the next request if it is not echoed back, in a non-standard field (tool_calls[0].extra_content.google.thought_signature).ToolCallcarries it through the agent loop as a blob the loop never inspects.
| In the console | What it exercises |
|---|---|
| "Launch a grid survey over the solar farm to check for thermal defects" | plan → assign aircraft → stops for a human → launches on approval |
| "Dispatch a drone at the refinery right now for a perimeter patrol" | gate refuses outright — gusts exceed the airframe limit |
| "What is the minimum visibility required to fly, and who decides?" | grounded retrieval with inline citations |
| "Analyse the latest thermal capture from the solar site" | vision model → structured detections → severity classification |
| "Rank the fleet by battery health" | telemetry → sandboxed Python, not prose arithmetic |
Most of what is interesting here is not that agents call tools. It is what sits between the model deciding to act and the action happening.
A guardrail written into a system prompt is a guardrail the model can be argued
out of. Every rule in app/graph/safety.py is
deterministic Python that reads live fleet state and returns a verdict the model
never sees until it is already decided.
Tools carry a risk tier. SAFE reads run freely. SENSITIVE writes are
reversible and logged. CRITICAL commands move aircraft and cannot execute
without a recorded human decision — regardless of how confident the model is.
Three properties fall out of that design, and each has a test:
- Blocking beats escalating. A launch that violates a pre-flight condition is refused outright, never offered to an operator. Escalating an unsafe command launders a refusal into a decision someone can get wrong under time pressure.
- Timeouts fail closed. An unanswered authorisation resolves to denied. Timing out into the permissive state is the single most common way a human-in-the-loop control is defeated in practice.
- Autonomy level cannot override physics. Raising autonomy to
autonomousremoves the approval step, not the pre-flight checks.
The perception agent reads untrusted image-derived text. It cannot see the
launch_mission schema, and the check is enforced at the call site rather than
trusted to the tool list in the prompt — a prompt-injected tool name would
otherwise sail straight through. Retrieved documents are also screened for
instruction-shaped text and downgraded to quoted evidence.
Hybrid BM25 + dense retrieval over a synthetic operations corpus, fused by weighted reciprocal rank, with small-to-big indexing: narrow units are embedded and scored, whole sections are returned to the generator.
$ python -m scripts.eval_retrieval --ablate
corpus: 76 indexed units over 24 labelled queries
dense only recall@1=0.792 recall@3=0.792 recall@5=0.833 mrr=0.802
lexical only (BM25) recall@1=0.958 recall@3=0.958 recall@5=1.000 mrr=0.969
hybrid (RRF) recall@1=0.958 recall@3=0.958 recall@5=1.000 mrr=0.967
Three findings from building that harness, all of which changed the code:
- Chunking was the bottleneck, not scoring. A six-item pre-flight checklist indexed as one blob diluted "visibility" across a passage six times longer than the rule itself, and no amount of scoring tuning recovered it. Splitting list items into their own retrieval units while returning the parent section moved recall@1 from 0.875 to 0.958 and MRR from 0.917 to 0.967, holding everything else constant.
- The canonical RRF constant of 60 is wrong for a small corpus. Tuned for TREC-scale collections, it flattens the score spread to nothing across a few dozen chunks. The constant now scales with corpus size.
- The hybrid does not beat BM25 here, and the ablation says so. A hashed TF-IDF projection is a lossy view of the same lexical signal, not an orthogonal one, so fusing them averages a strong ranker with a weaker copy of itself. The dense weight is set to 0.15 — enough to act as a recall backstop, not enough to overrule a confident lexical match. The eval queries also share the corpus's vocabulary, which flatters lexical retrieval; swapping in a sentence-transformer is a one-class change and the weight should be re-swept when that happens.
Reproduce any of it with --ablate, --sweep, --verbose.
An agent run is not a request, it is a call tree — supervisor → specialist →
tool → model. Flat logs cannot express that, so every unit of work opens a span
with a parent pointer and its own token and cost accounting. The console renders
the tree as a flame timeline; /api/traces/{id}/langsmith projects the same
trace into LangSmith's run-ingestion shape.
The payoff is immediate in practice: in a run that paused for an operator, the
dominant span is hitl.await_approval at 15.6 s against ~40 ms of model time.
Knowing whether latency is the model or the human is the difference between
tuning a prompt and re-designing an interface.
An agentic system that only runs with a funded API key is one nobody can exercise in CI, demo offline, or regression-test per commit. The simulated provider is deterministic and role-aware, so the 131-test suite asserts on routing decisions and tool sequences rather than model prose — including the full launch path: proposal → gate → approval → hardware state change.
It also earns its keep as a bug-finder. It caught the code sandbox rejecting
sorted(key=lambda …), which is the shape nearly every ranking snippet takes —
an over-restriction that bought no safety and would have quietly degraded every
analytics answer in production.
Episodic (the full transcript), working (rolling summary + last N turns, bounded by construction), and semantic (durable facts extracted from turns and retrievable across sessions). Compaction runs on the turns falling out of the working window rather than the whole transcript, so summarisation cost stays flat as a session grows.
$ pytest -q
131 passed
Concentrated where failure is expensive:
test_safety_gate.py (18 cases — battery
floors, wind envelopes, grounded packs, teardown intervals, airspace conflicts,
prompt-injection screening) and
test_hitl.py (approval, denial, timeout-fails-closed,
decisions cannot be overwritten, a dead websocket cannot block a decision).
A simulator that runs everywhere is also how a provider integration quietly
rots — nothing exercises it, so nothing tells you when it breaks.
test_anthropic_provider.py drives
AnthropicProvider through the real anthropic SDK with a mock HTTP transport
underneath. The SDK builds and validates the actual request, so the tests assert
on the wire format — adaptive thinking, output_config.effort nested rather
than top-level, cache_control on the system block, strict tool schemas,
base64 image ordering, streaming above the timeout threshold — and on response
parsing: tool_use blocks, thinking blocks kept out of the answer, cache-read
token accounting, typed errors on 400 and 429.
That caught a real defect. The project was pinned to anthropic==0.69.0, which
predates output_config entirely, so every live model call would have raised
TypeError on a parameter the code passes unconditionally. Now pinned to 1.2.0
and covered.
What this does not claim: the tests prove the SDK accepts the request and
the parser handles the documented response shape. They do not prove the live API
accepts it — the system has only ever been run end to end on the deterministic
provider. Set ANTHROPIC_API_KEY to exercise the real path.
Relatedly, get_llm() now raises ProviderUnavailable when a key is configured
but the provider will not construct. It used to fall back to the simulator,
which is the worst available behaviour: an operator who had configured a real
model would receive canned reasoning about live aircraft with no indication
anything was wrong.
- The code sandbox is not a security boundary. It is a restricted-namespace interpreter with a static AST screen and a wall-clock limit. It blocks the obvious escapes and would not stop a determined adversary. Production runs this in a network-less container; the interface is identical so the swap is a deployment change.
- The fleet is simulated.
FleetServiceis a real state machine — launching actually flips status and decrements battery — standing in for a vendor SDK. - The corpus is synthetic. The SOPs and type records were written for this project. They are not real regulatory instruments and are labelled as such in every document.
- The dense embedder is a hashed TF-IDF projection, chosen so the system runs offline in CI. Its measured contribution on this corpus is near zero (above).
- Traces live in memory with a bounded ring buffer plus disk export. A multi-instance deployment needs a real trace backend.
backend/
app/
agents/ supervisor, synthesiser, four specialists, shared loop
graph/ orchestration state machine, shared state, safety gate
llm/ provider abstraction: Anthropic + deterministic simulator
rag/ chunking, hybrid store, embeddings, retrieval pipeline
memory/ episodic / working / semantic memory over SQLite
tools/ registry with risk tiers; fleet, airspace, RAG, vision, sandbox
hitl/ approval queue
observability/ span tracing, cost accounting, LangSmith projection
scripts/ eval_retrieval.py, generate_imagery.py, verify_live.py
tests/ 131 tests
frontend/
src/app/ console, fleet, traces, knowledge
src/components/ agent stream, approval dialog, span tree, site map
scripts/ capture_screenshots.mjs (regenerates the images above)
| Endpoint | Purpose |
|---|---|
POST /api/missions/run |
Run a turn, return the finished state |
GET /api/missions/stream |
Same turn as an SSE event stream |
GET /api/approvals · POST /api/approvals/{id}/decide |
Authorisation queue |
WS /ws/events |
Live approval events |
GET /api/fleet · /api/fleet/missions |
Fleet and mission state |
POST /api/knowledge/search |
Hybrid retrieval, optional LLM rerank |
GET /api/traces · /api/traces/{id} · /api/traces/{id}/langsmith |
Observability |
GET /api/memory/facts · /api/memory/events |
Long-term memory |
Built by Poreddy Narendra Reddy.