A late vehicle stalls its own tick, not everyone else's flight

README.md

@@ -183,6 +183,33 @@ docker run --rm -p 8642:8642 -p 9002-9202:9002-9202/udp \
   -v $(pwd)/build/city/tiles:/world:ro -e SKYSIM_TILES=/world skysim
 ```
 
+`docker-entrypoint.sh` turns `SKYSIM_*` variables into flags: `SKYSIM_API_BIND`,
+`SKYSIM_API_PORT`, `SKYSIM_VEHICLES`, `SKYSIM_TILES`, `SKYSIM_STREAM_RADIUS`,
+`SKYSIM_STREAM_MAX`, `SKYSIM_DT`, `SKYSIM_TIME_MODE`, `SKYSIM_SPAWN_HOME`, and
+`SKYSIM_EXTRA_ARGS` for anything not listed.
+
+**Set `SKYSIM_TIME_MODE=interactive` on a long-lived server.** The default is `strict`,
+which is what determinism and CI replays need — a barrier every tick, and an abort when
+a vehicle misses it. A server that vehicles join and leave wants the abort replaced by
+eviction, or one lagging aircraft takes the fleet down with it. Both modes barrier on
+every connected vehicle's frame; what interactive adds is wall-clock pacing and a way
+out, freezing and then despawning a vehicle that has gone quiet for `--straggler-timeout`
+(default 2 s). Match `SKYSIM_DT` to the scheduler rate the vehicles are launched with,
+too: in lockstep, physics faster than the autopilot's loop means most ticks miss their
+deadline, and while that no longer corrupts the flight it does cost wall-clock pace.
+
+**`SKYSIM_CAMERA_FPS` is the camera switch.** Without it no frames are rendered and
+the camera endpoints serve nothing, whatever else is set. With it, `SKYSIM_CAMERA_SIZE`
+(default `256x144`), `SKYSIM_CAMERA_QUALITY`, `SKYSIM_CAMERA_THREADS`,
+`SKYSIM_CAMERA_FOV`, `SKYSIM_CAMERA_PITCH` and `SKYSIM_CAMERA_RANGE` apply:
+
+```bash
+docker run --rm -p 8642:8642 -p 9002-9202:9002-9202/udp \
+  -v $(pwd)/build/city/tiles:/world:ro -e SKYSIM_TILES=/world \
+  -e SKYSIM_CAMERA_FPS=10 -e SKYSIM_CAMERA_SIZE=256x144 skysim
+# http://localhost:8642/instances/0/camera.mjpg
+```
+
 ---
 
 ## Control plane (`--api-port`)
@@ -191,7 +218,7 @@ docker run --rm -p 8642:8642 -p 9002-9202:9002-9202/udp \
 |----------|--------|
 | `POST /vehicles` `{"instance":N, "launch_process":true}` (both fields optional) | reserve a requested or next-free ArduPilot instance (optionally forks arducopter) → `{id, instance, json_port, mavlink_tcp}` |
 | `DELETE /vehicles/{id}` | despawn, release instance, kill managed process |
-| `GET /vehicles` | per-vehicle: connected, frozen, held_ticks, pos_ned, `midair_collisions`, `building_contacts` |
+| `GET /vehicles` | per-vehicle: connected, frozen, silent_ticks, pos_ned, `midair_collisions`, `building_contacts` |
 | `GET /metrics` | tick p50/p99 µs, straggler_events, freezes, resident_tiles |
 | `POST /mission/check` `{"waypoints_ned":[[n,e,d],...], "clearance_m":1.0}` | sweep a planned route through loaded building geometry → `{checked, clear, hits}` |
 
@@ -206,6 +233,7 @@ the dashboard.
 | `--truth-log out.csv` | Ground truth per tick — judge the physics with this, not the EKF |
 | `--record-servo` / `--replay-servo` | Deterministic input tapes for reproducible runs |
 | `--time-mode strict` | Lockstep barrier on every vehicle; aborts on a straggler (CI) |
+| `--straggler-timeout <seconds>` | interactive: silence before a vehicle is frozen out of the barrier (2 s) |
 | `--stream-radius` / `--stream-max` | City tile residency around each vehicle |
 | `--physics-threads` | Opt a single huge world into Jolt's thread pool (only worth it past ~1000 bodies) |
 | `--io-threads` | Fan the reply path across cores (helps past ~48 vehicles) |

docker-entrypoint.sh

@@ -8,6 +8,16 @@ set -euo pipefail
 
 args=(--api-bind "${SKYSIM_API_BIND:-0.0.0.0}" --api-port "${SKYSIM_API_PORT:-8642}")
 
+# One mapping, used by everything below: set the variable, get the flag; leave it
+# unset and the binary's own default stands, so a default lives in one place
+# rather than being restated here.
+add_if_set() { # env-var-name flag
+    local value="${!1:-}"
+    if [[ -n "$value" ]]; then
+        args+=("$2" "$value")
+    fi
+}
+
 # Vehicles are normally spawned on demand over the control plane, so default to
 # starting with none rather than skysim's built-in default of one.
 args+=(--vehicles "${SKYSIM_VEHICLES:-0}")
@@ -24,12 +34,31 @@ if [[ -n "${SKYSIM_TILES:-}" ]]; then
     fi
 fi
 
-if [[ -n "${SKYSIM_DT:-}" ]]; then
-    args+=(--dt "${SKYSIM_DT}")
-fi
+add_if_set SKYSIM_DT --dt
+
+# Time mode, which a deployment could not set at all before.
+#
+# skysim defaults to strict because that is what determinism and CI replays need:
+# a barrier every tick, and an abort when a vehicle misses it. A long-lived server
+# wants the opposite — vehicles join and leave, some of them lag, and none of that
+# should take the fleet down. Without this the only way to say so was
+# SKYSIM_EXTRA_ARGS, so every deployment quietly ran strict.
+add_if_set SKYSIM_TIME_MODE --time-mode
+
+add_if_set SKYSIM_SPAWN_HOME --spawn-home
 
-if [[ -n "${SKYSIM_SPAWN_HOME:-}" ]]; then
-    args+=(--spawn-home "${SKYSIM_SPAWN_HOME}")
+# Camera. SKYSIM_CAMERA_FPS is the switch — without --camera-fps the render
+# service is never built and every other camera setting is inert, which is
+# exactly how a deployment ends up serving a control plane with no pictures on
+# it. The rest only apply once it is on.
+if [[ -n "${SKYSIM_CAMERA_FPS:-}" ]]; then
+    args+=(--camera-fps "${SKYSIM_CAMERA_FPS}")
+    add_if_set SKYSIM_CAMERA_SIZE --camera-size
+    add_if_set SKYSIM_CAMERA_QUALITY --camera-quality
+    add_if_set SKYSIM_CAMERA_THREADS --camera-threads
+    add_if_set SKYSIM_CAMERA_FOV --camera-fov
+    add_if_set SKYSIM_CAMERA_PITCH --camera-pitch
+    add_if_set SKYSIM_CAMERA_RANGE --camera-range
 fi
 
 if [[ -n "${SKYSIM_EXTRA_ARGS:-}" ]]; then

docs/DESIGN.md

@@ -33,12 +33,22 @@ run as separate `skysim` processes (sharding) for near-linear scaling.
 
 World advances at fixed `dt` (default 1/400 s; per-deployment). Modes:
 
-- **strict** (CI, replays): barrier — tick when *every* registered vehicle's next input frame
-  has arrived. Deterministic; one stalled SITL stalls the world. Timeout ⇒ abort with report.
-- **interactive** (SkyHub operators in the loop): tick on schedule. A vehicle that missed the
-  deadline gets its last PWM held for up to `k` ticks (default 3); beyond that it is **frozen**
+- **strict** (CI, replays): barrier — tick when the next input frame of every *connected*
+  vehicle has arrived. Deterministic; one stalled SITL stalls the world. Timeout ⇒ abort with
+  report. The timeout starts once something has connected: a world still waiting for its first
+  SITL is not stalled, but a fleet that has all gone silent is, and aborts rather than hanging.
+- **interactive** (SkyHub operators in the loop): the same barrier, paced to the wall clock,
+  with an exit from it instead of an abort. A vehicle that missed the deadline stalls its own
+  tick; once it has been silent for `--straggler-timeout` (default 2 s) it is **frozen**
   (kinematic hold, flagged in the API) until frames resume, and auto-despawned after a grace
-  period. Replies are sent immediately after the tick that consumed each vehicle's input.
+  period. Freezing takes it out of the barrier entirely — the fleet stops both blocking on it
+  and waiting out its share of the frame grace — while it stays polled so it can thaw.
+  Replies are sent immediately after the tick that consumed each vehicle's input.
+
+  A late vehicle is never stepped on its previous tick's PWM. ArduPilot derives its scheduler
+  rate from the timestamp we return (`adjust_frame_time(1.0 / deltat)` in SIM_JSON.cpp), so a
+  step it did not contribute to re-rates its control loop and hands it motion it never
+  commanded — it crashed every takeoff. Lateness may cost wall-clock pace, never fidelity.
 
 Spawn/despawn only ever happens at a tick boundary (step 1), so mid-step body creation never occurs.
 

src/api/control_server.cpp

@@ -21,6 +21,30 @@ namespace {
 // plane, which must stay answerable while they run.
 constexpr size_t kWorkerThreads = 64;
 
+// How often an MJPEG stream looks for a newly drawn frame. Short relative to any render rate
+// worth watching, because it bounds how long a finished frame waits before it goes out; it does
+// not set the rate, since a frame is only written once (see write_mjpeg_part callers).
+constexpr auto kFramePoll = std::chrono::milliseconds(5);
+
+// One multipart part: headers plus the JPEG.
+//
+// Only ever called for a frame the caller has not sent before. Re-sending whatever is in the
+// slot on a fixed cadence — which is what this did — puts duplicates on the wire at the poll
+// rate rather than the render rate: measured at 24.7 frames/s carrying 10 frames/s of content,
+// 59% repeats and 37 KB/s of them. Worse than the waste, it feeds a consumer faster than the
+// content arrives, and the excess lands in socket and queue buffers as latency the viewer sees
+// as lag. FrameStore drops rather than queues for exactly this reason; this route was undoing it.
+void write_mjpeg_part(httplib::DataSink &sink, const std::vector<uint8_t> &jpeg) {
+    char head[128];
+    const int n = std::snprintf(head, sizeof(head),
+                                "--skysimframe\r\nContent-Type: image/jpeg\r\n"
+                                "Content-Length: %zu\r\n\r\n",
+                                jpeg.size());
+    sink.write(head, static_cast<size_t>(n));
+    sink.write(reinterpret_cast<const char *>(jpeg.data()), jpeg.size());
+    sink.write("\r\n", 2);
+}
+
 // {"launch_process":true} — absent key means false.
 bool parse_launch_process(const std::string &body) {
     const char *p = std::strstr(body.c_str(), "\"launch_process\"");
@@ -138,11 +162,11 @@ std::string vehicle_json(const VehicleInfo &v) {
     char buf[512];
     std::snprintf(buf, sizeof(buf),
                   "{\"id\":%u,\"instance\":%d,\"json_port\":%d,\"mavlink_tcp\":%d,"
-                  "\"connected\":%s,\"frozen\":%s,\"held_ticks\":%llu,"
+                  "\"connected\":%s,\"frozen\":%s,\"silent_ticks\":%llu,"
                   "\"midair_collisions\":%llu,\"static_contacts\":%llu,"
                   "\"building_contacts\":%llu,\"pos_ned\":[%.3f,%.3f,%.3f]}",
                   v.id, v.instance, v.json_port, v.mavlink_tcp_port, v.connected ? "true" : "false",
-                  v.frozen ? "true" : "false", static_cast<unsigned long long>(v.held_ticks),
+                  v.frozen ? "true" : "false", static_cast<unsigned long long>(v.silent_ticks),
                   static_cast<unsigned long long>(v.midair_collisions),
                   static_cast<unsigned long long>(v.static_contacts),
                   static_cast<unsigned long long>(v.building_contacts), v.pos_ned[0], v.pos_ned[1],
@@ -305,7 +329,7 @@ ControlServer::ControlServer(const std::string &bind_addr, int port, CommandQueu
             res.set_content("{\"error\":\"no such instance\"}", "application/json");
             return;
         }
-        std::vector<uint8_t> jpeg = snapshots.camera_frame(id);
+        std::vector<uint8_t> jpeg = snapshots.camera_frame(id).jpeg;
         if (jpeg.empty()) {
             res.status = 404;
             res.set_content("{\"error\":\"no frame\"}", "application/json");
@@ -324,22 +348,17 @@ ControlServer::ControlServer(const std::string &bind_addr, int port, CommandQueu
         }
         res.set_chunked_content_provider(
             "multipart/x-mixed-replace; boundary=skysimframe",
-            [snapshots, resolve_instance, instance](size_t, httplib::DataSink &sink) {
+            [snapshots, resolve_instance, instance, last_sent = -1.0](size_t,
+                                                                      httplib::DataSink &sink) mutable {
                 // Re-resolved per frame, so a skysim restart mid-recording picks
                 // the vehicle back up instead of streaming nothing forever.
                 const uint32_t id = resolve_instance(instance);
-                std::vector<uint8_t> jpeg = id != 0 ? snapshots.camera_frame(id) : std::vector<uint8_t>{};
-                if (!jpeg.empty()) {
-                    char head[128];
-                    const int n = std::snprintf(head, sizeof(head),
-                                                "--skysimframe\r\nContent-Type: image/jpeg\r\n"
-                                                "Content-Length: %zu\r\n\r\n",
-                                                jpeg.size());
-                    sink.write(head, static_cast<size_t>(n));
-                    sink.write(reinterpret_cast<const char *>(jpeg.data()), jpeg.size());
-                    sink.write("\r\n", 2);
+                auto frame = id != 0 ? snapshots.camera_frame(id) : skysim::render::FrameStore::Frame{};
+                if (!frame.jpeg.empty() && frame.sim_time_s != last_sent) {
+                    last_sent = frame.sim_time_s;
+                    write_mjpeg_part(sink, frame.jpeg);
                 }
-                std::this_thread::sleep_for(std::chrono::milliseconds(40));
+                std::this_thread::sleep_for(kFramePoll);
                 return true;
             });
     });
@@ -354,8 +373,9 @@ ControlServer::ControlServer(const std::string &bind_addr, int port, CommandQueu
         // strtoul, not stoul: the route regex accepts any number of digits, and
         // stoul throws on a value too big for the type. Saturating is the same
         // answer as "no such vehicle" without the exception.
-        std::vector<uint8_t> jpeg = snapshots.camera_frame(
-            static_cast<uint32_t>(std::strtoul(req.matches[1].str().c_str(), nullptr, 10)));
+        std::vector<uint8_t> jpeg =
+            snapshots.camera_frame(static_cast<uint32_t>(std::strtoul(req.matches[1].str().c_str(), nullptr, 10)))
+                .jpeg;
         if (jpeg.empty()) {
             res.status = 404;
             res.set_content("{\"error\":\"no frame\"}", "application/json");
@@ -378,21 +398,13 @@ ControlServer::ControlServer(const std::string &bind_addr, int port, CommandQueu
         const auto id = static_cast<uint32_t>(std::strtoul(req.matches[1].str().c_str(), nullptr, 10));
         res.set_chunked_content_provider(
             "multipart/x-mixed-replace; boundary=skysimframe",
-            [snapshots, id](size_t /*offset*/, httplib::DataSink &sink) {
-                std::vector<uint8_t> jpeg = snapshots.camera_frame(id);
-                if (!jpeg.empty()) {
-                    char head[128];
-                    const int n = std::snprintf(head, sizeof(head),
-                                                "--skysimframe\r\nContent-Type: image/jpeg\r\n"
-                                                "Content-Length: %zu\r\n\r\n",
-                                                jpeg.size());
-                    sink.write(head, static_cast<size_t>(n));
-                    sink.write(reinterpret_cast<const char *>(jpeg.data()), jpeg.size());
-                    sink.write("\r\n", 2);
+            [snapshots, id, last_sent = -1.0](size_t /*offset*/, httplib::DataSink &sink) mutable {
+                auto frame = snapshots.camera_frame(id);
+                if (!frame.jpeg.empty() && frame.sim_time_s != last_sent) {
+                    last_sent = frame.sim_time_s;
+                    write_mjpeg_part(sink, frame.jpeg);
                 }
-                // Poll a little faster than frames are produced, so the stream
-                // tracks the render rate instead of setting its own.
-                std::this_thread::sleep_for(std::chrono::milliseconds(40));
+                std::this_thread::sleep_for(kFramePoll);
                 return true;
             });
     });

src/api/control_server.h

@@ -12,6 +12,7 @@
 #include <vector>
 
 #include "core/world.h"
+#include "render/frame_store.h"
 #include "vehicle/manager.h"
 
 namespace skysim::api {
@@ -23,7 +24,7 @@ struct VehicleInfo {
     int mavlink_tcp_port = 0;
     bool connected = false; // has ever delivered a servo frame
     bool frozen = false;    // straggler policy engaged (kinematic hold)
-    uint64_t held_ticks = 0;
+    uint64_t silent_ticks = 0;
     uint64_t midair_collisions = 0;  // vehicle-vehicle contact events (crash telemetry)
     uint64_t static_contacts = 0;    // any static touch, landings included
     uint64_t building_contacts = 0;  // building-tile strikes only — the "hit a building" signal
@@ -103,10 +104,11 @@ class ControlServer {
     struct Snapshots {
         std::function<std::vector<VehicleInfo>()> vehicles;
         std::function<MetricsInfo()> metrics;
-        // Latest camera frame for a vehicle as encoded JPEG; empty when the
-        // camera is off or that vehicle has not been rendered yet. Published by
-        // the tick thread like everything else here.
-        std::function<std::vector<uint8_t>(uint32_t)> camera_frame;
+        // Latest camera frame for a vehicle as encoded JPEG, with the sim time it was drawn
+        // at; empty when the camera is off or that vehicle has not been rendered yet. The
+        // timestamp is what lets the MJPEG routes send a frame once instead of re-sending
+        // whatever is in the slot. Published by the tick thread like everything else here.
+        std::function<skysim::render::FrameStore::Frame(uint32_t)> camera_frame;
     };
 
     // Binds bind_addr:port and serves on its own thread. Throws on bind failure (startup

src/core/clock.h

@@ -20,7 +20,11 @@ class PhysicsClock {
     uint64_t tick_ = 0;
 };
 
-// TODO(M1/M4): TickGate — strict: barrier on all vehicles' next frame (with abort timeout);
-// interactive: deadline pacing + straggler policy (hold-k, freeze, grace despawn).
+// TODO(M1/M4): TickGate — one barrier on the next frame of every *connected* vehicle, with
+// the straggler policy injected: strict aborts on timeout, interactive adds deadline pacing
+// and evicts (freeze, grace despawn). The barrier population is not the whole fleet: a slot
+// that has never connected has nothing to wait for, and interactive freezing takes a vehicle
+// out of the barrier rather than stepping it. A late vehicle that is still in the barrier
+// must never be stepped without its frame in either mode — see run_interactive for why.
 
 }  // namespace skysim::core