Take the import census ahead of the engine, and stop re-walking a finished ladder (issue 105)

CLAUDE.md

@@ -287,9 +287,11 @@ platform types it touches, and walks it over the shapes that set has actually
 produced: a rung that hangs, a rung a previous launch never came back from (once,
 which is asked again, and twice, which is a refusal), a ledger left by another
 build, a ledger that never opens, a page loaded before the walk, an install whose
-ledger says the engine already failed there (the walk goes ahead of the engine), and
-a clean run twice over. It needs a C compiler for the stand-in library and nothing
-else.
+ledger says the engine already failed there (the walk goes ahead of the engine), a
+clean run twice over (the second walk finds the verdict on the books and asks
+nothing — issue #105's set ended two launches on the first rung of a walk behind a
+verdict), and a walk started behind the early one on the same launch. It needs a C
+compiler for the stand-in library and nothing else.
 
 The ladder's only real property is that it converges — whatever one rung does, the
 rungs and locations behind it still get asked. Two builds have been spent finding out
@@ -312,9 +314,11 @@ builds a library in this box's own architecture that needs `libm`, `libc` and a
 `libblocked.so` made unopenable by permission, and holds the census to naming
 exactly the blocked library's two symbols (a function and a data object), the
 refusal with the loader's words, nothing that another library provides, and no
-weak import. The committed ARM `libovprobe.so` exercises the ELF32 path, a FIFO
-holds it to a recorded miss rather than a hang, and a second call must change
-nothing. Needs the .NET 6 SDK under `~/.dotnet-local` and a C compiler.
+weak import — taken inline and again on a thread of its own with the caller
+waiting, which is how `OnCreate` runs it ahead of the engine. The committed ARM
+`libovprobe.so` exercises the ELF32 path, a FIFO holds it to a recorded miss rather
+than a hang, and a second call must change nothing. Needs the .NET 6 SDK under
+`~/.dotnet-local` and a C compiler.
 
 ### The trail harness
 

docs/INTERNALS.md

@@ -1211,10 +1211,71 @@ reversal later:
   never arrives — the set parks the census like the Q80 parked its calls; the
   previous-run block says which step.
 
-**State of #105 (2026-10-02):** the ladder's answer is in and is the good one; the
-census shipped in `build-283f3e9`, and the reply asking the reporter to run it quotes
-that tag. Nothing here changes the Q80's close:
-that set refused the mapping the AU7200 allows.
+### The census has to go ahead of the engine too
+
+The `build-283f3e9` report (2026-10-02, the same afternoon) has no `engine imports:`
+line on it, and the trail says why. Its previous-run block is the build's first
+launch: the early walk reached its verdict in 248 ms, the engine failed line for line
+as before, and then the post-failure thread **started the walk again** — the ledger's
+answers replayed, `probe: copy libovprobe.so to data/`, `probe: open res/` — and that
+is the launch's last line. No `read header` behind it, no deadline miss five seconds
+later, no `still on the failure screen — +10s` from the heartbeat. The launch was over
+within five seconds of `ENGINE FAILURE`: the Q80's shape (*The ladder has to start*,
+above), on a set that otherwise answers every call we make. The census was queued
+behind that second walk and never reached.
+
+The `this run` block — the second launch, 23 s later — is three lines with no UI:
+`Main entered`, the socket, the engine preload. The `build-3996334` page had the same
+three lines at the same place, fetched minutes after launch both times, so this is not
+a quick reporter: **on this set the second launch of a build stops before or inside
+`Elementary.Initialize()`** and the only page it ever serves is the no-provider page.
+That page had no `build :` line, though the reply had promised one, and nothing of the
+census.
+
+Four things follow and are in the code:
+
+- **A walk behind a verdict asks nothing.** `NativeProbe.Walk` reads the verdict
+  after `Ledger.Open` and returns on it: `native probe: verdict already on the books
+  — …; not walking again`. The repeatable rungs (`open`, `header`, `read`, the copy
+  into `data/`) were re-made on every walk so the rungs behind them had a descriptor;
+  behind a verdict there are no rungs, and on this set the first of them was where
+  the launch ended with the census behind it. `tools/probeladder`'s `resume`, `peek`,
+  `early` and `notearly` hold it; `peek` used to assert the opposite.
+- **The census goes ahead of the engine, on the walk's gate.**
+  `NativeProbe.EngineFailedHere()` is true whenever a ledger exists, which it only
+  does on an install whose engine has failed. `OnCreate` then runs
+  `EngineImports.RunInBackground` after the walk and waits up to 60 s for it
+  (`engine imports: the engine waits for the census (up to 60 s)` … `census finished
+  after N ms — on to the engine`), the way it waits for the walk; every dlopen in it
+  is under its own deadline, and what the bound cuts off still reaches the trail from
+  the census's thread. The lines go to the trail as they are established, because
+  the page that shows them is the *next* launch's. The post-failure call stays for
+  the first failure on an install and finds the census taken on every launch after.
+  `tools/engineimports`'s `background` scenario drives this shape, and found the
+  claim race in the first version of it.
+- **The no-provider page carries `build :`, `engine imports:` and the stub block**,
+  because on this set every page is that page.
+- **`Program.Main` drops `Elementary: initializing` before the call**, so the second
+  launch's three lines become four and name the call they stop in front of.
+
+How to read the next report, in addition to the four readings above: the
+previous-run block should carry, between `OnCreate: UI built` and `EFL subsystems`,
+the walk's verdict (early or already on the books), then `engine imports: the engine
+failed on an earlier launch here — taking the census now, ahead of the engine`,
+`reading /usr/share/chromium-efl/lib/libchromium-impl.so`, the `ELF32 … needed, …
+imports (… weak)` counts line, one `refused` line per library the loader would not
+open, and the `N unresolved` line with up to forty names. A `DID NOT RETURN` on the
+read is the set parking the census; `still counting after 60 s` with lines behind it
+is a slow set, and the lines are still the answer. The reply also asks what the
+screen showed on the second open: a failure screen that never came is the second
+finding above, and the no-provider page is then the only page this set will ever
+serve, which the reporter's instructions have to say.
+
+**State of #105 (2026-10-02, evening):** the ladder's answer is in and is the good
+one; `build-283f3e9` shipped the census and the AU7200 never reached it; the build
+after it takes the census ahead of the engine and is the one the stub depends on. The
+reply quotes that tag. Nothing here changes the Q80's close: that set refused the
+mapping the AU7200 allows.
 
 ### `ELM_ACCEL` has to be set before the window exists
 
@@ -3254,17 +3315,36 @@ the one its report has to come from. The state is:
   three seconds before, paused the whole time. The fix is `NuiMseHold` (see
   *One video at a time*): the next reel's source is held back until TikTok
   plays it, so no second pipeline opens under the one on screen. Menu row to
-  switch it off. Shipped in `build-644c119`. **Waiting on:** his report from
-  that build, after a few reels. The trail answers it in three lines per reel: `hold
-  v2 blob while v1 plays`, then on the swipe `release v2 after Ns (play)`,
-  then the probe's `v2 playing`. Reels that play to the end with that sequence
-  is the fix. A `release` with no `playing` after it, or the next reel sitting
-  on its spinner, is TikTok's player waiting for the preload to be ready
-  before it will swap — the hold would then have to let the attach through and
-  the fallback is the automatic pause/play on the frozen reel, which recovers
-  after a freeze he would still see. Reels still freezing with `held 0` on the
-  `one video :` line is TikTok attaching by a path the hold does not see, and
-  the probe's `loadstart` line (`blob not ours`, `srcObject`) says which.
+  switch it off. Shipped in `build-644c119`. **His report from it (2026-10-02):
+  "if next is normal videos never freeze"**, and the one freeze left comes
+  "sometimes, if the next video is a live stream". Two trails. The one without a
+  freeze has TikTok's swipe as an element swap — `hold v2 blob while v1 plays`,
+  `v1 playing` two seconds later, v1 then playing for three minutes — and never
+  a release, because v1 had been readied before the script was in the page. The
+  one with the freeze has the hold doing exactly what it was built to do (`hold
+  v1 blob while v2 plays` at 17:08:19, `release v1 after 122.6s (play)` at
+  17:10:22, `v1 playing` three seconds after that) and, four seconds after the
+  hold, a path it does not cover: TikTok **emptied the on-screen element and
+  gave it a new source** — `v2 emptied`, `v2 loadstart blob not ours`, a new
+  MediaSource `ms4` at the same 768x576 — and v2 sat `rs0 starved stuck` for
+  thirteen seconds before it played, then paused and resumed every few seconds
+  for two minutes. Those pauses are the page's own: nothing of ours calls
+  `pause` or `play` but `NuiVideoCap`, and it ran zero times on both trails.
+  Whether the re-sourced v2 was the live stream, or a reel TikTok loaded cold into
+  the element on screen because the held one was not ready, the trail cannot say:
+  it had no duration and no address. The build after `build-644c119` adds both
+  to the probe's lines, read-only: `loadedmetadata 768x576, dur inf` (`inf` is
+  what a live stream reads, and also what an MSE clip reads until its page sets
+  a duration — `inf` with a playhead that never loops back is live) and
+  `, live page` on `loadstart` and `playing` while the address is `/@user/live`.
+  **Waiting on:** his report from that build with a freeze on it. `live page` or
+  `dur inf` on the re-sourced element is the live stream itself stalling, and a
+  live stream cannot buffer ahead: that is the proxy's throughput, the hold is
+  doing its job, and the answer is that live streams through that proxy will
+  stall. A finite `dur` on a `/video/` page with a `hold` a few seconds before is
+  TikTok loading into the on-screen element because the held one was not ready
+  — the hold would then have to release when the element on screen is emptied,
+  which is the next fix and a `tools/msehold` case before it is one.
 
 Five things about that set are settled and should not be re-derived: **key `5` is
 his, not ours** — the engine's overlay path is the only one that gives him a

src/common/DiagServer.cs

@@ -99,10 +99,24 @@ private static string Report()
                 // The probe's ledger outlives a launch, and the header line is the
                 // one a reporter copies first — so it is here too, on the page they
                 // get when the UI is not up yet.
+                //
+                // The build tag and the import census too (issue #105): on the AU7200
+                // every page has been this one, because the first launch of a build
+                // dies within seconds of the engine failing and the second never
+                // reaches the UI. The `build-283f3e9` reply told the reporter the
+                // page would start with `build :`, and it did not — this page had no
+                // such line — which is a round trip spent on doubt about which
+                // build was installed. What this page cannot show is what the
+                // *current* launch has not yet established; the census it carries
+                // is this launch's if it has run, and the previous run's lines are
+                // in the trail below.
                 return "Overscan — no report provider yet (UI has not started)\n\n" +
+                       "build     : " + BuildInfo.Tag + "\n" +
                        "own native : " + NativeProbe.Summary + "\n" +
+                       "engine imports: " + EngineImports.Summary + "\n" +
                        "trail write: " + Breadcrumbs.Status + "\n\n" +
                        "this run\n" + DiagLog.Dump() +
+                       "\nwhat a stub must provide (the implementation's imports)\n" + EngineImports.Dump() +
                        "\nprevious run (last line is where it died)\n" +
                        Breadcrumbs.Previous +
                        "\nengine stdout/stderr (previous run)\n" +

src/common/EngineImports.cs

@@ -4,6 +4,7 @@
 using System.IO;
 using System.Runtime.InteropServices;
 using System.Text;
+using System.Threading;
 
 namespace Overscan
 {
@@ -35,12 +36,24 @@ namespace Overscan
     ///    load. Whatever is still unresolved is what the stub has to provide, by name.
     ///
     /// Each step past managed code goes out under a <see cref="Deadline"/>, because
-    /// a call this firmware objects to is parked rather than refused, and it runs on
-    /// the post-failure probe thread after the engine has already failed: whatever
-    /// is most likely not to survive the firmware goes last. It is placed ahead of
-    /// the permission investigation on that thread because the investigation's last
-    /// step is the one the Q80 never came back from, and this census is the question
-    /// the next build is waiting on.
+    /// a call this firmware objects to is parked rather than refused.
+    ///
+    /// Where it runs was decided by the `build-283f3e9` report (2026-10-02). That build
+    /// put the census on the post-failure probe thread, behind the ladder's walk and
+    /// ahead of the permission investigation, and the census never ran: the launch
+    /// ended within five seconds of `ENGINE FAILURE` — no heartbeat tick, no deadline
+    /// miss — with the thread still on the walk's first repeatable rung. That is the
+    /// Q80's shape exactly (see <c>NativeProbe.StartEarlyIfUnfinished</c>): on these
+    /// sets a launch whose engine has failed is over before anything queued behind
+    /// the failure gets a turn, and the second launch of the same build never reached
+    /// the UI at all. So, like the walk, the census goes **ahead of the engine** on
+    /// an install whose ledger already says the engine fails here
+    /// (<c>NativeProbe.EngineFailedHere</c>), after the walk if one is due, and the
+    /// engine waits for it, bounded. On every set this app works on there is no
+    /// ledger and none of this runs. The post-failure call is still made, for the
+    /// first failure on an install, and finds the census already taken on the rest.
+    /// Its lines go to the trail as they are established, because the page that
+    /// shows them is served by the *next* launch, before its UI exists.
     ///
     /// How to read it: a short list of plain C function names is a no-op stub,
     /// buildable with the same toolchain recipe as `libovprobe.so`. C++ names (`_Z…`)
@@ -76,6 +89,9 @@ internal static class EngineImports
         private static readonly List<string> Lines = new List<string>();
         private static bool _ran;
 
+        /// <summary>Set whenever no census is running; reset while one is.</summary>
+        private static readonly ManualResetEvent Finished = new ManualResetEvent(true);
+
         /// <summary>One line for the diagnostics header.</summary>
         public static string Summary { get; private set; } = "(not asked)";
 
@@ -115,17 +131,36 @@ private static void Trace(string line)
         /// Never throws: a diagnostic has nobody to throw to.
         /// </summary>
         public static void Census(string implementation, string blocked)
+        {
+            if (Claim())
+            {
+                Take(implementation, blocked);
+            }
+        }
+
+        /// <summary>
+        /// Takes the one census this process gets, for whichever caller is first;
+        /// false for every caller after. The wait event is reset here, under the
+        /// same lock, so a caller that starts the census on a thread and then waits
+        /// cannot see the event still set from before the thread got going.
+        /// </summary>
+        private static bool Claim()
         {
             lock (Gate)
             {
                 if (_ran)
                 {
-                    return;
+                    return false;
                 }
 
                 _ran = true;
+                Finished.Reset();
+                return true;
             }
+        }
 
+        private static void Take(string implementation, string blocked)
+        {
             try
             {
                 Run(implementation, blocked);
@@ -135,6 +170,62 @@ public static void Census(string implementation, string blocked)
                 Summary = "threw " + ex.GetType().Name + ": " + ex.Message;
                 Trace(Summary);
             }
+            finally
+            {
+                try
+                {
+                    Finished.Set();
+                }
+                catch (Exception)
+                {
+                    // Nobody to tell.
+                }
+            }
+        }
+
+        /// <summary>
+        /// <see cref="Census"/> on a thread of its own, for the launch that runs it
+        /// ahead of the engine: the caller waits with <see cref="WaitForCensus"/>,
+        /// bounded, and a census still going when the bound passes carries on while
+        /// the engine is asked — its lines reach the trail either way. Claimed here,
+        /// before the thread exists, so the post-failure call finds it taken.
+        /// </summary>
+        public static void RunInBackground(string implementation, string blocked)
+        {
+            if (!Claim())
+            {
+                return;
+            }
+
+            try
+            {
+                var thread = new Thread(delegate () { Take(implementation, blocked); });
+                thread.IsBackground = true;
+                thread.Name = "engine-imports";
+                thread.Start();
+            }
+            catch (Exception ex)
+            {
+                // A set that will not give us a thread gets asked the dangerous way.
+                Trace("no thread (" + ex.GetType().Name + "), taking the census inline");
+                Take(implementation, blocked);
+            }
+        }
+
+        /// <summary>
+        /// Waits for a census started by <see cref="RunInBackground"/> to finish, for
+        /// at most <paramref name="timeoutMs"/>. True if it did (or none is running).
+        /// </summary>
+        public static bool WaitForCensus(int timeoutMs)
+        {
+            try
+            {
+                return Finished.WaitOne(timeoutMs);
+            }
+            catch (Exception)
+            {
+                return false;
+            }
         }
 
         private static void Run(string implementation, string blocked)