Chart loading: preload the module graph, warm exchange connections, stop reading the bars twice a beat

CHANGELOG.md

@@ -56,6 +56,27 @@ All notable changes to this project are documented here. Format: [Keep a Changel
 
 ### Fixed
 
+- **Chart loading and background traffic** (measured 6 Oct, one tab, the exchange mocked, headless Chromium; the
+  numbers are for localhost and grow with request latency).
+  - **The module graph is fetched in parallel.** Vela's workspace, vela-pinets, PineTS and ~140 small Zag /
+    floating-ui files were discovered one import level at a time — 17 sequential waves, about 1.3 s of a 1.8 s
+    boot. `index.html` now lists all 157 as `<link rel="modulepreload">`, generated by
+    `tools/modulepreload.py` (`--write` after a vendor upgrade; the test suite runs `--check`, so a stale list
+    fails there). First canvas 1,798 → 1,586 ms (median of 5); with 80 ms added per request 3,372 → 2,844 ms.
+    The block must come after the import map: a module fetch that starts first freezes the map and the workspace
+    silently falls back to the bare chart (a first version did exactly that and looked 300 ms faster — it was
+    measured by checking that the workspace really loaded; a test pins the order).
+  - **The exchange connections are warmed early** (`preconnect` for api.binance.com, fapi.binance.com and the
+    icon host), so the TLS handshakes happen while the module graph arrives instead of after it. Not measurable
+    offline; it removes a round trip the first bars wait on.
+  - **The heartbeat read the chart's bars twice every 4 s**, and when the chart hands none over (the normal case
+    on this Vela) each read was a live fetch to the venue through `/api/bars`: 16 in 30 idle seconds. It reads
+    once; `chartBars()` shares one fetch between callers within 3 s (and between callers arriving while one is in
+    flight); the server keeps a clean answer for 1.5 s. 16 → 8 in 30 s, and the overlay, the agent and a Pine run
+    no longer each pay their own round trip. An empty, failed or partial answer is never kept; every caller gets
+    its own array.
+  - Held by `test_boot_speed.py` (17 tests, each fix mutation-checked).
+
 - **Audit round 4, phase C: the four findings that were ours** (about 900 engine runs under Node; the engine-side
   findings are in the audit report, not here).
   - `docs/studies/pine/fvg-mitigation.pine` failed at its default inputs on any data without a gap yet

console/backend/server.py

@@ -1137,10 +1137,37 @@ def _fetch_klines(symbol: str, interval: str, limit: int, end_time: int | None =
         return json.load(res)
 
 
+# The same (symbol, interval, limit) asked for again within this many seconds is answered from memory. The page's
+# heartbeat, the overlay, the agent and a Pine run all ask for the chart's bars, and each used to be a live round
+# trip to the venue (measured 6 Oct: 16 in 30 idle seconds with one tab open). Only a clean answer is kept — a
+# failed or partial one is never served twice.
+BARS_TTL_S = 1.5
+_BARS_MEMO: dict = {}
+_BARS_MEMO_LOCK = threading.Lock()
+
+
 def fetch_bars(symbol: str, interval: str, limit: int) -> dict:
-    """`limit` most recent bars, oldest first, paged back from now. Never raises: a failed page is an
-    answer — the bars gathered so far come back with `partial: true` and the reason in `error`."""
+    """`limit` most recent bars, oldest first, paged back from now (the same ask within BARS_TTL_S is answered
+    from memory). Never raises: a failed page is an answer — the bars gathered so far come back with
+    `partial: true` and the reason in `error`."""
     limit = max(1, min(BARS_MAX, int(limit)))
+    key = (symbol, interval, limit)
+    now = time.monotonic()
+    with _BARS_MEMO_LOCK:
+        hit = _BARS_MEMO.get(key)
+        if hit and now - hit[0] < BARS_TTL_S:
+            return json.loads(hit[1])
+    out = _fetch_bars_uncached(symbol, interval, limit)
+    if not out.get("error") and out.get("bars"):
+        with _BARS_MEMO_LOCK:
+            if len(_BARS_MEMO) > 64:
+                for k in [k for k, v in _BARS_MEMO.items() if now - v[0] >= BARS_TTL_S] or list(_BARS_MEMO)[:32]:
+                    _BARS_MEMO.pop(k, None)
+            _BARS_MEMO[key] = (time.monotonic(), json.dumps(out))
+    return out
+
+
+def _fetch_bars_uncached(symbol: str, interval: str, limit: int) -> dict:
     rows: list = []
     error = None
     end_time = None

console/backend/tests/test_boot_speed.py

@@ -0,0 +1,226 @@
+"""Chart loading and background traffic (6 Oct 2026), measured and then held.
+
+What was found, with one tab open and the exchange mocked (localhost, headless Chromium):
+
+  * the ES-module graph (workspace.js -> Vela, vela-pinets, PineTS, ~140 Zag files) is discovered one import level at
+    a time: 17 sequential waves, about 1.3 s of a 1.8 s boot -> a generated `<link rel="modulepreload">` block;
+  * the first chart data waits for TLS to the exchange, which only started once the chart did -> `preconnect`;
+  * the 4-second heartbeat read the chart's bars TWICE, and when the chart hands none over each read was a live
+    round trip to the venue (16 in 30 idle seconds) -> one read, shared between callers, and cached on the server.
+
+The one trap, found by measuring (a first version "improved" boot by quietly breaking it): a module fetch that
+starts before the import map is parsed FREEZES the map, every bare specifier ("pinets") then fails to resolve, and
+the workspace falls back to the bare chart. The preload block must come after the import map.
+"""
+import json
+import os
+import re
+import shutil
+import subprocess
+import sys
+import unittest
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, HERE)
+sys.path.insert(0, os.path.dirname(HERE))
+ROOT = os.path.abspath(os.path.join(HERE, "..", "..", ".."))
+FRONT = os.path.join(ROOT, "console", "frontend")
+NODE = shutil.which("node")
+
+import server as srv  # noqa: E402
+
+
+def read(*parts):
+    with open(os.path.join(ROOT, *parts), encoding="utf-8") as fh:
+        return fh.read()
+
+
+INDEX = read("console", "frontend", "index.html")
+APP = read("console", "frontend", "app.js")
+BRIDGE = read("console", "frontend", "chart-bridge.js")
+
+
+class ModulePreload(unittest.TestCase):
+    def test_the_block_is_exactly_what_the_generator_derives(self):
+        done = subprocess.run([sys.executable, os.path.join(ROOT, "tools", "modulepreload.py"), "--check"],
+                              capture_output=True, text=True)
+        self.assertEqual(done.returncode, 0, done.stderr or "tools/modulepreload.py --write")
+
+    def test_the_block_comes_after_the_import_map(self):
+        """Before it, the browser freezes an empty map and the workspace silently falls back to the bare chart."""
+        importmap = INDEX.index('<script type="importmap">')
+        first_preload = INDEX.index('rel="modulepreload"')
+        first_module = INDEX.index('type="module"', importmap)
+        self.assertLess(importmap, first_preload)
+        self.assertLess(first_preload, first_module)
+        self.assertEqual(INDEX.count("<!-- modulepreload:start"), 1)
+        self.assertNotIn('rel="modulepreload"', INDEX[:importmap], "no module fetch may start before the import map")
+
+    def test_every_listed_file_exists_and_the_whole_graph_is_listed(self):
+        links = re.findall(r'<link rel="modulepreload" href="\./([^"]+)">', INDEX)
+        self.assertGreater(len(links), 100)
+        for href in links:
+            self.assertTrue(os.path.isfile(os.path.join(FRONT, href)), href)
+        self.assertEqual(len(links), len(set(links)), "no file twice")
+        for needed in ("vendor/vela/dist/workspace.js", "vendor/vela-pinets/dist/index.js", "vendor/pinets/pinets.min.browser.es.js"):
+            self.assertIn(needed, links)
+        # the generator must not mistake vela's own workspace.js for the page's entry file of the same name
+        self.assertIn("vendor/vela/dist/workspace.js", links)
+
+    def test_the_connections_the_first_data_needs_are_warmed(self):
+        for host, cors in (("https://api.binance.com", True), ("https://fapi.binance.com", True), ("https://crypto-icons.ledger.com", False)):
+            tag = f'<link rel="preconnect" href="{host}"' + (" crossorigin>" if cors else ">")
+            self.assertIn(tag, INDEX)
+        self.assertLess(INDEX.index('rel="preconnect"'), INDEX.index('type="importmap"'))
+
+
+class OneReadPerHeartbeat(unittest.TestCase):
+    def test_the_heartbeat_reads_the_bars_once(self):
+        body = BRIDGE.split("async function heartbeat() {", 1)[1].split("await api('/api/chart/state'", 1)[0]
+        self.assertEqual(body.count("window.chartBars()"), 1, "it used to read them twice per beat")
+        self.assertIn("const last = barsList.length ? barsList[barsList.length - 1].close : null;", body)
+
+
+@unittest.skipUnless(NODE, "node is not installed")
+class SharedBars(unittest.TestCase):
+    def helper(self):
+        m = re.search(r"const BARS_SHARE_MS = \d+;\nconst sharedBars = new Map\(\);\nfunction consoleBars\(.*?\n\}\n", APP, re.S)
+        self.assertTrue(m, "consoleBars not found in app.js")
+        return m.group(0)
+
+    def run_js(self, body):
+        script = "let NOW = 1000000;\nDate.now = () => NOW;\n" + self.helper() + "\n" + body
+        done = subprocess.run([NODE, "-e", script], capture_output=True, text=True, timeout=30)
+        if done.returncode != 0:
+            raise AssertionError(done.stderr[-1200:])
+        return json.loads(done.stdout.strip().splitlines()[-1])
+
+    FAKE = """
+      let calls = 0, mode = 'ok';
+      const row = (t) => ({ time: t, open: 1, high: 2, low: 0.5, close: 1.5, volume: 3, extra: 'x' });
+      globalThis.fetch = async () => { calls++; if (mode === 'throw') throw new TypeError('down');
+        return { json: async () => (mode === 'empty' ? { ok: true, data: { bars: [] } } : { ok: true, data: { bars: [row(1), row(2)] } }) }; };
+    """
+
+    def test_asks_in_the_same_moment_and_within_three_seconds_share_one_fetch(self):
+        got = self.run_js(self.FAKE + """
+          (async () => {
+            const [a, b] = await Promise.all([consoleBars('BTCUSDT', '1h', 500), consoleBars('BTCUSDT', '1h', 500)]);   // in flight together
+            NOW += 2900; const c = await consoleBars('BTCUSDT', '1h', 500);                                          // just inside
+            NOW += 200; const d = await consoleBars('BTCUSDT', '1h', 500);                                           // just outside
+            console.log(JSON.stringify({ calls, sizes: [a, b, c, d].map((x) => x.length), keys: Object.keys(a[0]) }));
+          })();""")
+        self.assertEqual(got["calls"], 2, "the first three asks are one fetch; the fourth is a new one")
+        self.assertEqual(got["sizes"], [2, 2, 2, 2])
+        self.assertEqual(got["keys"], ["time", "open", "high", "low", "close", "volume"], "only the bar fields come through")
+
+    def test_another_market_or_size_is_another_fetch(self):
+        got = self.run_js(self.FAKE + """
+          (async () => { await consoleBars('BTCUSDT', '1h', 500); await consoleBars('ETHUSDT', '1h', 500); await consoleBars('BTCUSDT', '4h', 500); await consoleBars('BTCUSDT', '1h', 2000);
+            console.log(JSON.stringify({ calls })); })();""")
+        self.assertEqual(got["calls"], 4)
+
+    def test_every_caller_gets_its_own_array(self):
+        got = self.run_js(self.FAKE + """
+          (async () => { const a = await consoleBars('BTCUSDT', '1h', 500); a.pop(); a.length = 0; const b = await consoleBars('BTCUSDT', '1h', 500);
+            console.log(JSON.stringify({ calls, b: b.length })); })();""")
+        self.assertEqual(got, {"calls": 1, "b": 2})
+
+    def test_an_empty_or_failed_answer_is_never_kept(self):
+        got = self.run_js(self.FAKE + """
+          (async () => {
+            mode = 'empty'; const a = await consoleBars('BTCUSDT', '1h', 500);
+            mode = 'ok'; const b = await consoleBars('BTCUSDT', '1h', 500);            // not served the empty one
+            mode = 'throw'; let threw = false; try { await consoleBars('SOLUSDT', '1h', 500); } catch (e) { threw = true; }
+            mode = 'ok'; const c = await consoleBars('SOLUSDT', '1h', 500);            // not served the failure
+            console.log(JSON.stringify({ calls, a: a.length, b: b.length, threw, c: c.length }));
+          })();""")
+        self.assertEqual(got, {"calls": 4, "a": 0, "b": 2, "threw": True, "c": 2})
+
+    def test_chartbars_uses_it(self):
+        self.assertIn("const rows = await consoleBars(symbol, interval, want);", APP)
+        self.assertIn("fetch(`/api/bars?", APP, "the same-origin fetch is still the one path to the venue")
+
+
+class ServerBarsMemo(unittest.TestCase):
+    def setUp(self):
+        self._orig = srv._fetch_klines
+        srv._BARS_MEMO.clear()
+        self.calls = []
+
+    def tearDown(self):
+        srv._fetch_klines = self._orig
+        srv._BARS_MEMO.clear()
+
+    @staticmethod
+    def kline(t):
+        return [t, "1", "2", "0.5", "1.5", "3", 0, "0", 0, "0", "0", "0"]
+
+    def fake(self, fail=False):
+        def f(symbol, interval, limit, end_time=None):
+            self.calls.append((symbol, interval, limit))
+            if fail:
+                raise OSError("upstream down")
+            return [self.kline(1), self.kline(2)]
+        srv._fetch_klines = f
+
+    def test_the_same_ask_within_the_window_is_one_upstream_call(self):
+        self.fake()
+        a = srv.fetch_bars("BTCUSDT", "1h", 500)
+        b = srv.fetch_bars("BTCUSDT", "1h", 500)
+        self.assertEqual(len(self.calls), 1)
+        self.assertEqual(a, b)
+        self.assertEqual(a["count"], 2)
+        a["bars"].clear()                                   # a caller changing its answer cannot change the next one
+        self.assertEqual(srv.fetch_bars("BTCUSDT", "1h", 500)["count"], 2)
+
+    def test_a_different_market_interval_or_size_is_its_own_call(self):
+        self.fake()
+        for args in (("BTCUSDT", "1h", 500), ("ETHUSDT", "1h", 500), ("BTCUSDT", "4h", 500), ("BTCUSDT", "1h", 1000)):
+            srv.fetch_bars(*args)
+        self.assertEqual(len(self.calls), 4)
+
+    def test_the_answer_expires(self):
+        self.fake()
+        srv.fetch_bars("BTCUSDT", "1h", 500)
+        key = ("BTCUSDT", "1h", 500)
+        stamp, payload = srv._BARS_MEMO[key]
+        srv._BARS_MEMO[key] = (stamp - srv.BARS_TTL_S - 0.1, payload)
+        srv.fetch_bars("BTCUSDT", "1h", 500)
+        self.assertEqual(len(self.calls), 2)
+
+    def test_a_failed_or_partial_answer_is_never_served_twice(self):
+        self.fake(fail=True)
+        first = srv.fetch_bars("BTCUSDT", "1h", 500)
+        self.assertIn("error", first)
+        srv.fetch_bars("BTCUSDT", "1h", 500)
+        self.assertEqual(len(self.calls), 2)
+        self.assertEqual(srv._BARS_MEMO, {})
+
+    def test_a_partial_answer_is_not_kept_either(self):
+        """Bars gathered, then a page failed: `partial: true` with an error. Serving that again would hide that the
+        rest was never fetched; the next ask tries again."""
+        def f(symbol, interval, limit, end_time=None):
+            self.calls.append(end_time)
+            if end_time is not None:
+                raise OSError("second page down")
+            return [self.kline(1000 + i) for i in range(1000)]
+        srv._fetch_klines = f
+        first = srv.fetch_bars("BTCUSDT", "1h", 1500)
+        self.assertTrue(first.get("partial"))
+        self.assertEqual(srv._BARS_MEMO, {})
+        srv.fetch_bars("BTCUSDT", "1h", 1500)
+        self.assertEqual(len(self.calls), 4, "two pages asked for, twice")
+
+    def test_the_memo_stays_small(self):
+        self.fake()
+        for i in range(200):
+            srv.fetch_bars(f"S{i}USDT", "1h", 500)
+        self.assertLessEqual(len(srv._BARS_MEMO), 70)
+
+    def test_the_window_is_short(self):
+        self.assertLessEqual(srv.BARS_TTL_S, 2.0)
+
+
+if __name__ == "__main__":
+    unittest.main()

console/frontend/app.js

@@ -683,6 +683,32 @@ function marketFromDom() {
   };
 }
 
+/* The console's bars for one market, shared. The heartbeat (every 4 s), the overlay and a Pine run each asked for
+   them, and each ask was a live round trip to the venue — 16 in 30 idle seconds with one tab open (measured 6 Oct).
+   Asks within BARS_SHARE_MS of each other, and asks that arrive while one is in flight, share ONE fetch; an empty or
+   failed answer is never kept; every caller gets its own array. */
+const BARS_SHARE_MS = 3000;
+const sharedBars = new Map();
+function consoleBars(symbol, interval, want) {
+  const key = symbol + '|' + interval + '|' + want;
+  const now = Date.now();
+  const hit = sharedBars.get(key);
+  if (hit && now - hit.at < BARS_SHARE_MS) return hit.promise.then((rows) => rows.slice());
+  for (const [k, v] of sharedBars) if (now - v.at >= BARS_SHARE_MS) sharedBars.delete(k);
+  const promise = (async () => {
+    const res = await fetch(`/api/bars?symbol=${encodeURIComponent(symbol)}&interval=${encodeURIComponent(interval)}&limit=${want}`);
+    const payload = await res.json();
+    // The console wraps every endpoint the same way: { ok, data: { bars, count, ... } }.
+    const rows = payload && payload.ok && payload.data && Array.isArray(payload.data.bars) ? payload.data.bars : [];
+    return rows.map((r) => ({ time: r.time, open: r.open, high: r.high, low: r.low, close: r.close, volume: r.volume }));
+  })();
+  const entry = { at: now, promise };
+  sharedBars.set(key, entry);
+  const forget = () => { if (sharedBars.get(key) === entry) sharedBars.delete(key); };
+  promise.then((rows) => { if (!rows.length) forget(); }, forget);
+  return promise.then((rows) => rows.slice());
+}
+
 /**
  * The bars PineTS runs over. Vela does not document one public bars getter across
  * builds, so try each accessor that exists and fall back to the venue's own public
@@ -719,17 +745,8 @@ async function chartBars(opts) {
     // Access-Control-Allow-Origin for http://127.0.0.1:8787, so the browser blocked this fetch and
     // every run reported "0 bars available". Same origin, and the console normalises the chart's
     // display timeframe ("30M") to the lowercase interval the venue accepts.
-    const res = await fetch(`/api/bars?symbol=${encodeURIComponent(symbol)}&interval=${encodeURIComponent(interval)}&limit=${want}`);
-    const payload = await res.json();
-    // The console wraps every endpoint the same way: { ok, data: { bars, count, ... } }.
-    const rows = payload && payload.ok && payload.data && Array.isArray(payload.data.bars)
-      ? payload.data.bars
-      : null;
-    if (rows && rows.length) {
-      return rows.map((r) => ({
-        time: r.time, open: r.open, high: r.high, low: r.low, close: r.close, volume: r.volume
-      }));
-    }
+    const rows = await consoleBars(symbol, interval, want);
+    if (rows.length) return rows;
   } catch {}
   return [];
 }

console/frontend/chart-bridge.js

@@ -227,14 +227,11 @@
     try {
       const c = chart();
       const market = marketFromDom();
-      let last = null;
-      if (typeof window.chartBars === 'function') {
-        const bars = await window.chartBars();
-        if (bars && bars.length) last = bars[bars.length - 1].close;
-      }
+      // One read of the bars per beat (it used to be two, each a live fetch when the chart hands none over).
       // chart.market.timeframe is stale in this build (says 4h on a 15m chart), so the heartbeat
       // reports the timeframe the bars actually have; the raw field rides along for diagnostics.
-      const barsList = typeof window.chartBars === 'function' ? await window.chartBars() : [];
+      const barsList = typeof window.chartBars === 'function' ? ((await window.chartBars()) || []) : [];
+      const last = barsList.length ? barsList[barsList.length - 1].close : null;
       const inferred = (window.PineTSRunner && window.PineTSRunner.marketContext)
         ? window.PineTSRunner.marketContext(barsList) : null;
       await api('/api/chart/state', {