M69: web fetch on both backends, with M44b's network safety

.github/workflows/build.yml

@@ -161,15 +161,17 @@ jobs:
       - run: npm ci
       - run: npm run package
       # What must be inside the package: the licence, the third-party
-      # notices, both dictation helpers and the five bundles (the webview
+      # notices, both dictation helpers and the six bundles (the webview
       # loads main.css beside main.js; the extension requires modelApi.js
-      # when the Model API backend first starts, M57).
+      # when the Model API backend first starts, M57, and starts
+      # pageWorker.js as a worker for each web page it converts, M69).
       - name: the .vsix carries the notices, the helpers, the bundles and the manifest's text
         run: |
           listing="$(unzip -Z1 ./*.vsix)"
           for entry in extension/LICENSE.txt extension/THIRD_PARTY_NOTICES.txt extension/package.nls.json \
             extension/native/windows/dictate.ps1 extension/native/darwin/muse-dictate \
             extension/dist/extension.js extension/dist/modelApi.js extension/dist/searchWorker.js \
+            extension/dist/pageWorker.js \
             extension/dist/webview/main.js extension/dist/webview/main.css; do
             if ! grep -qx "$entry" <<< "$listing"; then
               echo "::error::$entry is missing from the .vsix" >&2

.vscodeignore

@@ -2,6 +2,7 @@
 !dist/extension.js
 !dist/modelApi.js
 !dist/searchWorker.js
+!dist/pageWorker.js
 !dist/webview/main.js
 !dist/webview/main.css
 !media/icon.svg

AGENTS.md

@@ -110,14 +110,19 @@ them, the milestone plan, and the certification checklist.
 src/extension.ts      activation: the view, the panel, the commands, the openers
 src/host/**           VS Code adapters (views, conversation, backend managers,
                       the Model API bundle's entry (dist/modelApi.js, loaded
-                      when that backend first starts) and the search worker,
+                      when that backend first starts), the search worker and
+                      web fetch's page converter worker (dist/pageWorker.js,
+                      started for each page),
                       commands, auth, settings, mentions,
-                      editor tracking, usage trace logs, voice, the diagnostics
-                      MCP server, the MCP servers' spawner, the network posture)
+                      editor tracking, usage trace logs, voice, the IDE tool
+                      MCP server (diagnostics, images, web fetch), the MCP
+                      servers' spawner, the network posture, web fetch's
+                      pinned transport)
 src/core/**           backend-agnostic logic; must not import `vscode`
                       (MSP host, Model API client and tools, the MCP client,
                       context, Muse Code's memory, export, worktrees, usage,
-                      dictation, Muse Voice, the paid gate, network failures)
+                      dictation, Muse Voice, the paid gate, network failures,
+                      web fetch: public-address checks and the HTML converter)
 src/shared/**         constants + zod protocol shared by host and webview
 src/shared/l10n/**    the English table (en.ts), fill/plural/Intl helpers, the
                       table checks and the list of translated languages

CHANGELOG.md

@@ -7,6 +7,68 @@ happened, not what was planned; superseded entries are kept.
 
 ## [Unreleased]
 
+### Added
+
+- **Web fetch on both backends** (M69, PLAN.md D49; folds in M44b). The
+  model can read one public web page it found or you named: `web_fetch` on
+  the Model API backend, and `mcp__ide__webFetch` on Muse Code, whose own
+  `web_fetch` is switched off. The extension fetches the page from your
+  machine; it is free, not Meta's paid search.
+  - `https://` only, public internet addresses only: the name is resolved
+    here and refused when any answer is loopback, private, link-local,
+    carrier-grade NAT, cloud metadata or reserved (IPv4-mapped, NAT64 and
+    6to4 forms judged by the IPv4 inside), and local or reserved names are
+    refused before any lookup. The connection goes to the address that was
+    checked, never to a second lookup; TLS still verifies the name.
+  - Same-host redirects are checked and pinned again, at most five; a
+    redirect to another host is handed back to the model. 5 MiB after
+    decompression, 30 seconds, an allow-list of text types; HTML becomes
+    Markdown, text stays as it is, anything else is refused with the reason.
+  - Through VS Code's proxy and certificates: the proxy is asked to tunnel
+    to the checked address, and a proxy's own answer is refused as such,
+    never read as the page.
+  - On the Model API backend it asks per host in Manual, Edit automatically
+    and Auto ("Always allow in this session" covers that host), runs in
+    Bypass, and is refused in Plan and in Restricted Mode. On Muse Code the
+    tool is listed only in a trusted workspace whose
+    `museSpark.sandboxNetwork` is not `restricted`, declares itself
+    open-world and not read-only, and the extension asks in its own dialog
+    before every fetch.
+  - The model receives the page between random markers, with a note that it
+    is untrusted content; the row shows the URL, the size and type, and what
+    the model read. 23 new strings in fifteen languages.
+  - After review: Muse Code's Stop (a closed request, or
+    `notifications/cancelled`) stops the fetch and voids a later answer in
+    the dialog, which is also asked only once per URL at a time and checks
+    the workspace again after it; the checked addresses are raced as RFC 8305
+    says; failures name the page's host and the addresses tried instead of
+    M56's advice about Meta, and a network failure's detail only by its
+    error codes (never a certificate's names); a server's text reaches the
+    model outside the markers only as short tokens; the HTML converter is bounded; names with
+    trailing dots or empty labels are refused; a network's own NAT64 prefix
+    is discovered (RFC 7050), and while it cannot be learned no IPv6 answer
+    is used (only a DNS answer proves there is none); pages are parsed by
+    HTML's own rules (implied ends, misnested and self-closed tags, SVG and
+    MathML, comments and scripts), and the Markdown is the page's text as
+    served, which can include text a browser would not show (no stylesheet
+    or hiding attribute is read, since hiding cannot be worked out
+    completely and visible small print hides nothing), all of it between
+    the untrusted markers, as the tool's description and the note now say;
+    a hook's "allow" no longer replaces
+    the per-host card; trust
+    and the mode are asked again after the card, before each request and
+    before the page reaches the model; damaged compression and unknown charsets are
+    handled as a browser would; `museSpark.sandboxNetwork`'s description now
+    says it also hides web fetch from Muse Code. Fifteen more strings, one
+    changed and one dropped, and two changed setting descriptions, in
+    fifteen languages.
+- **Dependencies.** Web fetch parses HTML with `parse5` 8.0.1 (MIT)
+  and sniffs its encoding with `html-encoding-sniffer` 6.0.0 (MIT), both
+  already in the tree through the test tools. They load only in
+  `dist/pageWorker.js` (201 KiB, budget 300 KiB), on a worker thread started for each page (at
+  most two at once) and stopped at 10 seconds or 512 MiB; `dist/extension.js` does not carry
+  them. `entities` is no longer a direct dependency.
+
 ### Changed
 
 - **Every paid use asks first, in a popup** (M58, PLAN.md D48): **Allow

CONTRIBUTING.md

@@ -186,6 +186,15 @@ stand when it runs, never a copy taken at activation. Tests never need a
 proxy or a certificate: they use the failure shapes Node 24 was seen to
 throw (`docs/certification/m56.md`).
 
+Web fetch (PLAN.md M69) is the one exception to `fetch`: it must connect to
+the address it checked, which VS Code's patched `fetch` cannot do, so it
+uses Node's `https` (`src/host/web/pinnedRequest.ts`), which VS Code patches
+for its proxy and certificates too. Keep every destination check in
+`src/core/web/` and test it over the fake resolver and transport in
+`test/unit/webFetch.test.ts`; unit tests never reach the internet. What
+only VS Code can show (the proxy asked for the pinned address) is in
+`test/integration/webFetch.test.ts`, against a loopback proxy.
+
 ## Licence
 
 By contributing you agree that your contribution is licensed under the MIT