Sync upstream (September 26)

.version

@@ -1 +1 @@
-2.15.1
+2.16.0

CHANGELOG.md

@@ -4,6 +4,8 @@ All notable changes to the NPMplus Security Fork are documented here. The fork u
 
 ## Unreleased
 
+- Reconciled upstream develop through `a4a29e09`, closing the 71-commit gap since `03688155` (regular merge commit, ancestry preserved). Upstream moved the nginx online/err, directory, mTLS, and DNS-challenge state out of row `meta` into `npmplus_*` columns (`meta_to_columns`, which also wipes `meta`), and its new global `jsonReplacer` — wired through `app.set("json replacer", …)` — blanks `password` and drops `certificate`, `certificate_key`, and the dns-provider credentials from every API response, superseding the fork's per-site meta masking for those fields. The fork's forward-destination reachability probe survives as the only remaining meta writer (`reach_ok`/`reach_err`, patched after `configure` persists its own online/err columns), and `proxy_host`'s `$parseDatabaseJson` keeps `meta` readable so the UI reach badge still works; `Table.jsx` sorts enabled → offline → unreachable → online and reads `npmplus_nginx_online` with the meta reach state. The access-list row masking (`maskItems`/`maskAccessListItems`, lodash `_.omit` plus blanked item passwords) is kept — upstream's recursive replacer may now cover part of it, and per FORK.md's replacement rule that redundancy sweep happens in a normal reviewable commit, not in the merge. The merged permission model takes upstream's sync `canAdmin()`/`canUser(id)` (throwing) and async `can()` with the crowdsec routes gating through a local `requireAdmin`; host CRUD takes upstream's `savedRow`/`try`/`finally` shape with the fork's `assertPrivilegedNginxFields`, `validateIncomingPort`, mTLS, and SSL/HSTS cleanup intact; `user.js` takes upstream's email/avatar handling with the fork's bounded gravatar fetch. `lodash` returns to the backend manifest for the masking functions. Dependency aging held: `@tabler/core` 1.6.0 and `vite` 8.3.1 failed the `minimumReleaseAge` policy and stay at 1.5.1/8.3.0, with both lockfiles regenerated under the enforced window. Adopted upstream's ECH key rotation (`/opt/npmplus/tls/ech/cron.sh`, `ECH_ROTATION_INTERVAL`, the cloudflare example script); the README section keeps to the release-discipline word budget and the full guide moved to `docs/ech.md`. Kept the fork's caddy source build with its CVE pin matrix, and the goaccess CSP/no-cache headers in `npmplus.conf`; upstream's goaccess dinit flag reorder landed. Test fixtures and the live smokes track the new columns (`certificate-dns.test.js`, the `sqlite-upgrade.test.js` raw-knex legacy seed, `security-regressions.mjs` and `rc5-features.mjs` reading `npmplus_nginx_online` plus `meta.reach_ok`). Verified by 163 backend and 15 frontend tests, `validate-schema`, biome on both LF-normalized trees, the vite build, `tests/security-invariants.mjs`, frozen lockfiles, the 45-check in-process backend smoke, and the full disposable-container smoke (socket-injection regressions, MFA replay, browser-driven first-admin setup and UI flows); the Linux-only python contracts reproduce identically on a pristine fork-HEAD worktree and stay green on CI.
+
 - Third same-day upstream push (`03688155`–`a2b765f8`, the locale simplification pass): adopted the reload-based `changeLocale`, module-level `document.dir`/`lang` application, and `formatDateTime` reading the active locale internally — the fork's CrowdSec views dropped the now-deleted `LocaleContext`/`useLocaleState` plumbing along with the locale argument, while keeping their `crowdsec-decision` audit rendering (gavel icon, `Decision #<id>`, i18n'd unknown-type fallback) that upstream cannot know about. The tabler stylesheet selection moved to a top-level `await` keyed on `document.dir` (set synchronously at `src/locale` module init, so it is readable before the awaits), with `installDeploymentRecovery()` called before them so a broken deploy 404ing the chunks cannot take the boot guard down too. `vite.config.js` keeps `tabler.rtl.min.css` out of the `ui-vendor` code-splitting group — required for any dynamic-tabler shape, since the group otherwise merges both stylesheets into one always-loaded asset and applies RTL rules to LTR pages (the +12px dialog overflow that broke the responsive smoke checks in the second pass). `api/backend/base.js` keeps the fork's 401 handling — the 401 branch runs before `response.json()` so a proxy's HTML error page cannot crash the parse, logout still calls `deleteToken()`, `error.status` survives, and null `params`/bare `AbortSignal` are tolerated — capabilities upstream's simplification removed from its own copy. Verified by the full container smoke (171 checks, including the five responsive-layout checks that exposed the chunking bug), 163 backend and 15 frontend tests, and biome clean on the LF-normalized tree.
 
 - Second reconciliation of upstream develop after another force-push (merge-base regressed to `20b56ee8` again; patch-id comparison shows everything already carried was re-offered rebased, and eight commits are genuinely new). Kept the fork's DB-level TOTP/challenge claims over upstream's new claim-before-verify reorder, which exists to serialize their in-memory Maps; the SQL step claim already makes that race impossible atomically and preserves the friendlier mistyped-code retry. Adopted upstream's removal of the `validate: { trustProxy: false }` limiter suppressions — dead code since `trust proxy` became `1`, and dropping it restores fail-loud behavior if the setting ever returns to `true`. Dockerfile reconciled as fork-plus-deltas: the fork's pip/certbot/luarocks pins and the crowdsec telemetry instrumentation survive, while upstream's `RCP_VER`→`ORP_VER` rename, the nginx patch-2 swap to PR 1756, and openresty patches 8–10 (resolver hosts, upstream pipelining, reuseport fd cleanup) land; nginx.conf auto-merged to `resolver local=on hosts=on ipv6=on` and the reworked quoted `alog` format, with goaccess taking upstream's matching log-format fields plus the fork's `--external-assets` for the strict CSP. Dependencies split by maturity: took `multer 2.4.0` and `react-router 8.4.0` (now past the 7-day window), held back `@biomejs/biome 2.5.14`, `undici 8.11.0`, `json-schema-ref-parser 16.0.3`, `swagger-parser 13.1.0`, `@tabler/icons-react 3.48.0`, `react-query 5.103.2`, `markdown-to-jsx 9.10.3`, and `react-intl 12.1.2` (a two-major jump; kept at 10.1.26) for the renovate cron, with lockfiles regenerated under the enforced `minimumReleaseAge`. Adopted upstream's RTL support on top of the fork's `installDeploymentRecovery()` in `main.jsx` (dynamic RTL tabler stylesheet, document `dir`/`lang` handling, Persian and Turkish translations); the RTL code uses only APIs present in react-intl 10. Certificates keep the fork's stricter multer limits (`fields: 0, parts: 2, fieldNameSize: 64` were already there), and the fail-closed CrowdSec `APPSEC_URL` default stays.

Dockerfile

@@ -84,6 +84,7 @@ RUN git-clone-commit.sh https://github.com/nginx/nginx "$NGINX_VER" /src/nginx &
     wget -q https://raw.githubusercontent.com/openresty/openresty/master/patches/nginx/"$ORP_VER"/nginx-"$ORP_VER"-resolver_hosts.patch -O /src/nginx/8.patch && \
     echo "7a3e9ebe4fafaef0a90773ed093bed83c3e753af5983718b0aee50881c32151b  /src/nginx/8.patch" | sha256sum -c - && \
     git apply /src/nginx/8.patch && \
+    sed -i "s|ngx_destroy_pool(r->hosts->pool);|r->hosts->pool->log = r->log; &|" /src/nginx/src/core/ngx_resolver.c && \
     wget -q https://raw.githubusercontent.com/openresty/openresty/master/patches/nginx/"$ORP_VER"/nginx-"$ORP_VER"-upstream_pipelining.patch -O /src/nginx/9.patch && \
     echo "f147c9724a0ad33084a3cb51acdafbd4dbd2c40ba5840af1684b7b9595b24ae2  /src/nginx/9.patch" | sha256sum -c - && \
     git apply /src/nginx/9.patch && \
@@ -251,6 +252,7 @@ RUN apk upgrade --no-cache -a && \
     luarocks-5.1 install lua-resty-http 0.18.0-0 && \
     luarocks-5.1 install lua-resty-string 0.09-0 && \
     luarocks-5.1 install lua-resty-openssl 1.9.0-1 && \
+    sed -i 's|^local legacy_nids = {}$|C.ERR_clear_error()\n&|' /usr/local/share/lua/5.1/resty/openssl/pkey.lua && \
     \
     git config --global advice.detachedHead false && \
     git config --global init.defaultBranch main && \

FORK.md

@@ -528,3 +528,68 @@ moved to a top-level `await` keyed on `document.dir` (set synchronously at
 `main.jsx` adopts the top-level `await` shape with `installDeploymentRecovery()`
 called before the stylesheet awaits, so the boot guard installs even when a
 broken deploy 404s the chunks it awaits.
+
+## Upstream merge resolution notes (September 26)
+
+The September 26 reconciliation on `fix/upstream-sync-20260926` merges
+upstream `a4a29e09` (merge-base `03688155`, the 71-commit gap) into the
+fork's develop state after PRs #35–#39 landed. The dominant upstream change
+is the move from row `meta` to dedicated `npmplus_*` columns
+(`meta_to_columns`: nginx online/err, directory, mTLS verify, and the
+certificate dns-provider fields), together with a global `jsonReplacer` in
+`helpers.js` — wired with `app.set("json replacer", …)` — that blanks
+`password` and drops `certificate`, `certificate_key`, and the dns-provider
+credentials from every response. That replacer supersedes the fork's old
+per-site meta masking for those fields, so the merge adopts upstream's
+shapes. Two fork behaviors ride on the new model:
+
+- The forward-destination reachability probe (`configureWithReachability`)
+  survives as the only remaining meta writer: it runs `internalNginx.configure`
+  — which now persists `npmplus_nginx_online`/`npmplus_nginx_err` itself —
+  then patches only `reach_ok`/`reach_err` into meta. `proxy_host`'s
+  `$parseDatabaseJson` no longer destructures `meta` out (upstream's version
+  stripped it), so the probe state reaches the API; the frontend sorts
+  enabled → offline → unreachable → online and passes
+  `npmplus_nginx_online`/`meta.reach_ok` separately.
+- The access-list row masking (`maskItems`/`maskAccessListItems`, lodash
+  `_.omit` plus blanked item passwords) is kept, and `lodash` returns to the
+  backend manifest for it. Upstream's recursive replacer may now blank the
+  nested item passwords too; per this file's replacement rule, any redundancy
+  sweep happens later in a normal reviewable commit — never inside the merge.
+
+The permission model is a true MERGE-BOTH: upstream's sync `canAdmin()` and
+`canUser(id)` (both throwing, ids must be positive) and async `can()` are
+adopted, while the fork-only crowdsec routes gate through a local
+`requireAdmin(res)` helper because their handlers are not per-user scoped.
+Host CRUD takes upstream's `savedRow`/`try`/`finally` shape with the fork's
+`assertPrivilegedNginxFields`, `validateIncomingPort`, mTLS, and SSL/HSTS
+cleanup intact; `user.js` takes upstream's email/avatar handling and keeps
+the fork's bounded gravatar fetch. `token.js` keeps the fork's session-token
+architecture. Dependency aging held: `@tabler/core` 1.6.0 and `vite` 8.3.1
+failed the `minimumReleaseAge` window and stay at 1.5.1/8.3.0 (both
+lockfiles regenerated under the policy). Upstream's ECH key rotation is
+adopted (rootfs hooks, `ECH_ROTATION_INTERVAL`, the cloudflare example);
+the README section keeps the 1300-word release-discipline budget and the
+full guide lives in `docs/ech.md`. The caddy build keeps the fork's source
+build and CVE pin matrix, `npmplus.conf` keeps the goaccess CSP/no-cache
+headers, and upstream's goaccess dinit flag reorder landed.
+
+Test fixtures and smokes track the new contract per this file's rules:
+`certificate-dns.test.js` moved its fixture to the `npmplus_*` columns,
+`sqlite-upgrade.test.js` seeds the legacy proxy row through raw knex with
+the columns the pre-replay schema actually had and asserts survival at the
+field level, and the live smokes (`security-regressions.mjs`,
+`rc5-features.mjs`) read `npmplus_nginx_online` plus `meta.reach_ok`. The
+security-regressions assertions were updated because the response shape
+changed upstream, not to loosen a check: the container smoke caught the
+first miss against the real image.
+
+Local validation: 163 backend and 15 frontend tests, `validate-schema`,
+biome on both LF-normalized trees, the vite build,
+`tests/security-invariants.mjs`, frozen lockfiles, the 45-check
+in-process backend smoke, and the full disposable-container smoke
+(security-regressions, modal-ui, ui-driver, and browser-driven
+security-ui against the image built from this branch). The Linux-only
+python contracts (installer-recovery, heal-migration, upstream-sync) and
+`sort-locale.sh` (needs jq) reproduce identically on a pristine fork-HEAD
+worktree on this rig and stay green on CI.

README.md

@@ -73,6 +73,10 @@ Every service should say `Up`, and `npmplus` should become `healthy` after start
 
 Moving to another machine, reading logs, recovering a failed update, and the advanced opt-ins an ordinary update deliberately preserves (AppSec, protected startup, the Cloudflare origin lock) are all covered in [host setup and operations](docs/setup-npmplus.md) - migration is three commands, and [Diagnostics](docs/setup-npmplus.md#diagnostics) covers logs, CrowdSec checks, and backup restoration.
 
+## Encrypted Client Hello (ECH)
+
+NPMplus can generate and automatically rotate ECH keys: fill `/opt/npmplus/tls/ech/cron.sh` with a script that calls the built-in `ech.sh` and pushes the resulting config to your DNS provider's HTTPS records. The container runs the script hourly (`ECH_ROTATION_INTERVAL` in `compose.yaml`), enables ECH in nginx, and reloads. Clearing the file disables ECH again. Full instructions and a Cloudflare example script ([`ech-cron-cloudflare-example.sh`](ech-cron-cloudflare-example.sh)) are in [docs/ech.md](docs/ech.md).
+
 ## What this fork adds
 
 - One interactive installer for installation, updates, diagnostics, restore, and uninstall, with loopback-only dashboard access by default.

backend/.smoke/rc5-features.mjs

@@ -88,13 +88,12 @@ const created = await request("POST", "/api/nginx/proxy-hosts", createBody);
 check("proxy host is created", created.status === 200 || created.status === 201, JSON.stringify(created));
 const hostId = created.json?.id;
 const listAfterCreate = await request("GET", "/api/nginx/proxy-hosts");
-const createdMeta = listAfterCreate.json?.find((item) => item.id === hostId)?.meta;
+const createdRow = listAfterCreate.json?.find((item) => item.id === hostId);
+const createdMeta = createdRow?.meta;
 check(
 	"created host appears in the list with online meta and a reachability probe",
-	listAfterCreate.json?.some((item) => item.id === hostId && item.meta) &&
-		createdMeta?.nginx_online === true &&
-		createdMeta?.reach_ok === false,
-	JSON.stringify(createdMeta),
+	createdRow?.npmplus_nginx_online === true && createdMeta?.reach_ok === false,
+	JSON.stringify(createdRow),
 );
 
 const updated = await request("PUT", `/api/nginx/proxy-hosts/${hostId}`, {

backend/.smoke/security-regressions.mjs

@@ -102,7 +102,7 @@ for (const destination of [
 }
 const created = await delegated.request("POST", "/nginx/proxy-hosts", host);
 check("normal delegated proxy created", created.status, 201);
-check("real nginx accepts normal proxy", created.data.meta.nginx_online, true);
+check("real nginx accepts normal proxy", created.data.npmplus_nginx_online, true);
 const hostUrl = `/nginx/proxy-hosts/${created.data.id}`;
 check(
 	"socket update rejected",
@@ -151,7 +151,7 @@ const app = await admin.request("POST", "/nginx/proxy-hosts", {
 	forward_port: null,
 });
 check("admin application socket remains supported", app.status, 201);
-check("nginx accepts application socket", app.data.meta.nginx_online, true);
+check("nginx accepts application socket", app.data.npmplus_nginx_online, true);
 await admin.request("DELETE", `/nginx/proxy-hosts/${app.data.id}`);
 for (const username of ["bad:name", "bad\nname", "bad\rname"]) {
 	check(

backend/.smoke/smoke-backend.mjs

@@ -70,6 +70,7 @@ app.use(
 			Object.defineProperty(res.locals, "access", {
 				get: () => ({
 					can: async () => true,
+					canAdmin: () => true,
 					token: { getUserId: () => 1 },
 				}),
 				set: () => {},
@@ -207,21 +208,22 @@ const server = app.listen(13000, "127.0.0.1", async () => {
 	// 7. alerts context for an ip: attacker geo + sanitized event meta
 	const alerts = await fetch(`${base}/alerts?scope=Ip&value=198.51.100.7`, { headers: admin });
 	const alertsBody = await alerts.json();
+	const alertsItems = alertsBody.items ?? [];
 	check(
 		"GET alerts -> 200 with the alert",
-		alerts.status === 200 && alertsBody.length === 1 && alertsBody[0].scenario === "crowdsecurity/http-probing",
+		alerts.status === 200 && alertsItems.length === 1 && alertsItems[0].scenario === "crowdsecurity/http-probing",
 		`got ${alerts.status} ${JSON.stringify(alertsBody).slice(0, 80)}`,
 	);
 	check(
 		"alert carries attacker geo details",
-		alertsBody[0]?.source?.country === "DE" && alertsBody[0]?.source?.as_name === "Example ASN",
-		JSON.stringify(alertsBody[0]?.source),
+		alertsItems[0]?.source?.country === "DE" && alertsItems[0]?.source?.as_name === "Example ASN",
+		JSON.stringify(alertsItems[0]?.source),
 	);
 	check(
 		"alert events keep attack meta but strip unknown keys",
-		alertsBody[0]?.events?.[0]?.meta?.some((m) => m.key === "target_uri" && m.value === "/.env") &&
-			!alertsBody[0]?.events?.[0]?.meta?.some((m) => m.key === "raw_request"),
-		JSON.stringify(alertsBody[0]?.events?.[0]?.meta),
+		alertsItems[0]?.events?.[0]?.meta?.some((m) => m.key === "target_uri" && m.value === "/.env") &&
+			!alertsItems[0]?.events?.[0]?.meta?.some((m) => m.key === "raw_request"),
+		JSON.stringify(alertsItems[0]?.events?.[0]?.meta),
 	);
 
 	// 8. no machine key file -> 503 not-wired-machine

backend/app.js

@@ -1,9 +1,13 @@
 import crypto from "node:crypto";
 import cookieParser from "cookie-parser";
 import express from "express";
+import multer from "multer";
+import { jsonReplacer } from "./lib/helpers.js";
 import { debug, express as logger } from "./logger.js";
 import mainRoutes from "./routes/main.js";
 
+Object.assign(multer.MulterError.prototype, { public: true, status: 400 });
+
 /**
  * App
  */
@@ -12,6 +16,7 @@ const app = express();
 app.set("trust proxy", 1);
 app.disable("x-powered-by");
 app.set("json spaces", 2);
+app.set("json replacer", jsonReplacer);
 
 app.use(cookieParser(process.env.COOKIE_SECRET || crypto.randomBytes(16).toString("hex")));
 app.use(express.json({ limit: "1mb" }));
@@ -50,11 +55,13 @@ app.use("/", mainRoutes);
 // production error handler
 // no stacktraces leaked to user
 app.use((err, req, res, _) => {
+	const status = err.status || 500;
+	const exposed = err.public || err.expose;
 	const requestId = crypto.randomUUID();
 	const payload = {
 		error: {
-			code: err.status || 500,
-			message: err.public ? err.message : "Internal Error",
+			code: status,
+			message: exposed ? err.message : "Internal Error",
 			request_id: requestId,
 		},
 	};
@@ -71,7 +78,7 @@ app.use((err, req, res, _) => {
 	// Not every error is worth logging - but this is good for now until it gets annoying.
 	if (typeof err.stack !== "undefined" && err.stack) {
 		debug(logger, `[${requestId}] ${err.stack}`);
-		if (typeof err.public === "undefined" || !err.public) {
+		if (!exposed) {
 			logger.warn(`[${requestId}] ${req.method.toUpperCase()} ${req.originalUrl}: ${err}`);
 		}
 	}