merge upstream develop while preserving fork security contracts

.github/workflows/dependency-updates.yml

@@ -176,9 +176,6 @@ jobs:
           - name: nginx
             repo: nginx/nginx
             pull: 1333
-          - name: nginx
-            repo: nginx/nginx
-            pull: 1430
           - name: fancyindex
             repo: aperezdc/ngx-fancyindex
             pull: 176

.github/workflows/lint-and-format.yml

@@ -18,7 +18,7 @@ jobs:
     runs-on: ubuntu-slim
     steps:
       - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
-      - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
+      - uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
         with:
           node-version: lts/*
       - uses: pnpm/action-setup@ea17c68df8912ef543352723c149a84f56e3d413 # v6.1.0

.github/workflows/spellcheck.yml

@@ -22,4 +22,4 @@ jobs:
           # These animated SVGs embed raster frames as base64. Random binary
           # runs trip the dictionary, but there is no prose to spellcheck.
           skip: pnpm-lock.yaml,./frontend/translations,./frontend/public/images/crowdsec-logo-animated.svg,./docs/animated-logo.svg
-          ignore_words_list: afterAll,alog,clen
+          ignore_words_list: afterAll,alog,clen,hasTable

CHANGELOG.md

@@ -4,6 +4,8 @@ All notable changes to the NPMplus Security Fork are documented here. The fork u
 
 ## Unreleased
 
+- Reconcile upstream through October 16: migrate custom locations and their ordered access lists into relational storage, adopt host hard-deletion and mTLS auth-request controls, and update navigation and translations. Keep the fork's session revocation, replay protection, ACL rollback, bounded fetches, setup gate, certificate pipeline, and reviewed dependency versions. Adapt Anubis coverage and ACL recovery to the new relations, with a populated-database upgrade and snapshot regression test. Template version 23 regenerates existing nginx configurations. Database rollback requires restoring the matching pre-upgrade snapshot.
+
 - Refresh nginx to 1.31.7 with matching OpenResty patches and njs 1.0.2. Update the Caddy, CrowdSec, and Anubis security builds to Go 1.26.9 and patched networking dependencies; update CrowdSec's OpenAPI parser and Caddy to 2.11.7. Override vulnerable proxy-addr and source-map-js transitive dependencies. Reassess the pinned pip vendor inventory and scope the remaining packaging-tool exceptions to exact package versions until October 23, preserving runtime Certbot DNS-plugin installation.
 
 - Installer v1.69 adds the admin-only **Disable CRS observation** action under CrowdSec → WAF, the mirror of the v1.66 enable button: it removes the observation policy, restores the default acquisition list, strips the bouncer marker, and leaves the installed CRS rules inert while high-confidence blocking keeps running. Both verbs now write a persistent pre-change snapshot to `/var/lib/npmplus/crs-control/snapshot/` — kept after success and after rollback failure as the known-good instance for manual recovery — and the host helper gained a `DISABLE` socket verb behind the same audit, cooldown, and transactional rollback as enable. Uninstall removes the snapshot with the rest of the helper state.

FORK.md

@@ -61,6 +61,34 @@ and a fake GitHub CLI; no real branch or PR is created by the tests.
 
 ## Reviewing an upstream update
 
+The October 16 reconciliation incorporates upstream `5ba8fe1c` while retaining
+the fork's reviewed dependency versions (including Objection 3 during the new
+major release's quarantine), database-backed session/MFA replay guards, and
+certificate implementation. Custom locations now use `npmplus_location` and
+ordered ACL relations. Fork reporting and ACL rollback must fetch those relations;
+reading the legacy `proxy_host.locations` JSON column loses the new data.
+
+The hard-delete and location migrations cannot migrate down. Roll back with the
+safe updater's matching database/files snapshot and image, rather than only
+switching image tags. `backend/test/upstream-location-upgrade.test.js` reconstructs
+a populated pre-migration fork database, verifies ordered ACLs and inherited
+Anubis coverage after upgrade, and checks the pre-upgrade SQLite snapshot. Its
+coverage assertion was first observed failing on the unadapted query.
+
+The fork's fresh-session helper remains asynchronous: MFA enrollment and backup
+code regeneration must await it before setting the replacement cookie. The HTTP
+regression reproduced enrollment's 500 response before restoring both awaits.
+ACL form validation continues to accept explicit `deny all` policies; the browser
+regression first timed out waiting for a save request under upstream's empty-policy
+check, then verified the exact deny-all payload after the correction.
+
+Certificate renewal and downloads remain under the owner's existing restriction:
+do not import upstream's certificate restructuring without deliberate replacement
+pins. The UI continues to offer downloads only for Certbot certificates. Custom
+auth-request upstreams retain the fork's URL-only validation contract; upstream's
+named-upstream extension needs a separate privilege review. These are deliberate
+integration boundaries, not conflicts to erase in a later automated sync.
+
 From a clean checkout, fetch and inspect before changing the working branch:
 
 ```bash

backend/.smoke/access-list-ui.mjs

@@ -85,8 +85,26 @@ try {
 	assert.equal(await dialog.getByRole("button", { name: "Save", exact: true }).isEnabled(), true);
 	await dialog.getByRole("button", { name: "Cancel", exact: true }).click();
 	await dialog.waitFor({ state: "hidden" });
+	fixture.items = [];
+	fixture.clients = [{ directive: "deny", address: "all" }];
+	await page.reload({ waitUntil: "networkidle" });
+	await row.locator("button.dropdown-toggle").click();
+	await row.getByRole("button", { name: "Edit", exact: true }).click();
+	const denyAllSave = page.waitForResponse(
+		(response) =>
+			response.request().method() === "PUT" && response.url().includes("/api/nginx/access-lists/900001"),
+	);
+	await dialog.getByRole("button", { name: "Save", exact: true }).click();
+	const denyAllResponse = await denyAllSave;
+	assert.equal(denyAllResponse.status(), 400, "deny-all policy must reach the API");
+	assert.deepEqual(denyAllResponse.request().postDataJSON().clients, [{ directive: "deny", address: "all" }]);
+	await dialog.getByText(message, { exact: true }).waitFor();
+	await dialog.getByRole("button", { name: "Cancel", exact: true }).click();
+	await dialog.waitFor({ state: "hidden" });
 	assert.deepEqual(errors, [], "rejected creates must not raise a rollback TypeError");
-	console.log("PASS access-list browser validation, masked credentials, and rejected-create recovery");
+	console.log(
+		"PASS access-list browser validation, masked credentials, deny-all policy and rejected-create recovery",
+	);
 } finally {
 	await browser.close();
 }

backend/.smoke/modal-ui.mjs

@@ -137,6 +137,10 @@ try {
 		redirection_hosts: [],
 		dead_hosts: [],
 		streams: [],
+		mtls_proxy_hosts: [],
+		mtls_redirection_hosts: [],
+		mtls_dead_hosts: [],
+		mtls_streams: [],
 	};
 	let renewalRequests = 0;
 	await page.route(certificateRoutes, async (route) => {

backend/biome.json

@@ -42,6 +42,7 @@
 			"style": {
 				"noContinue": "off",
 				"noDefaultExport": "off",
+				"noExcessiveClassesPerFile": "off",
 				"noExcessiveLinesPerFile": "off",
 				"noIncrementDecrement": "off",
 				"noMagicNumbers": "off",

backend/certbot/dns-plugins.json

@@ -168,7 +168,7 @@
     "package_name": "certbot-dns-easydns"
   },
   "edgedns": {
-    "credentials": "edgedns_client_secret = as3d1asd5d1a32sdfsdfs2d1asd5=\nedgedns_host = sdflskjdf-dfsdfsdf-sdfsdfsdf.luna.akamaiapis.net\nedgedns_access_token = kjdsi3-34rfsdfsdf-234234fsdfsdf\nedgedns_client_token = dkfjdf-342fsdfsd-23fsdfsdfsdf",
+    "credentials": "client_token = YOUR_CLIENT_TOKEN\naccess_token = YOUR_ACCESS_TOKEN\nclient_secret = YOUR_CLIENT_SECRET\nhost = YOUR_HOST.luna.akamaiapis.net\n# Optional: account_key = YOUR_ACCOUNT_KEY",
     "full_plugin_name": "edgedns",
     "name": "Akamai Edge DNS",
     "package_name": "certbot-plugin-edgedns"

backend/db.js

@@ -1,15 +1,9 @@
 import knex from "knex";
-import { configGet, configHas } from "./lib/config.js";
+import { configGet } from "./lib/config.js";
 
 let instance = null;
 
 const generateDbConfig = () => {
-	if (!configHas("database")) {
-		throw new Error(
-			"Database config does not exist! Please read the instructions: https://github.com/ZoeyVid/NPMplus",
-		);
-	}
-
 	const cfg = configGet("database");
 
 	if (cfg.engine === "knex-native") {
@@ -33,9 +27,7 @@ const generateDbConfig = () => {
 };
 
 const getInstance = () => {
-	if (!instance) {
-		instance = knex(generateDbConfig());
-	}
+	instance ??= knex(generateDbConfig());
 	return instance;
 };
 

backend/internal/access-list.js

@@ -7,6 +7,7 @@ import { access as logger } from "../logger.js";
 import accessListModel from "../models/access_list.js";
 import accessListAuthModel from "../models/access_list_auth.js";
 import accessListClientModel from "../models/access_list_client.js";
+import locationModel from "../models/location.js";
 import proxyHostModel from "../models/proxy_host.js";
 import internalAuditLog from "./audit-log.js";
 import internalNginx from "./nginx.js";
@@ -106,7 +107,11 @@ const internalAccessList = {
 			access.can("access_lists:manage");
 			const row = await internalAccessList.get(access, {
 				id: data.id,
-				expand: ["items", "clients", "proxy_hosts.[certificate,access_lists.[clients,items]]"],
+				expand: [
+					"items",
+					"clients",
+					"proxy_hosts.[certificate,access_lists.[clients,items],locations.access_lists.[clients,items]]",
+				],
 			});
 			const existingUsernames = new Set((row.items || []).map((item) => item.username));
 			if ((data.items || []).some((item) => !item.password && !existingUsernames.has(item.username))) {
@@ -154,7 +159,11 @@ const internalAccessList = {
 						access,
 						{
 							id: data.id,
-							expand: ["items", "clients", "proxy_hosts.[certificate,access_lists.[clients,items]]"],
+							expand: [
+								"items",
+								"clients",
+								"proxy_hosts.[certificate,access_lists.[clients,items],locations.access_lists.[clients,items]]",
+							],
 						},
 						trx,
 					);
@@ -235,7 +244,9 @@ const internalAccessList = {
 			.where("access_list.is_deleted", 0)
 			.andWhere("access_list.id", thisData.id)
 			.groupBy("access_list.id")
-			.allowGraph("[items,clients,proxy_hosts.[certificate,access_lists.[clients,items]]]")
+			.allowGraph(
+				"[items,clients,proxy_hosts.[certificate,access_lists.[clients,items],locations.access_lists.[clients,items]]]",
+			)
 			.first();
 
 		if (access.visibility !== "all") {
@@ -252,6 +263,9 @@ const internalAccessList = {
 			throw new errs.ItemNotFoundError(thisData.id);
 		}
 
+		if (Array.isArray(row.proxy_hosts)) {
+			row.proxy_hosts = [...new Map(row.proxy_hosts.map((host) => [host.id, host])).values()];
+		}
 		return row;
 	},
 
@@ -266,53 +280,36 @@ const internalAccessList = {
 		access.can("access_lists:manage");
 		const row = await internalAccessList.get(access, {
 			id: data.id,
-			expand: ["proxy_hosts.[certificate, access_lists.[clients,items]]", "items", "clients"],
+			expand: [
+				"proxy_hosts.[certificate, access_lists.[clients,items], locations.access_lists.[clients,items]]",
+				"items",
+				"clients",
+			],
 		});
-
-		if (!row?.id) {
-			throw new errs.ItemNotFoundError(data.id);
-		}
 		// 1. update row to be deleted
 		// 2. update any proxy hosts that were using it (ignoring permissions)
 		// 3. reconfigure those hosts
 		// 4. audit log
 
-		// 1. update row to be deleted
-		await accessListModel.query().where("id", row.id).patch({
-			is_deleted: 1,
-		});
-
-		// 2. update any proxy hosts that were using it (ignoring permissions)
 		const affectedHosts = (row.proxy_hosts || []).map((host) => {
 			const updatedHost = { ...host };
-			// check in case something crazy happened. This should never be the case, but safeguard
-			if (!Array.isArray(updatedHost.npmplus_access_list_ids)) {
-				updatedHost.npmplus_access_list_ids = [];
-			}
-			updatedHost.npmplus_access_list_ids = updatedHost.npmplus_access_list_ids.filter((id) => id !== row.id);
-
-			// update the access_lists object (separate from the access_lists_ids)
+			// update the access_lists object
 			if (!Array.isArray(updatedHost.access_lists)) {
 				updatedHost.access_lists = [];
 			}
 			updatedHost.access_lists = updatedHost.access_lists.filter((acl) => acl.id !== row.id);
 
-			if (updatedHost.npmplus_access_list_ids.length === 0) {
+			if (updatedHost.access_lists.length === 0) {
 				updatedHost.npmplus_access_list_type = "public";
 			}
 			if (!Array.isArray(updatedHost.locations)) {
 				throw new errs.ConfigurationError("Invalid location structure. Expected an array");
 			}
 			updatedHost.locations = updatedHost.locations.map((location) => {
 				const updatedLocation = { ...location };
-				if (!Array.isArray(updatedLocation.npmplus_access_list_ids)) {
-					updatedLocation.npmplus_access_list_ids = [];
-				}
-				updatedLocation.npmplus_access_list_ids = updatedLocation.npmplus_access_list_ids.filter(
-					(id) => id !== row.id,
-				);
+				updatedLocation.access_lists = updatedLocation.access_lists.filter((acl) => acl.id !== row.id);
 				if (
-					updatedLocation.npmplus_access_list_ids.length === 0 &&
+					updatedLocation.access_lists.length === 0 &&
 					updatedLocation.npmplus_access_list_type === "custom"
 				) {
 					updatedLocation.npmplus_access_list_type = "global";
@@ -323,57 +320,43 @@ const internalAccessList = {
 		});
 		const deletedRow = { ...row, proxy_hosts: undefined };
 
-		const finalize = async () => {
-			// 4. audit log
-			await internalAuditLog.add(access, {
-				action: "deleted",
-				object_type: "access-list",
-				object_id: row.id,
-				meta: deletedRow,
-			});
-		};
-		try {
-			// 3. Write the changes to the database and the config
-			if (affectedHosts.length > 0) {
-				await proxyHostModel.transaction(async (trx) => {
-					await Promise.all(
-						affectedHosts.map(async (host) => {
-							await proxyHostModel.query(trx).patchAndFetchById(host.id, {
-								npmplus_access_list_ids: host.npmplus_access_list_ids,
-								npmplus_access_list_type: host.npmplus_access_list_type,
-								locations: host.locations,
-							});
-
-							return internalProxyHostAccessList.syncAccessListRelations(trx, host.id, host);
-						}),
-					);
+		await accessListModel.transaction(async (trx) => {
+			// 1. update row to be deleted
+			await trx("npmplus_proxy_host_access_list").where("access_list_id", row.id).delete();
+			await accessListAuthModel.query(trx).where("access_list_id", row.id).delete();
+			await accessListClientModel.query(trx).where("access_list_id", row.id).delete();
+			await accessListModel.query(trx).deleteById(row.id);
+			// 2. update any proxy hosts that were using it (ignoring permissions)
+			for (const host of affectedHosts) {
+				await proxyHostModel.query(trx).findById(host.id).patch({
+					npmplus_access_list_type: host.npmplus_access_list_type,
 				});
-				row.proxy_hosts = affectedHosts;
-				// step 4. Regenerate configs and htpasswd files
-				// locations don't have accessList objects, only IDs, so populate it with the object itself
-				row.proxy_hosts = await Promise.all(
-					(row.proxy_hosts || []).map((host) => {
-						const cleanedHost = internalProxyHostAccessList.cleanAccessListTypes(host);
-						return internalProxyHostAccessList.populateLocationAccessLists(cleanedHost);
-					}),
-				);
-				await internalNginx.bulkGenerateConfigs(proxyHostModel, "proxy_host", row.proxy_hosts);
+				for (const location of host.locations) {
+					await locationModel.query(trx).where("id", location.id).patch({
+						npmplus_access_list_type: location.npmplus_access_list_type,
+					});
+				}
 			}
+		});
+
+		try {
+			// 3. Write the changes to the config
+			// Regenerate configs and htpasswd files
+			for (const host of affectedHosts) await internalNginx.configure(proxyHostModel, "proxy_host", host);
 
 			await internalNginx.reload();
 
 			// delete the htpasswd file
 			await rm(internalAccessList.getFilename(row), { force: true });
-		} catch (operationError) {
-			// the audit write must never replace the operation's own error
-			try {
-				await finalize();
-			} catch (cleanupError) {
-				logger.error(`Error auditing access list delete ${row.id}: ${cleanupError.message}`);
-			}
-			throw operationError;
+		} finally {
+			// 4. audit log
+			await internalAuditLog.add(access, {
+				action: "deleted",
+				object_type: "access-list",
+				object_id: row.id,
+				meta: deletedRow,
+			});
 		}
-		await finalize();
 		return deletedRow;
 	},
 

backend/internal/anubis-reporting.js

@@ -155,7 +155,8 @@ const hostCoverage = async (page = 1) => {
 	const query = ProxyHost.query().where("is_deleted", 0).where("enabled", 1);
 	const count = await query.clone().resultSize();
 	const hosts = await query
-		.select("id", "domain_names", "npmplus_auth_request", "npmplus_auth_request_upstream", "locations")
+		.select("id", "domain_names", "npmplus_auth_request", "npmplus_auth_request_upstream")
+		.withGraphFetched("locations")
 		.orderBy("id")
 		.limit(25)
 		.offset((page - 1) * 25);