Repository navigation
fix: audit leftovers (live-alert batches, DM ticket lookups, event image cache, restore role warning) - #367
Ferdinand99 wants to merge 22 commits into
Conversation
…he shared poll One login or slug that the platform answers with a 4xx used to fail the whole batch, so no server got go-live posts. A rejected batch is now bisected down to the offending value, which is quarantined with a growing backoff (10 min up to 6 h) and logged with the servers that saved it. The other values keep polling. 5xx, 401 and 429 still fail the poll, and so does a poll where many different values are rejected (a request-level problem such as bad app credentials).
A DM used to fetch the sender as a member in every ticket-enabled server of the instance. The lookup now uses the member cache first (a server whose member list is fully cached answers with no request), caps REST fetches at 8 per DM and 120 per minute for the bot, remembers "not a member" for two minutes, tries servers where the user was found before first, and reuses a sender's answer for 10 seconds. Without the Server Members intent senders are still found by the capped fetches; a message that could not check every server yet asks the sender to try again, which continues with the unchecked servers.
The public event page image cache was one shared pool, so a server with more large images than fit could evict every other server's images and make each page view download again. Entries now belong to the server whose page asked for them: a server holds at most 4 MiB and 16 entries, its own oldest entries go first, and when the whole cache is full the biggest holder gives way. A server may start at most 20 downloads a minute and the instance 8 at once; a request over a limit gets no image and starts no download. Failures were already remembered and concurrent requests for one image already shared one download.
…e restoring an uploaded dump pg_restore runs every statement of a restored dump as the database role in DATABASE_URL. Sylo cannot change that role, so: - at start-up on Postgres it reads the role and logs one warning when it is a superuser or has CREATEROLE, CREATEDB or one of the pg_execute_server_program, pg_read_server_files and pg_write_server_files memberships, with a link to the docs; - /api/v2/health reports the same risks (db.roleRisks, db.roleDocsUrl) and the Health page shows a note; - restoring an uploaded (imported) dump asks for confirmation that says its SQL runs with the rights of the database role, and importing a dump on Postgres says the same. Restores are not blocked; - docs/postgres.md describes a least-privilege role setup and how to check one.
requireTicketAccess was removed with the classic dashboard, and its two tests in classicBanner.test.js with it. The same guarantees are now tested against the current guard (ticketAccessLevel, run first by the v2 tickets router): a person without access gets a plain 403 with no page and no data, and a failed access lookup reaches the central error handler (a 500 without detail) instead of hanging the request.
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 3 advisory finding(s) below merit a look before merge.
Not checked: tests were not run; no formal proof of the changed code.
Graphify review — findings
Adds a capped ticket-server lookup that checks the member cache first, then asks Discord for the single member, at most 8 servers per message and 120 requests a minute bot-wide, and remembers "not a member" for two minutes. When the cap leaves servers unchecked, the DM reply asks the member to resend in a few seconds instead of claiming no server is set up. Adds a start-up check of the DATABASE_URL Postgres role that logs one warning for SUPERUSER, CREATEROLE, CREATEDB or the three file/program memberships, mirrors it on the Health page, and documents a least-privilege role without ever blocking a restore. Also caps rejected items in isolated batch fetches, applies a per-owner quota to cached event images, and bounds v2 API request bodies.
Worth a look
- Rejected-item cap lets one server's bad logins break alert polling for every server —
src/modules/lib/batchIsolation.js:78· Escalate · medium- NOT verified (no proof, no reproducing execution) — needs human review
- Incomplete lookup is cached, so a quick resend repeats the same partial answer —
src/modules/tickets.js:278· Escalate · medium- NOT verified (no proof, no reproducing execution) — needs human review
- One-argument loadImage calls are now capped by the per-owner quota —
src/modules/lib/eventImage.js:146· Escalate · medium- NOT verified (no proof, no reproducing execution) — needs human review
Review partial — this diff was larger than one review pass covers, so later files were not reviewed; some findings may be missing.
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 228 functions depend on the 211 functions this change touches.
Health — this change adds coupling hotspots:
- worse:
register()— 9 callers, 5 callees - worse:
loadImage()— 5 callers, 8 callees - new:
canOpenTicketIn()— 3 callers, 9 callees - worse:
Health()— 2 callers, 12 callees - worse:
tick()— 1 callers, 20 callees - worse:
tick()— 1 callers, 18 callees - new:
lookupTicketGuilds()— 4 callers, 4 callees - worse:
main()— 1 callers, 15 callees - …and 1 more — each is listed as a finding
Verification — 228 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 228 function(s) in the blast radius were not formally verified this run
Test selection
Test selection
679 of 679 test file(s) selected (100%) via static blast radius.
Escalated to a full run for safety (
noncode-change,uncovered-code) — the selection is not trustworthy on its own (see below). CI should run the whole suite.
apps/sylo-ios/SyloKit/Tests/SyloKitTests/Access/DashboardAccessTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Access/Round3Tests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Account/AccountConnectTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Account/AccountModelTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Account/ConnectCallbackSourceTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Account/NoDashboardLinkSourceTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Appeals/AppealsReviewTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Applications/ApplicationFormsTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Applications/ApplicationPassthroughTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Applications/ApplicationSubmissionsTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Applications/ApplicationWireTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Audit/AuditHistoryTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Audit/AuditLogTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Auth/CallbackIssuerAddressTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Auth/CallbackIssuerTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Auth/KeychainTokenStoreTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Auth/PKCETests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Auth/SignInRequestTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Auth/TokenStoreTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Changelog/ChangelogTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Commands/CommandOverridesTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Economy/CollectiblePictureTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Economy/EconomyAdminTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Economy/EconomyMembersTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Economy/InventoryTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Economy/ShopExtrasTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Economy/ShopItemsTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Economy/ShopModelTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Embeds/EmbedDraftTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Embeds/ModuleLimitsTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Events/CommunityReviewFixesTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Events/EventEndpointsTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Giveaways/GiveawayActionsTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/GuildSettings/CommandPrefixTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/GuildSettings/GuildSettingsTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Health/HealthTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Import/P6Tests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Insights/InsightsTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Leaderboard/LeaderboardTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Members/MemberDataTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Members/MemberSearchTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Messages/ComposedMessagesModelTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Messages/ComposedMessagesTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Models/ModelDecodingTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Models/ServerListTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Models/Sylo52PolishTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Models/UnknownKeysTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Models/Utf16TextLimitTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/Moderation/ModerationTests.swift— full-run-safetyapps/sylo-ios/SyloKit/Tests/SyloKitTests/ModuleSettings/AfkSettingsTests.swift— full-run-safety- … and 629 more
non-code file(s) changed (
docs/modules/tickets.md,docs/postgres.md,docs/self-hosting.md) → running the full suite for safety (a code graph can't see config/fixture/data deps)
changed code file(s) with no mapped test (
docs/modules/tickets.md,docs/postgres.md,docs/self-hosting.md,src/index.js,src/web/routes/v2Api.js) — a coverage gap or a missing link — running the full suite rather than only the selected tests
Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.
…own evidence The count rule (more than 10 rejected values means the request is at fault) let one guild with 11 bad values bring the outage back, and quarantined every good value on small instances when the platform answered 403 for a while. - A value rejected on its own is at fault when at least one request of the poll succeeded, however many there are. When no request succeeded the request is at fault: nothing is quarantined and the poll fails as before. - Only 400, 404 and 422 can mean a bad value; 403, 408, 409, 425 and the rest fail the poll. - The cost is capped by a per-poll request budget; batches not reached are left out of that poll, not failed, and each poll quarantines what it found. - The job loop leaves alone every value the poll did not resolve (quarantined, rejected, over budget) instead of asking the quarantine again later, which could end a live post by mistake. - One aggregated warning per poll, only for newly rejected values. - The user lookup of Twitch no longer fails the poll.
- The servers where the member has an open ticket come from the database (new openTicketGuildKeys, indexed on Postgres) and are checked first, outside the caps, so a follow-up reaches its ticket after a restart, with the intent off and in large servers. - A lookup that the caps cut short never opens a new ticket: the member is asked to send again. Only the single open ticket (a complete list) is routed to. - Per-user budget (16 fetches a minute) next to the bot's 120; open-ticket checks and the server pick are not counted against the shared budget. - An incomplete answer is not reused, so a retry makes progress; servers are asked in the order not asked yet / longest ago, and the not-a-member answer is remembered for 10 minutes, so the first minute's answers do not expire before the last servers are reached. - A server that refuses member lookups (403) is skipped for that time instead of keeping the member waiting. - A refused server pick keeps the stashed message and the menu.
When the whole image cache was over its entry count, the holder with the most bytes gave up its entry, so tiny or failed entries of other servers could keep a real server from holding more than one image. The holder with the most entries now gives way when the entry limit is hit, and the one with the most bytes when the byte limit is hit.
…peruser membership
- POST /api/v2/health/backups/:name/restore: a confirm that is sent must be the
snapshot name. An uploaded (sylo-imported-*) dump on Postgres needs
{ confirm: <name>, acknowledgeSql: true }, else 400. Other snapshots need no
body, so the Sylo app, which sends none, keeps working. The dashboard sends
both after its warning.
- The role check also reports membership of a superuser role (SET ROLE), and
falls back to the shorter query only for an unknown role or function, never on
other errors.
- The docs link names the docs of the running version.
The central error handler is exported from server.js instead of being found as the first four-argument layer, and the lookup failure is tested with a real server instead of a missing params object.
…, check with a canary - Phase one sends one request per batch, so a clean batch is never left out because poisoned values filled an earlier one, and each clean batch is evidence that the request is fine. - Phase two halves only the failed batches, in turn, within its own budget. Running out of budget leaves the unreached items for the next poll and never fails the poll. The start of the list rotates, so the same servers are not always last. - Before values are quarantined, one request with a value accepted in an earlier poll tells a refused value from a refusing platform (outage, flapping): if it is refused too, nothing is quarantined, and the poll fails only when nothing succeeded. With nothing known to be good yet and nothing succeeding, nothing is quarantined and the poll says so.
…airer - The open ticket step honours the remembered not-a-member answers, has its own cap (5 fetches per DM, 10 per user per minute, separate from the lookup budget), and remembers a member who left such a server for ten minutes. Past the cap the open list is marked incomplete, which makes the handler ask again. A menu pick honours the same memory and the same allowance, so clicking the menu over and over costs one fetch. - The last 30 of the shared 120 fetches a minute are kept for users who have used fewer than 4 of their own. - Not a member is remembered for three minutes again (a member join clears it where Sylo sees the join); only a refused lookup (403 / 50001) is remembered for ten.
…estore rules through the route The role warning links docs/postgres.md on the default branch, which has the section from the merge on, instead of a release tag that may not. A Postgres-only route test checks the confirmation for an uploaded dump: every missing or wrong field is a 400, and with both fields the request reaches the dump check without restoring anything.
…es progress Breadth-first halving spent the whole budget on the first levels when several batches were full of rejected values, so no value was ever isolated, nothing was quarantined and the cost stayed high for good. The search is now depth first: one failed batch is searched down to single values before the next is started, so each poll quarantines about 60 dense or 8 sparse values. Chunks left unchecked when the budget ends are remembered as suspects and batched apart from the clean values in the next poll, so the clean ones are always fetched. One warning per poll, with counts and no values, says when the search ran out. A 429 or 5xx on the canary request counts as a refused canary: the poll keeps what it fetched and only skips the quarantine.
… remembered 'left' When the server of an open ticket was skipped because an earlier answer said the member had left (they may have come back, and without the Server Members intent no join event arrives), a follow-up that finds exactly one other server now gets the picker instead of a new ticket there.
…the canary before the search - A server is flagged for two hours after one of its values is found rejected; its values are batched on their own, never with other servers' values, so a server that saves fresh rejected values every poll cannot keep others unresolved. Servers with no rejected value still share batches, which keeps the cost at one request per hundred values. - When no batch answers, the canary is asked before the search budget is spent, and a refused canary fails the poll at once. - The comment describes the order the batches are cut and searched in.
…ick ask again When the picker shows because an open ticket server was skipped from memory, that server is one of the options. Picking a server where the member has an open ticket asks Discord once even if it is remembered as left, and a fresh member answer clears the memory.
The iOS app detects restore-confirmation support (confirm + acknowledgeSql) by db.roleRisks being an array in the health report. Pin that contract with a route test and note it in the mobile app docs.
…as a warning in the restore dialog The role warning moves to the top of Database & backups and says in one line when the role is a superuser, with the full list of rights folded behind Details. In the restore dialog the SQL warning for an uploaded file is a warning callout with the same short wording, and the link out of the dialog is gone (the page callout has it).
The pick was resolved on any platform and checked with the picked server's client. The menu only offers servers of the DM's platform and select values are validated upstream, so this is defence in depth: a value from another platform is refused like any pick that cannot be confirmed.
What
Leftovers from the 5.2 security audit (the four "needs validation" leads), one commit each, each with a regression test.
requireTicketAccesstests dropped in 5.2 are ported to the current guard (ticketAccessLevelin the v2 tickets router).Not covered
Tests
npm teston SQLite and on Postgres 16,npm run lint, prettier,npm run test:v2andnpm run build:v2pass.