Skip to content

fix: audit leftovers (live-alert batches, DM ticket lookups, event image cache, restore role warning) - #367

Open
Ferdinand99 wants to merge 22 commits into
v5.3from
fix/53-audit-leftovers
Open

Ferdinand99 wants to merge 22 commits into
v5.3from
fix/53-audit-leftovers

Conversation

@Ferdinand99

Copy link
Copy Markdown
Owner

What

Leftovers from the 5.2 security audit (the four "needs validation" leads), one commit each, each with a regression test.

  • Live alerts: one Twitch login or Kick slug that the platform rejects with a 4xx no longer stops alerts for every server. A rejected batch is bisected down to the offending value, which is quarantined with a growing backoff (10 min up to 6 h) and logged with the servers that saved it. 5xx, 401, 429 and a poll where many different values are rejected (a request-level problem) still fail the poll.
  • DM tickets: a DM no longer fetches the sender as a member in every ticket-enabled server. The member cache is used first, a server whose member list is fully cached answers without a request (Server Members intent on), REST fetches are capped at 8 per DM and 120 per minute for the bot, "not a member" is remembered for 2 minutes, servers where the sender was found before go first, and a sender's answer is reused for 10 seconds. Without the Server Members intent senders are still found by the capped fetches; if a message could not check every server yet, the sender is asked to send it again, which continues with the unchecked servers.
  • Event page images: the cache gives each server a fair share (4 MiB, 16 entries), evicts the biggest holder first when full, and limits downloads to 20 per server per minute and 8 at once. Failures were already cached and concurrent requests for one image already shared one download.
  • pg_restore (hardening): startup warning when the database role is a superuser or has CREATEROLE, CREATEDB or the server-program/file memberships; the Health page shows the same; restoring an uploaded dump asks for a confirmation that says its SQL runs with the rights of the database role; docs/postgres.md describes a least-privilege role. Restores are not blocked.
  • Tests: the two requireTicketAccess tests dropped in 5.2 are ported to the current guard (ticketAccessLevel in the v2 tickets router).

Not covered

  • The Twitch and Kick answers for malformed or unknown logins (third-party behaviour) were not checked against the live APIs; the fix is correct whichever way they answer.
  • Per-DM lookups can still cost up to 8 fetches for a sender with no cached membership; that is the intended bound.
  • Whether the hosted database role is privileged is a deployment fact; the new warning will say.

Tests

npm test on SQLite and on Postgres 16, npm run lint, prettier, npm run test:v2 and npm run build:v2 pass.

…he shared poll

One login or slug that the platform answers with a 4xx used to fail the whole
batch, so no server got go-live posts. A rejected batch is now bisected down to
the offending value, which is quarantined with a growing backoff (10 min up to
6 h) and logged with the servers that saved it. The other values keep polling.
5xx, 401 and 429 still fail the poll, and so does a poll where many different
values are rejected (a request-level problem such as bad app credentials).
A DM used to fetch the sender as a member in every ticket-enabled server of the
instance. The lookup now uses the member cache first (a server whose member
list is fully cached answers with no request), caps REST fetches at 8 per DM
and 120 per minute for the bot, remembers "not a member" for two minutes,
tries servers where the user was found before first, and reuses a sender's
answer for 10 seconds. Without the Server Members intent senders are still
found by the capped fetches; a message that could not check every server yet
asks the sender to try again, which continues with the unchecked servers.
The public event page image cache was one shared pool, so a server with more
large images than fit could evict every other server's images and make each
page view download again. Entries now belong to the server whose page asked for
them: a server holds at most 4 MiB and 16 entries, its own oldest entries go
first, and when the whole cache is full the biggest holder gives way. A server
may start at most 20 downloads a minute and the instance 8 at once; a request
over a limit gets no image and starts no download. Failures were already
remembered and concurrent requests for one image already shared one download.
…e restoring an uploaded dump

pg_restore runs every statement of a restored dump as the database role in
DATABASE_URL. Sylo cannot change that role, so:
- at start-up on Postgres it reads the role and logs one warning when it is a
  superuser or has CREATEROLE, CREATEDB or one of the pg_execute_server_program,
  pg_read_server_files and pg_write_server_files memberships, with a link to
  the docs;
- /api/v2/health reports the same risks (db.roleRisks, db.roleDocsUrl) and the
  Health page shows a note;
- restoring an uploaded (imported) dump asks for confirmation that says its SQL
  runs with the rights of the database role, and importing a dump on Postgres
  says the same. Restores are not blocked;
- docs/postgres.md describes a least-privilege role setup and how to check one.
requireTicketAccess was removed with the classic dashboard, and its two tests in
classicBanner.test.js with it. The same guarantees are now tested against the
current guard (ticketAccessLevel, run first by the v2 tickets router): a person
without access gets a plain 403 with no page and no data, and a failed access
lookup reaches the central error handler (a 500 without detail) instead of
hanging the request.

@graphify-labs graphify-labs Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify reviewed this change.

Worth a look — the grounded gate found no coupling regressions or blocking issues, but 3 advisory finding(s) below merit a look before merge.

Not checked: tests were not run; no formal proof of the changed code.


Graphify review — findings

Adds a capped ticket-server lookup that checks the member cache first, then asks Discord for the single member, at most 8 servers per message and 120 requests a minute bot-wide, and remembers "not a member" for two minutes. When the cap leaves servers unchecked, the DM reply asks the member to resend in a few seconds instead of claiming no server is set up. Adds a start-up check of the DATABASE_URL Postgres role that logs one warning for SUPERUSER, CREATEROLE, CREATEDB or the three file/program memberships, mirrors it on the Health page, and documents a least-privilege role without ever blocking a restore. Also caps rejected items in isolated batch fetches, applies a per-owner quota to cached event images, and bounds v2 API request bodies.

Worth a look

  • Rejected-item cap lets one server's bad logins break alert polling for every server — src/modules/lib/batchIsolation.js:78 · Escalate · medium
    • NOT verified (no proof, no reproducing execution) — needs human review
  • Incomplete lookup is cached, so a quick resend repeats the same partial answer — src/modules/tickets.js:278 · Escalate · medium
    • NOT verified (no proof, no reproducing execution) — needs human review
  • One-argument loadImage calls are now capped by the per-owner quota — src/modules/lib/eventImage.js:146 · Escalate · medium
    • NOT verified (no proof, no reproducing execution) — needs human review

Review partial — this diff was larger than one review pass covers, so later files were not reviewed; some findings may be missing.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 228 functions depend on the 211 functions this change touches.

Health — this change adds coupling hotspots:

  • worse: register() — 9 callers, 5 callees
  • worse: loadImage() — 5 callers, 8 callees
  • new: canOpenTicketIn() — 3 callers, 9 callees
  • worse: Health() — 2 callers, 12 callees
  • worse: tick() — 1 callers, 20 callees
  • worse: tick() — 1 callers, 18 callees
  • new: lookupTicketGuilds() — 4 callers, 4 callees
  • worse: main() — 1 callers, 15 callees
  • …and 1 more — each is listed as a finding

Verification — 228 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

  • verification_scope: 228 function(s) in the blast radius were not formally verified this run

Test selection

Test selection

679 of 679 test file(s) selected (100%) via static blast radius.

Escalated to a full run for safety (noncode-change, uncovered-code) — the selection is not trustworthy on its own (see below). CI should run the whole suite.

  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Access/DashboardAccessTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Access/Round3Tests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Account/AccountConnectTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Account/AccountModelTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Account/ConnectCallbackSourceTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Account/NoDashboardLinkSourceTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Appeals/AppealsReviewTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Applications/ApplicationFormsTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Applications/ApplicationPassthroughTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Applications/ApplicationSubmissionsTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Applications/ApplicationWireTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Audit/AuditHistoryTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Audit/AuditLogTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Auth/CallbackIssuerAddressTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Auth/CallbackIssuerTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Auth/KeychainTokenStoreTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Auth/PKCETests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Auth/SignInRequestTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Auth/TokenStoreTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Changelog/ChangelogTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Commands/CommandOverridesTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Economy/CollectiblePictureTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Economy/EconomyAdminTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Economy/EconomyMembersTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Economy/InventoryTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Economy/ShopExtrasTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Economy/ShopItemsTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Economy/ShopModelTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Embeds/EmbedDraftTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Embeds/ModuleLimitsTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Events/CommunityReviewFixesTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Events/EventEndpointsTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Giveaways/GiveawayActionsTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/GuildSettings/CommandPrefixTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/GuildSettings/GuildSettingsTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Health/HealthTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Import/P6Tests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Insights/InsightsTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Leaderboard/LeaderboardTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Members/MemberDataTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Members/MemberSearchTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Messages/ComposedMessagesModelTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Messages/ComposedMessagesTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Models/ModelDecodingTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Models/ServerListTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Models/Sylo52PolishTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Models/UnknownKeysTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Models/Utf16TextLimitTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/Moderation/ModerationTests.swift — full-run-safety
  • apps/sylo-ios/SyloKit/Tests/SyloKitTests/ModuleSettings/AfkSettingsTests.swift — full-run-safety
  • … and 629 more

non-code file(s) changed (docs/modules/tickets.md, docs/postgres.md, docs/self-hosting.md) → running the full suite for safety (a code graph can't see config/fixture/data deps)

changed code file(s) with no mapped test (docs/modules/tickets.md, docs/postgres.md, docs/self-hosting.md, src/index.js, src/web/routes/v2Api.js) — a coverage gap or a missing link — running the full suite rather than only the selected tests

Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.

…own evidence

The count rule (more than 10 rejected values means the request is at fault)
let one guild with 11 bad values bring the outage back, and quarantined every
good value on small instances when the platform answered 403 for a while.

- A value rejected on its own is at fault when at least one request of the poll
  succeeded, however many there are. When no request succeeded the request is at
  fault: nothing is quarantined and the poll fails as before.
- Only 400, 404 and 422 can mean a bad value; 403, 408, 409, 425 and the rest
  fail the poll.
- The cost is capped by a per-poll request budget; batches not reached are left
  out of that poll, not failed, and each poll quarantines what it found.
- The job loop leaves alone every value the poll did not resolve (quarantined,
  rejected, over budget) instead of asking the quarantine again later, which
  could end a live post by mistake.
- One aggregated warning per poll, only for newly rejected values.
- The user lookup of Twitch no longer fails the poll.
- The servers where the member has an open ticket come from the database (new
  openTicketGuildKeys, indexed on Postgres) and are checked first, outside the
  caps, so a follow-up reaches its ticket after a restart, with the intent off
  and in large servers.
- A lookup that the caps cut short never opens a new ticket: the member is asked
  to send again. Only the single open ticket (a complete list) is routed to.
- Per-user budget (16 fetches a minute) next to the bot's 120; open-ticket checks
  and the server pick are not counted against the shared budget.
- An incomplete answer is not reused, so a retry makes progress; servers are
  asked in the order not asked yet / longest ago, and the not-a-member answer is
  remembered for 10 minutes, so the first minute's answers do not expire before
  the last servers are reached.
- A server that refuses member lookups (403) is skipped for that time instead of
  keeping the member waiting.
- A refused server pick keeps the stashed message and the menu.
When the whole image cache was over its entry count, the holder with the most
bytes gave up its entry, so tiny or failed entries of other servers could keep a
real server from holding more than one image. The holder with the most entries
now gives way when the entry limit is hit, and the one with the most bytes when
the byte limit is hit.
…peruser membership

- POST /api/v2/health/backups/:name/restore: a confirm that is sent must be the
  snapshot name. An uploaded (sylo-imported-*) dump on Postgres needs
  { confirm: <name>, acknowledgeSql: true }, else 400. Other snapshots need no
  body, so the Sylo app, which sends none, keeps working. The dashboard sends
  both after its warning.
- The role check also reports membership of a superuser role (SET ROLE), and
  falls back to the shorter query only for an unknown role or function, never on
  other errors.
- The docs link names the docs of the running version.
The central error handler is exported from server.js instead of being found as
the first four-argument layer, and the lookup failure is tested with a real
server instead of a missing params object.
…, check with a canary

- Phase one sends one request per batch, so a clean batch is never left out
  because poisoned values filled an earlier one, and each clean batch is
  evidence that the request is fine.
- Phase two halves only the failed batches, in turn, within its own budget.
  Running out of budget leaves the unreached items for the next poll and never
  fails the poll. The start of the list rotates, so the same servers are not
  always last.
- Before values are quarantined, one request with a value accepted in an
  earlier poll tells a refused value from a refusing platform (outage,
  flapping): if it is refused too, nothing is quarantined, and the poll fails
  only when nothing succeeded. With nothing known to be good yet and nothing
  succeeding, nothing is quarantined and the poll says so.
…airer

- The open ticket step honours the remembered not-a-member answers, has its own
  cap (5 fetches per DM, 10 per user per minute, separate from the lookup
  budget), and remembers a member who left such a server for ten minutes. Past
  the cap the open list is marked incomplete, which makes the handler ask
  again. A menu pick honours the same memory and the same allowance, so
  clicking the menu over and over costs one fetch.
- The last 30 of the shared 120 fetches a minute are kept for users who have
  used fewer than 4 of their own.
- Not a member is remembered for three minutes again (a member join clears it
  where Sylo sees the join); only a refused lookup (403 / 50001) is remembered
  for ten.
…estore rules through the route

The role warning links docs/postgres.md on the default branch, which has the
section from the merge on, instead of a release tag that may not. A Postgres-only
route test checks the confirmation for an uploaded dump: every missing or wrong
field is a 400, and with both fields the request reaches the dump check without
restoring anything.
…es progress

Breadth-first halving spent the whole budget on the first levels when several
batches were full of rejected values, so no value was ever isolated, nothing was
quarantined and the cost stayed high for good. The search is now depth first:
one failed batch is searched down to single values before the next is started,
so each poll quarantines about 60 dense or 8 sparse values. Chunks left
unchecked when the budget ends are remembered as suspects and batched apart
from the clean values in the next poll, so the clean ones are always fetched.
One warning per poll, with counts and no values, says when the search ran out.
A 429 or 5xx on the canary request counts as a refused canary: the poll keeps
what it fetched and only skips the quarantine.
… remembered 'left'

When the server of an open ticket was skipped because an earlier answer said the
member had left (they may have come back, and without the Server Members intent
no join event arrives), a follow-up that finds exactly one other server now gets
the picker instead of a new ticket there.
…the canary before the search

- A server is flagged for two hours after one of its values is found rejected;
  its values are batched on their own, never with other servers' values, so a
  server that saves fresh rejected values every poll cannot keep others
  unresolved. Servers with no rejected value still share batches, which keeps
  the cost at one request per hundred values.
- When no batch answers, the canary is asked before the search budget is spent,
  and a refused canary fails the poll at once.
- The comment describes the order the batches are cut and searched in.
…ick ask again

When the picker shows because an open ticket server was skipped from memory, that
server is one of the options. Picking a server where the member has an open
ticket asks Discord once even if it is remembered as left, and a fresh member
answer clears the memory.
The iOS app detects restore-confirmation support (confirm + acknowledgeSql) by db.roleRisks being an array in the health report. Pin that contract with a route test and note it in the mobile app docs.
…as a warning in the restore dialog

The role warning moves to the top of Database & backups and says in one line
when the role is a superuser, with the full list of rights folded behind
Details. In the restore dialog the SQL warning for an uploaded file is a
warning callout with the same short wording, and the link out of the dialog
is gone (the page callout has it).
The pick was resolved on any platform and checked with the picked server's
client. The menu only offers servers of the DM's platform and select values
are validated upstream, so this is defence in depth: a value from another
platform is refused like any pick that cannot be confirmed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant