Skip to content
2 changes: 1 addition & 1 deletion docs/fluxer.md
Original file line number Diff line number Diff line change
Expand Up @@ -308,7 +308,7 @@ codebase to keep up with and no separate releases.
| The bot connects, commands do nothing | Use the prefix (`!ping`) or a mention (`@Sylo ping`), not `/`. If you changed `COMMAND_PREFIX` or the server's **Command prefix** setting, use the new one. The bot needs permission to read and send messages in that channel. |
| "You don't have permission to use this command" | The command needs the same permission as on Discord, or a per-server override blocks it. Check the dashboard's Commands page. |
| Login with Fluxer fails or loops | The redirect URL on the application must be exactly `<DASHBOARD_URL>/auth/fluxer/callback`. Behind a proxy set `DASHBOARD_URL` to the public https address. |
| Logged in, but my Fluxer server is missing | You need to own the server, or have Administrator or Manage Server there, and the bot has to be in it. Sign out and in again to refresh the list (it is refreshed automatically about every 10 minutes). |
| Logged in, but my Fluxer server is missing | You need to own the server, or have Administrator or Manage Server there, and the bot has to be in it. A server the bot joins after you signed in appears on your next page load; if it does not, open the server switcher and choose **Refresh server list** (once every 30 seconds), or sign out and in again. |
| `Log in with Fluxer` button missing | `FLUXER_CLIENT_SECRET` (and `FLUXER_CLIENT_ID`) must both be set. |
| `/health` says forbidden | Your id has to be in `FLUXER_OWNER_IDS` (Fluxer) or `OWNER_IDS` (Discord). The lists are separate on purpose. |
| A temporary voice channel is created but I am not moved | Expected on Fluxer. Open the link in the DM from the bot. |
Expand Down
7 changes: 6 additions & 1 deletion docs/mobile-app.md
Original file line number Diff line number Diff line change
Expand Up @@ -435,7 +435,12 @@ difference is how server access is checked:
- Both use the server list stored at login. The bot keeps it current: when it
sees someone gain or lose Manage Server or Administrator, or become the owner,
that server is added to or removed from their list. Neither the dashboard nor
the app sends anyone through the platform login again to refresh it.
the app sends anyone through the platform login again to refresh it. A server the
bot joins later is looked for once on each signed-in person's list (their next
request), and the dashboard's **Refresh server list** action
(`POST /api/v2/account/refresh-guilds`, once per 30 seconds per account, the
caller's own sign-ins only) does the same on demand. Both read the person's
rights through the bot, never through their platform login.
- The web dashboard trusts that list for 10 minutes after login, then checks
live through the bot, like the app.
- For app requests Sylo checks **live, through the bot, on every request**:
Expand Down
7 changes: 6 additions & 1 deletion src/bot/events/adminSnapshot.js
Original file line number Diff line number Diff line change
@@ -1,11 +1,16 @@
// Discord side of src/bot/lib/adminSnapshot.js: a member's roles change, or the server changes
// owner, and the signed-in person's list of servers they run follows.
// owner, and the signed-in person's list of servers they run follows. Joining a server is noted too, so
// the dashboard looks for it on the lists of the people who run it (src/web/lib/guildListRefresh.js).
import { Events } from 'discord.js';
import { guildKey } from '../../platform/platformOf.js';
import { syncAdminSnapshot, syncNewOwner } from '../lib/adminSnapshot.js';
import { noteGuildJoin } from '../lib/guildJoins.js';

/** @param {import('discord.js').Client} client */
export function register(client) {
client.on(Events.GuildCreate, (guild) => {
noteGuildJoin('discord', guildKey(guild), guild, { ready: client.isReady() });
});
client.on(Events.GuildMemberUpdate, (_old, member) => {
if (member?.guild) void syncAdminSnapshot('discord', guildKey(member.guild), member.guild, member);
});
Expand Down
7 changes: 6 additions & 1 deletion src/bot/fluxerModuleEvents.js
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import { fluxerReactionAdd } from '../platform/fluxerEvents.js';
import { beginModeration } from '../modules/lib/messageModeration.js';
import { log } from '../lib/log.js';
import { syncAdminSnapshot, syncNewOwner } from './lib/adminSnapshot.js';
import { noteGuildJoin } from './lib/guildJoins.js';
import { restoreOldRoles } from '../platform/compat.js';

const keyOf = (rawGuildId) => (rawGuildId ? makePlatformGuildId('fluxer', rawGuildId) : null);
Expand Down Expand Up @@ -117,7 +118,11 @@ export function registerFluxerModuleEvents(

on('inviteCreate', (invite) => send('inviteCreate', invite.guildSnapshot?.id, invite));
on('inviteDelete', (payload) => send('inviteDelete', payload.guildId, payload));
on('guildCreate', (guild) => send('guildCreate', guild.id, guild));
on('guildCreate', (guild) => {
// The dashboard looks for the new server on the lists of the people who run it (src/web/lib/guildListRefresh.js).
noteGuildJoin('fluxer', keyOf(guild.id), guild, { ready: client.isReady?.() !== false });
return send('guildCreate', guild.id, guild);
});
on('guildDelete', async (guild) => {
// Also fires when a guild merely becomes unavailable; don't wipe data then.
if (guild.available === false) return;
Expand Down
49 changes: 49 additions & 0 deletions src/bot/lib/guildJoins.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
// When the bot joined each server, as far as this process saw it. A signed-in person's list of
// servers they run comes from sign-in (src/db/accounts.js) and follows the bot's rights events
// (src/bot/lib/adminSnapshot.js), but a server the bot joins afterwards is on nobody's list, so the
// dashboard would not show it until they sign in again. The dashboard compares these times with when
// each person's list was last looked at (src/web/lib/guildListRefresh.js) and re-checks only then.

/** A join older than this when its event arrives is startup catch-up, not a new server. */
const RECENT_MS = 5 * 60 * 1000;
/** How long a join is remembered; the dashboard looks at a list on the person's next request. */
const KEEP_MS = 24 * 60 * 60 * 1000;

/** @type {Map<string, number>} guild key -> when the bot joined */
const joinedAt = new Map();
/** @type {Record<string, number>} platform -> when the bot last joined a server there */
const lastJoin = {};

/**
* The bot joined a server (guildCreate). Startup replays every existing server through the same event;
* `ready` is false then, and a join the platform dates long ago is ignored too.
* @param {'discord'|'fluxer'} platform
* @param {string} key the guild key
* @param {{ joinedTimestamp?: number } | null} guild
* @param {{ ready?: boolean, now?: number }} [opts]
*/
export function noteGuildJoin(platform, key, guild, { ready = true, now = Date.now() } = {}) {
if (!key || !ready) return;
const stamp = Number(guild?.joinedTimestamp);
if (Number.isFinite(stamp) && stamp > 0 && now - stamp > RECENT_MS) return;
for (const [k, at] of joinedAt) if (now - at > KEEP_MS) joinedAt.delete(k);
joinedAt.set(key, now);
lastJoin[platform] = now;
}

/** When the bot last joined a server on this platform (0: not since this process started). */
export const lastGuildJoin = (platform) => lastJoin[platform] ?? 0;

/** When the bot joined this server: seen by this process, else the platform's own date, else 0. */
export function guildJoinedAt(key, guild) {
const seen = joinedAt.get(key);
if (seen) return seen;
const stamp = Number(guild?.joinedTimestamp);
return Number.isFinite(stamp) && stamp > 0 ? stamp : 0;
}

/** For tests: forget every join. */
export function resetGuildJoins() {
joinedAt.clear();
for (const p of Object.keys(lastJoin)) delete lastJoin[p];
}
133 changes: 133 additions & 0 deletions src/web/lib/guildListRefresh.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
// Brings a signed-in person's list of servers they run up to date with servers the bot has joined since
// they signed in. The list comes from the platform at sign-in and then follows the bot's own events;
// a new server is on nobody's list until this adds it. Rights are read through the bot (owner, cached
// member, one member fetch), never through the person's OAuth token, which Sylo does not keep.
import { allGuildKeys, guildByKey } from '../../platform/platformOf.js';
import { splitPlatformGuildId } from '../../lib/platformId.js';
import { setIdentityGuildAdmin } from '../../db/accounts.js';
import { guildJoinedAt, lastGuildJoin } from '../../bot/lib/guildJoins.js';
import { log } from '../../lib/log.js';
import { refreshStaleMember } from '../middleware/access.js';

/** Most member lookups one refresh may make: each is one platform API call. */
const MAX_LOOKUPS = 25;
/** Most member lookups all refreshes together may make in a minute, however many accounts ask. */
const LOOKUPS_PER_MINUTE = 100;

let budget = { left: LOOKUPS_PER_MINUTE, resetAt: 0, size: LOOKUPS_PER_MINUTE };

/** Take one lookup from the instance-wide budget; false when this minute's lookups are used up. */
function takeLookup(now = Date.now()) {
if (now >= budget.resetAt) budget = { ...budget, left: budget.size, resetAt: now + 60_000 };
if (budget.left <= 0) return false;
budget.left -= 1;
return true;
}

/** @type {Map<string, number>} "platform:userId" -> when the automatic check last ran for that identity */
const checkedAt = new Map();
/** @type {Map<string, Promise<number>>} */
const running = new Map();

const idKey = (identity) => `${identity.platform}:${identity.platformUserId}`;
const platformOfKey = (key) => splitPlatformGuildId(key)?.platform ?? 'discord';

/**
* Add the servers this identity runs but has not listed. `check(key, userId)` is the live rights check
* (serverRightsLive in auth.js: `{ right }` or null). Owners are found for free; anyone else costs a
* lookup, made only for servers the bot joined after `since`: the newest first, at most MAX_LOOKUPS of
* them, and only while the instance-wide budget lasts.
* @returns {Promise<number>} how many servers were added
*/
export async function addJoinedGuilds(identity, check, { since = identity.guildsFetchedAt } = {}) {
const listed = new Set(identity.guildIds);
const owned = [];
const others = [];
for (const key of allGuildKeys()) {
if (listed.has(key) || platformOfKey(key) !== identity.platform) continue;
const guild = guildByKey(key);
if (!guild) continue;
if (guild.ownerId === identity.platformUserId) owned.push(key);
else {
const joined = guildJoinedAt(key, guild);
if (joined > since) others.push({ key, joined });
}
}
// The newest joins first: the cap must never leave out the server that was just added.
others.sort((a, b) => b.joined - a.joined);
let added = 0;
const add = async (key) => {
if (await setIdentityGuildAdmin(identity.platform, identity.platformUserId, key, true)) added += 1;
};
for (const key of owned) {
if ((await check(key, identity.platformUserId).catch(() => null))?.right) await add(key);
}
for (const { key } of others.slice(0, MAX_LOOKUPS)) {
if (!takeLookup()) break;
// Without the members intent a cached member can be out of date: look it up fresh, like every live access check.
await refreshStaleMember(key, identity.platformUserId, { write: true }).catch(() => {});
if ((await check(key, identity.platformUserId).catch(() => null))?.right) await add(key);
}
return added;
}

/**
* The automatic refresh, run for the account on each request: an identity is looked at once after the
* bot joins a server on its platform, and not again until the next join. Costs one map lookup per
* identity otherwise.
* @returns {Promise<{ changed: boolean, looked: Set<string> }>} whether any list changed (the caller then
* reads the identities again), and the identities that were looked at (the manual refresh skips them)
*/
export async function refreshAfterJoins(account, check, now = Date.now()) {
let changed = false;
const looked = new Set();
for (const identity of account?.identities ?? []) {
const key = idKey(identity);
if (lastGuildJoin(identity.platform) <= (checkedAt.get(key) ?? 0)) continue;
let work = running.get(key);
if (!work) {
// Only servers joined since the last look (or since sign-in): the earlier ones were checked already.
const since = checkedAt.get(key) ?? identity.guildsFetchedAt;
checkedAt.set(key, now);
work = addJoinedGuilds(identity, check, { since })
.catch((err) => {
log.warn('auth', `could not refresh the server list of ${key}: ${err.message}`);
return 0;
})
.finally(() => running.delete(key));
running.set(key, work);
}
looked.add(key);
if ((await work) > 0) changed = true;
}
return { changed, looked };
}

/**
* The "Refresh server list" action: every identity of this account, now, whatever the join times say.
* Only the account's own identities are touched. An identity the automatic check already looked at in this
* same request (`account.guildListLooked`) is left alone: that look was as thorough.
* @returns {Promise<number>} servers added
*/
export async function refreshAccountGuilds(account, check, now = Date.now()) {
let added = 0;
for (const identity of account?.identities ?? []) {
const key = idKey(identity);
if (account.guildListLooked?.has(key)) continue;
const inFlight = running.get(key);
if (inFlight) {
added += await inFlight;
continue;
}
checkedAt.set(key, now);
added += await addJoinedGuilds(identity, check);
}
return added;
}

/** For tests: forget everything; `lookups` sets this minute's instance-wide lookup budget. */
export function resetGuildListRefresh({ lookups = LOOKUPS_PER_MINUTE } = {}) {
checkedAt.clear();
running.clear();
budget = { left: lookups, resetAt: 0, size: lookups };
}
1 change: 1 addition & 0 deletions src/web/lib/routeAccess.js
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ export const ROUTE_ACCESS = Object.freeze({
'GET /guilds': 'self',
'GET /account': 'self',
'POST /account/devices/:id/revoke': 'self',
'POST /account/refresh-guilds': 'self',
'POST /account/disconnect/:platform': 'self',
'POST /account/delete': 'self',
'GET /changelog': 'self',
Expand Down
28 changes: 26 additions & 2 deletions src/web/middleware/auth.js
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ import { bearerToken, isTokenRequest } from '../lib/bearer.js';
import { enabledProviders, fluxerOrigins, providers, adminGuildIdsFromOAuth } from '../lib/oauthProviders.js';
import { buildBotInvites } from '../lib/botInvites.js';
import { adminKeysOf, identityFor, isOperator, primaryUserId, sessionUserLabel } from '../lib/accountAuth.js';
import { refreshAfterJoins, refreshAccountGuilds } from '../lib/guildListRefresh.js';
import { clientKey, rateLimit } from './rateLimit.js';
import { hasValidCsrfToken } from './csrf.js';
import { log } from '../../lib/log.js';
Expand Down Expand Up @@ -250,6 +251,23 @@ export async function serverRightsLive(key, userId, fetchMember) {
return { right, member };
}

/** The live rights check the server-list refresh uses: the bot's cache, else one member lookup. */
const rightsLive = (key, userId) => serverRightsLive(key, userId, fetchMemberFromGuild);

/**
* The account with its server lists brought up to date for servers the bot has joined since (see
* src/web/lib/guildListRefresh.js); the same account when nothing changed.
*/
async function withJoinedServers(account) {
const { changed, looked } = await refreshAfterJoins(account, rightsLive);
// `guildListLooked`: whom this request already looked at, so "Refresh server list" does not repeat it.
const base = changed ? { ...account, identities: await identitiesOf(account.id) } : account;
return looked.size ? { ...base, guildListLooked: looked } : base;
}

/** "Refresh server list": look at every server the bot joined for this account now. Resolves to the servers added. */
export const refreshServerLists = (account) => refreshAccountGuilds(account, rightsLive);

/**
* Does this person own the guild or hold Manage Server (or Administrator) in it right now? The
* live check the dashboard access levels rely on, minus the bot-master role: for what only real server
Expand Down Expand Up @@ -355,7 +373,10 @@ async function attachAccount(req, _res, next) {
if (token !== null) {
const record = await findApiToken(token);
if (record) {
req.account = { id: record.accountId, identities: await identitiesOf(record.accountId) };
req.account = await withJoinedServers({
id: record.accountId,
identities: await identitiesOf(record.accountId),
});
req.viaToken = true;
req.apiTokenId = record.id;
touchApiToken(record, { ttlMs: API_TOKEN_TTL_MS }).catch((err) =>
Expand All @@ -368,7 +389,10 @@ async function attachAccount(req, _res, next) {
if (sid) {
const session = await getSession(sid);
if (session) {
req.account = { id: session.accountId, identities: await identitiesOf(session.accountId) };
req.account = await withJoinedServers({
id: session.accountId,
identities: await identitiesOf(session.accountId),
});
} else {
req.session.sid = undefined; // expired or unknown
}
Expand Down
22 changes: 21 additions & 1 deletion src/web/routes/v2Api.js
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ import {
currentUser,
sessionUserId,
botInviteUrls,
refreshServerLists,
} from '../middleware/auth.js';
import {
checkLevelRewardGrants,
Expand All @@ -49,7 +50,7 @@ import {
accountGuildAccess,
checksLive,
} from '../middleware/access.js';
import { limitWhen, platformActionLimit, rateLimit } from '../middleware/rateLimit.js';
import { clientKey, limitWhen, platformActionLimit, rateLimit } from '../middleware/rateLimit.js';
import { asyncHandler } from '../lib/asyncHandler.js';
import { changelogHandler } from '../lib/changelogFeed.js';
import { ticketsRouter, staffRouter } from './v2Staff.js';
Expand Down Expand Up @@ -444,6 +445,25 @@ router.post(
})
);

// "Refresh server list": look again for servers the bot joined since sign-in, for this account's own
// identities only. Each look can cost the platform a member lookup, so once per 30 seconds per account.
const refreshGuildsLimit = rateLimit({
windowMs: 30_000,
max: 1,
message: 'The server list was just refreshed. Wait a few seconds and try again.',
keyFn: (req) => `refresh-guilds|${req.account?.id ?? clientKey(req)}`,
});

router.post(
'/account/refresh-guilds',
refreshGuildsLimit,
asyncHandler(async (req, res) => {
// Open mode lists every server already.
if (!req.account) return res.json({ ok: true, added: 0 });
res.json({ ok: true, added: await refreshServerLists(req.account) });
})
);

router.post(
'/account/disconnect/:platform',
asyncHandler(async (req, res) => {
Expand Down
Loading
Loading