Pre-fetch attribute keys on login and setup for completion #1787

Shell completion for dataset fields (explore --field/--sort, dashboard widget --group-by/--query/--sort, alert metrics --aggregate) reads the cached span-attribute, log-attribute and metric-attribute keys (#1785). Those kinds are only seeded from a GET /organizations/{org}/trace-items/attributes/ response, and today only sentry attribute list makes that request. Until someone runs it, field completion offers nothing. This issue analyzes warming that cache during auth login and cli setup (which cli upgrade runs). It was requested on #1785, and no code is attached.

Prior art

• warmOrgCache() in packages/cli/src/commands/auth/login.ts fires listOrganizationsUncached() after a successful login without awaiting it, and swallows failures. The reason given is that the first real command then skips about 800 ms of cold start.
• The fetch layer already seeds completion_cache from every response it sees (src/lib/api/completion-seed.ts, attributeEntities, matched on the dataset query parameter). A warm-up only has to make the request, unless the order of what it stores needs to change (see below).

Where to hook in

• auth login: next to warmOrgCache(), after the org list (or the default org) is known.
• cli setup: cli upgrade launches the new binary with cli setup, so hooking setup covers both upgrades and first installs. Only do it when setup installs or updates completions (not with --no-completions) and an account is stored. Shell completions are the only consumer of this cache.

Cost

• 3 requests per org (spans, logs, tracemetrics), which can run in parallel.
• For a large org, each dataset returns thousands of keys: roughly 1 MB of raw JSON (tens of KB gzipped), in under a second.
• The default page size is the server's explore.trace-items.keys.max option, and per_page applies per attribute type, so one page can contain several times that many rows. A small per_page makes the response small but doesn't fix the ordering problem below.

The 500 cap and name order

completion_cache keeps at most 500 entries per kind per org (MAX_ENTITIES_PER_ORG). cacheCompletionEntities keeps the first 500 in the order the API returned them. The endpoint returns stored keys sorted by name and puts Sentry-defined aliases after them. In a large org, numeric and vendor-specific keys therefore fill the cap before span.* and the other conventions, and the cached set is mostly unusable. This also affects sentry attribute list today.

Each row carries attributeSource.source_type. Sentry-defined (sentry) keys are a small subset of each dataset, in the low hundreds at most, so they fit under the cap. Ranking source_type == "sentry" keys first, then filling the rest with user keys, would keep the keys that matter. The natural place for this is the attributeEntities extractor, so that attribute list also benefits.

Server-side vs client-side filtering

Checked in getsentry/sentry src/sentry/api/endpoints/organization_trace_item_attributes.py:

• Query parameters are dataset (or the deprecated itemType), attributeType (multi), substringMatch, query and expand=context (OrganizationTraceItemAttributesEndpointSerializer), plus the usual project, environment and time-range parameters and per_page/cursor.
• There is no source_type filter. attributeSource is computed per key in as_attribute_key when the response is serialized.
• substringMatch is a case-sensitive contains match, not a prefix match. It is applied to the stored-key RPC (value_substring_match) and to the aliases.
• Without substringMatch, the first page always includes SENTRY_ALWAYS_INCLUDED_ATTRIBUTES (span.description for spans, message for logs, nothing for tracemetrics) and the virtual contexts, "so they aren't paged out". That covers one key per dataset, not the Sentry-defined set.

So the only option today is client-side filtering, which still downloads the full page per dataset. A source_type filter or a Sentry-keys-first order on the server would make the request small. That would need a separate getsentry/sentry change.

Which orgs to warm

• Every org a user belongs to means 3×N requests. That is too many for users in many orgs.
• The default org (getDefaultOrganization() in src/lib/db/defaults.ts) is the obvious first choice when set. Without one, a few recently used orgs (or the only org) are an option. Bare field completion with no target already falls back to every cached org.

Failure, timeout and offline behavior

• The warm-up must be best-effort. It must never fail, delay or change the output of auth login or cli setup: no thrown errors, no prompts, no stderr noise beyond log.debug.
• It needs a short timeout and an abort signal. When offline or the token is rejected, skip quietly.
• Fire-and-forget still keeps the process alive until the request settles, so its timeout must stay well under what login or setup can tolerate. Alternatively, await it with a hard cap at a point where the user already expects to wait.
• A missing feature, a 403 or a 404 from the endpoint (it returns 404 without the trace-items feature) should be treated as "nothing to warm".

Staleness and refresh

• completion_cache rows expire after 7 days (CACHE_TTL_MS in src/lib/db/index.ts, cleaned up in maybeCleanupCaches()), and logout clears the table.
• Refreshing on login and setup roughly matches how often keys change for completion purposes. attribute list and any other caller of the endpoint keep refreshing it in between.

Test plan

• Unit: the extractor ranks source_type: sentry keys first and keeps at most 500. A fixture whose first 500+ rows are user keys sorted by name still keeps span.duration.
• Login: the warm-up runs after a successful login, targets the default org, and makes one request per dataset. A 500 error, a timeout, a network error or a 404 doesn't change the exit code or output.
• Setup: the warm-up runs only when completions are installed and an account exists. It is skipped with --no-completions or when no account is stored.
• Completion: after the warm-up, explore -d spans -F span.d completes span.duration without running attribute list first.

Open questions

• Should the warm-up be awaited with a cap or left fire-and-forget, given that both commands exit soon after?
• Default org only, or the default org plus the most recently used orgs? How many?
• Is it worth asking getsentry/sentry for a source_type filter, or for Sentry-defined keys first, so the warm-up downloads a few KB instead of a full page?
• Should cli setup warm the cache on every upgrade, or only when the cache for that org is empty or older than some threshold?

Related: #1785, getsentry/sentry#129450.

via @BYK.

--

View Session in Sentry