Skip to content

docs: document the settings that only existed in source - #282

Merged
AKogut merged 1 commit into
mainfrom
docs/undocumented-env
Aug 18, 2026
Merged

docs: document the settings that only existed in source#282
AKogut merged 1 commit into
mainfrom
docs/undocumented-env

Conversation

@AKogut

@AKogut AKogut commented Aug 18, 2026

Copy link
Copy Markdown
Owner

Closing the two threads the audit left open, rather than letting "everything is fixed" stand on three findings when I had stopped short on two others.

Thread one: dead code — clean

The sweep I abandoned as inconclusive, done properly at module level. Five files nothing imports: the CLI binary, a type declaration, and three Next route handlers. All entry points. No dead modules.

Thread two: undocumented settings — eight

Everything below works today and was discoverable only by reading source.

The AI block is the one that matters. The configuration reference documented ai.rca and ai.dailyTokenBudget — how to switch root-cause analysis on, and how to bound its spend — and never mentioned the provider, the key, the model or the endpoint. Those live only in docker-compose.yml, the Helm templates and .env.example.

So an operator reading the reference to enable AI RCA sets FLAKEMETRY_AI_RCA=true, gets a feature that stays silent, and has nothing to tell them why: the worker has no provider to ask. The reference now has its own section saying exactly that.

Also written down for the first time:

FLAKEMETRY_QUEUE_VISIBILITY_MS with the warning that setting it below the slowest job causes the same run to be processed twice
FLAKEMETRY_COMMIT_SHA / _BRANCH for CI that detection does not recognise — a run without a commit lands on 0000000, where every such run looks like the same commit to the scorer, which is the strongest flakiness signal there is
FLAKEMETRY_SHARD_INDEX / _TOTAL for sharding schemes detection does not recognise

The guard

apps/api/src/__tests__/env-documented.test.ts walks the source for FLAKEMETRY_ variables and fails when one appears in neither the configuration reference nor an action definition.

Three are excluded by name with a reason each — the token and endpoint are the credential and the URL rather than knobs, and FLAKEMETRY_SEED_FORCE is internal to a seed whose documented interface is pnpm demo. Silence should be chosen, not fall out.

Proved non-vacuous by removing one entry from the reference and watching it fail:

× finds each one in the documentation or an action definition
+   "FLAKEMETRY_QUEUE_VISIBILITY_MS",

It lives in apps/api beside api-surface.test.ts — the same genre of check — because packages/contracts deliberately has no @types/node and adding it to host a filesystem walk would spoil a package that is pure schemas.

62/62 turbo tasks.

Closing the two threads the audit left open rather than claiming it was
finished. One found nothing; the other found eight settings that work and were
discoverable only by reading the code.

The worst is the AI block. The reference documented `ai.rca` and the token
budget — how to turn root-cause analysis on and how to bound its spend — and
never mentioned the provider, the key, the model or the endpoint. Setting
FLAKEMETRY_AI_RCA=true from the reference alone gets you a feature that stays
silent, because the worker has nothing to ask.

Also newly written down: FLAKEMETRY_QUEUE_VISIBILITY_MS, with the warning that
setting it below the slowest job makes the same run process twice, and the
commit, branch and shard overrides for CI that detection does not recognise —
a run without a commit lands on 0000000, where every run looks like the same
commit to the scorer.

The guard walks the source for FLAKEMETRY_ variables and fails when one appears
in neither the reference nor an action definition. Three are excluded by name
with a reason each, because silence should be chosen rather than fall out.
Proved by removing one entry and watching it fail.

The other thread — dead modules — came back clean. The five files nothing
imports are the CLI binary, a type declaration and three Next route handlers.
@AKogut
AKogut merged commit 3182fc2 into main Aug 18, 2026
5 checks passed
@AKogut
AKogut deleted the docs/undocumented-env branch August 18, 2026 11:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant