docs: document the settings that only existed in source - #282
Merged
Conversation
Closing the two threads the audit left open rather than claiming it was finished. One found nothing; the other found eight settings that work and were discoverable only by reading the code. The worst is the AI block. The reference documented `ai.rca` and the token budget — how to turn root-cause analysis on and how to bound its spend — and never mentioned the provider, the key, the model or the endpoint. Setting FLAKEMETRY_AI_RCA=true from the reference alone gets you a feature that stays silent, because the worker has nothing to ask. Also newly written down: FLAKEMETRY_QUEUE_VISIBILITY_MS, with the warning that setting it below the slowest job makes the same run process twice, and the commit, branch and shard overrides for CI that detection does not recognise — a run without a commit lands on 0000000, where every run looks like the same commit to the scorer. The guard walks the source for FLAKEMETRY_ variables and fails when one appears in neither the reference nor an action definition. Three are excluded by name with a reason each, because silence should be chosen rather than fall out. Proved by removing one entry and watching it fail. The other thread — dead modules — came back clean. The five files nothing imports are the CLI binary, a type declaration and three Next route handlers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closing the two threads the audit left open, rather than letting "everything is fixed" stand on three findings when I had stopped short on two others.
Thread one: dead code — clean
The sweep I abandoned as inconclusive, done properly at module level. Five files nothing imports: the CLI binary, a type declaration, and three Next route handlers. All entry points. No dead modules.
Thread two: undocumented settings — eight
Everything below works today and was discoverable only by reading source.
The AI block is the one that matters. The configuration reference documented
ai.rcaandai.dailyTokenBudget— how to switch root-cause analysis on, and how to bound its spend — and never mentioned the provider, the key, the model or the endpoint. Those live only indocker-compose.yml, the Helm templates and.env.example.So an operator reading the reference to enable AI RCA sets
FLAKEMETRY_AI_RCA=true, gets a feature that stays silent, and has nothing to tell them why: the worker has no provider to ask. The reference now has its own section saying exactly that.Also written down for the first time:
FLAKEMETRY_QUEUE_VISIBILITY_MSFLAKEMETRY_COMMIT_SHA/_BRANCH0000000, where every such run looks like the same commit to the scorer, which is the strongest flakiness signal there isFLAKEMETRY_SHARD_INDEX/_TOTALThe guard
apps/api/src/__tests__/env-documented.test.tswalks the source forFLAKEMETRY_variables and fails when one appears in neither the configuration reference nor an action definition.Three are excluded by name with a reason each — the token and endpoint are the credential and the URL rather than knobs, and
FLAKEMETRY_SEED_FORCEis internal to a seed whose documented interface ispnpm demo. Silence should be chosen, not fall out.Proved non-vacuous by removing one entry from the reference and watching it fail:
It lives in
apps/apibesideapi-surface.test.ts— the same genre of check — becausepackages/contractsdeliberately has no@types/nodeand adding it to host a filesystem walk would spoil a package that is pure schemas.62/62 turbo tasks.