Skip to content

Reject unrecognized options in steps and tests - #98

Merged
bgrewell merged 1 commit into
mainfrom
feature/reject-unknown-options
Aug 8, 2026
Merged

Reject unrecognized options in steps and tests#98
bgrewell merged 1 commit into
mainfrom
feature/reject-unknown-options

Conversation

@bgrewell

@bgrewell bgrewell commented Aug 8, 2026

Copy link
Copy Markdown
Owner

Stacked on #97.

The problem

An option key that nothing consumes was dropped in silence, so the suite read as though it were set. The worst form:

    options:
      command: systemctl is-active nginx
      evaluatte:              # misspelled
        exit_code: 0

The test runs with zero checks, is reported as ran rather than passed or failed, and the suite exits 0 having asserted nothing. Same for an evaluator written one level too high (exit_code: beside command: instead of inside evaluate:), and for capture:/extract: on a type that does not honour them.

The approach

A hand-maintained list of valid keys per type would drift the moment a factory gained an option. Instead the option accessors record every key they read while a step or test is constructed; whatever the factory never asked for is, by definition, not an option of that type. The handful of factories that reach into the options map directly record their reads explicitly.

This is self-maintaining: a new option is accepted the moment a factory reads it, with nothing else to update.

Error: unknown option "evaluatte" in test "service responds"
(an execute test accepts: capture, command, evaluate, extract, timeout)

Error: unknown option "timout" in step "wait a moment"
(an execute step accepts: command, timeout)

--check reports them, so a typo surfaces before any infrastructure exists.

It found a real bug on its first run

message on the simulated step appears in every shipped example and was read by nothing — a dead option users have been copying since the examples were written. It now sets the status shown while the step waits, which is what the examples always implied. All 7 affected example suites validate again.

That is the check working as intended: it distinguishes "the user typo'd" from "we documented an option we never implemented", and this was the latter.

Verification

  • Every example suite under examples/ passes --check.
  • New tests assert the rejection for both classic false-green forms, options borrowed from another type, and — importantly — that every documented option of every step and test type is still accepted, so the tracking cannot silently break valid suites. Aliases (path/filename) are covered too.
  • go build, go vet, gofmt, and the full suite are green; the docs site builds.

Known gap, documented rather than fixed

Keys at the test or step level rather than inside options: — a stray key next to name:/node:/type:, or test: instead of tests: — are dropped by the YAML decoder before any of this runs. Catching those means a strict decode at the config layer, which is a separate change with its own blast radius. The --check suite summary remains the way to spot them, and the docs now say so plainly instead of describing the whole class as unchecked.

An option key nothing consumes was dropped in silence, so the suite read as
though it were set. The worst form is a misspelled or misplaced `evaluate`:
the test then runs with zero checks and is reported as `ran`, and the suite
exits 0 having asserted nothing.

Rather than maintain a list of valid keys per type, which drifts the moment a
factory gains an option, the option accessors record every key they read
while a step or test is constructed. Whatever the factory never asked for is
not an option of that type. Factories that reach into the options map
directly record their reads explicitly.

Errors name the offending key and the accepted set:

  Error: unknown option "evaluatte" in test "service responds"
  (an execute test accepts: capture, command, evaluate, extract, timeout)

--check reports them, so a typo surfaces before any infrastructure exists.

The check found a real dead option on its first run: `message` on the
simulated step appears in every shipped example and was read by nothing. It
now sets the status shown while the step waits, which is what the examples
always implied it did.

Note: keys at the test or step level rather than inside options: are still
dropped by the YAML decoder before this runs. The suite summary from --check
remains the way to catch those.
Base automatically changed from feature/docs-driven-fixes to main August 8, 2026 15:42
@bgrewell
bgrewell merged commit 945b679 into main Aug 8, 2026
2 checks passed
@bgrewell
bgrewell deleted the feature/reject-unknown-options branch August 8, 2026 15:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant