From 195367a230b04e767b1dd4d53a75614ebc9443c8 Mon Sep 17 00:00:00 2001 From: Dara Rockwell Date: Sun, 1 Feb 2026 15:23:24 -0700 Subject: [PATCH 1/2] feat: add runner-level hooks and metadata MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Why This Change Was Made - Provide a first-class way to apply setup/teardown across suites without editing each suite’s internal hooks. - Expose deterministic metadata (name, full name, tags, kind, source) so runner hooks can make decisions based on identifiers. - Keep execution behavior explicit and predictable under parallelism while improving documentation and tested examples. ## What Was Changed - Added runner-level hook APIs in `dream_test/runner` and executed them in `dream_test/parallel` with sandboxing. - Introduced `types.TestInfo` and `types.SuiteInfo`, plus `source` propagation from discovery and Gherkin parsing. - Updated docs, snippets, tests, compatibility reports, and bumped the project to 2.1.0 with release notes. ## Note to Future Engineer - Runner hook failures have specific semantics (`SetupFailed` for before_each_test, synthetic failures for suite/run hooks); please keep these aligned if you refactor execution. - The `source` field is best-effort and will be `None` for manually constructed suites—sorry in advance to your future self’s forensic ambitions. --- CHANGELOG.md | 31 +- COMPATIBILITY.md | 6 +- README.md | 5 +- documentation/02-quick-start.md | 2 + documentation/03-writing-tests.md | 2 + documentation/04-context-aware-tests.md | 2 + documentation/06-lifecycle-hooks.md | 2 + documentation/07-runner-and-execution.md | 80 ++ documentation/README.md | 2 +- examples/failure_showcase/manifest.toml | 5 +- examples/snippets/manifest.toml | 6 +- .../test/snippets/gherkin/gherkin_types.gleam | 2 + .../test/snippets/runner/runner_hooks.gleam | 43 + .../utils/parallel_with_reporter.gleam | 2 + gleam.toml | 8 +- manifest.toml | 6 +- releases/release-2.1.0.md | 48 ++ src/dream_test/discover.gleam | 50 +- src/dream_test/gherkin/discover.gleam | 9 + src/dream_test/gherkin/feature.gleam | 3 + src/dream_test/gherkin/parser.gleam | 26 +- src/dream_test/gherkin/types.gleam | 2 + src/dream_test/parallel.gleam | 378 ++++++-- src/dream_test/runner.gleam | 809 +++++++++++++++--- src/dream_test/types.gleam | 40 + src/dream_test/unit.gleam | 22 +- src/dream_test/unit_context.gleam | 21 +- test/dream_test/gherkin/parser_api_test.gleam | 20 +- test/dream_test/runner_hooks_test.gleam | 384 +++++++++ 29 files changed, 1815 insertions(+), 201 deletions(-) create mode 100644 examples/snippets/test/snippets/runner/runner_hooks.gleam create mode 100644 releases/release-2.1.0.md create mode 100644 test/dream_test/runner_hooks_test.gleam diff --git a/CHANGELOG.md b/CHANGELOG.md index bf80167..d368ce0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,34 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [2.1.0] - 2026-01-31 + +### Added + +- **Runner-level hooks** (`dream_test/runner`, `dream_test/parallel`) + + - New `before_each_test`, `after_each_test`, `before_each_suite`, `after_each_suite`, + `before_all_suites`, and `after_all_suites` hooks + - Hooks receive structured metadata via `types.TestInfo` and `types.SuiteInfo` + - Per-test hooks run in the executor with sandboxing and correct ordering around suite hooks + +- **Test/suite metadata** (`dream_test/types`) + + - New `TestInfo` and `SuiteInfo` types, including best-effort `source` + - Source tracking for unit discovery (module name) and Gherkin discovery (feature path) + +### Changed + +- **Setup failure status** (`dream_test/parallel`) + + - Runner `before_each_test` failures now yield `SetupFailed` and skip test bodies + +### Documentation + +- Added runner hook section to the runner guide with a tested snippet +- Updated README and documentation index to highlight runner hooks +- Refreshed compatibility reports (`COMPATIBILITY.md`, `COMPATIBILITY_REPORT.md`, `CONSTRAINT_ANALYSIS.md`) + ## [2.0.0] - 2025-12-27 ### Added @@ -283,7 +311,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - STANDARDS document for code conventions - API documentation for all public modules -[Unreleased]: https://github.com/TrustBound/dream_test/compare/2.0.0...HEAD +[Unreleased]: https://github.com/TrustBound/dream_test/compare/2.1.0...HEAD +[2.1.0]: https://github.com/TrustBound/dream_test/compare/2.0.0...2.1.0 [2.0.0]: https://github.com/TrustBound/dream_test/compare/1.2.0...2.0.0 [1.2.0]: https://github.com/TrustBound/dream_test/compare/1.1.0...1.2.0 [1.1.0]: https://github.com/TrustBound/dream_test/compare/1.0.3...1.1.0 diff --git a/COMPATIBILITY.md b/COMPATIBILITY.md index e550274..b8cf3be 100644 --- a/COMPATIBILITY.md +++ b/COMPATIBILITY.md @@ -3,11 +3,11 @@ | Package | Minimum | Maximum | Status | |--------------|---------|---------|--------| | gleam_erlang | 1.0.0 | 1.3.0 | Tested | -| gleam_json | 3.0.1 | 3.1.0 | Tested | +| gleam_json | 3.0.2 | 3.1.0 | Tested | | gleam_otp | 1.1.0 | 1.2.0 | Tested | | gleam_regexp | 1.0.0 | 1.1.1 | Tested | -| gleam_stdlib | 0.60.0 | 0.67.1 | Tested | +| gleam_stdlib | 0.60.0 | 0.68.1 | Tested | All versions tested with: `gleam test` -Last tested: 2025-12-27 \ No newline at end of file +Last tested: 2026-02-01 \ No newline at end of file diff --git a/README.md b/README.md index bf17c70..c06e5d2 100644 --- a/README.md +++ b/README.md @@ -22,6 +22,8 @@ gleam add --dev dream_test ``` +Current version: `2.1.0` + ## Why Dream Test? Rapid application development needs testing tools that scale and support the growing needs of the application without slowing down progress. Dream test was designed to help engineers write expressive unit and integration tests for their applications using the tools and techniques they know from other ecosystems; adapted properly to gleam and the beam. @@ -60,6 +62,7 @@ Rapid application development needs testing tools that scale and support the gro | ⏱️ **Timeouts** | Per-test timeout control | | 🔍 **Test discovery** | Find tests from file paths | | 🚨 **Exit-on-failure** | Fail fast for CI | +| 🪝 **Runner hooks** | Runner-level hooks for per-test/per-suite/per-run setup and teardown | | 🧩 **Suite-specific execution config** | Run some suites sequential/with custom timeouts in the same runner (`runner.add_suites_with_config(...)`) | ### Reporting @@ -84,7 +87,7 @@ Dream Test splits reporting into: 4. [Context-Aware Tests](documentation/04-context-aware-tests.md) 5. [Assertions & Matchers](documentation/05-assertions-and-matchers.md) 6. [Lifecycle Hooks](documentation/06-lifecycle-hooks.md) -7. [Runner & Execution](documentation/07-runner-and-execution.md) +7. [Runner & Execution](documentation/07-runner-and-execution.md) — includes runner hooks 8. [Reporters](documentation/08-reporters.md) 9. [Snapshot Testing](documentation/09-snapshot-testing.md) 10. [Gherkin BDD](documentation/10-gherkin-bdd.md) diff --git a/documentation/02-quick-start.md b/documentation/02-quick-start.md index 895e083..bac55b9 100644 --- a/documentation/02-quick-start.md +++ b/documentation/02-quick-start.md @@ -12,6 +12,8 @@ That explicitness is the source of most of Dream Test’s reliability: when a te If you’re looking for a mental model of “how does the runner find my tests?”: in Dream Test, **`main()` chooses what runs** by passing suites to the runner. You can list suites explicitly, or generate the list via discovery. +If you need setup/teardown that spans multiple suites, see the runner-level hooks section in [Runner & execution model](07-runner-and-execution.md). + ### Choose your first runner style There are two good starting points: diff --git a/documentation/03-writing-tests.md b/documentation/03-writing-tests.md index 1a3f644..b713069 100644 --- a/documentation/03-writing-tests.md +++ b/documentation/03-writing-tests.md @@ -11,6 +11,8 @@ Dream Test’s unit DSL is built for that style, but there’s a deeper design g - **Your test module stays ordinary Gleam** (no hidden discovery side-effects). - **Failures should read well** (because tests are communication, not just verification). +If you need setup/teardown that spans multiple suites, use runner-level hooks in [Runner & execution model](07-runner-and-execution.md). + ### `describe` + `it` (the core loop) ```gleam diff --git a/documentation/04-context-aware-tests.md b/documentation/04-context-aware-tests.md index d9f8b5a..a11d3da 100644 --- a/documentation/04-context-aware-tests.md +++ b/documentation/04-context-aware-tests.md @@ -33,6 +33,8 @@ This is the right tool when: If you don’t need an explicit context, prefer `dream_test/unit` — it’s simpler. +For setup/teardown that spans multiple suites, use runner-level hooks in [Runner & execution model](07-runner-and-execution.md). + ### The idea: context flows through the suite - You give `describe` an initial `seed` value. diff --git a/documentation/06-lifecycle-hooks.md b/documentation/06-lifecycle-hooks.md index 6f7bd6c..2a467b3 100644 --- a/documentation/06-lifecycle-hooks.md +++ b/documentation/06-lifecycle-hooks.md @@ -4,6 +4,8 @@ Hooks are a power tool: they remove repetition, but they can also hide the story The goal of Dream Test’s hook design is to keep hooks **predictable and debuggable**, especially under parallel execution. +For runner-level hooks (per-test/per-suite/per-run), see [Runner & execution](07-runner-and-execution.md). + ### Mental model Hooks are part of the nested test structure that the runner executes around tests: diff --git a/documentation/07-runner-and-execution.md b/documentation/07-runner-and-execution.md index 87b41f4..86bbe2a 100644 --- a/documentation/07-runner-and-execution.md +++ b/documentation/07-runner-and-execution.md @@ -137,6 +137,86 @@ pub fn main() { 🧪 [Tested source](../examples/snippets/test/snippets/runner/suite_specific_config.gleam) +### Runner hooks (test/suite/run) + +Runner hooks let you apply setup/teardown **across suites** without modifying +each suite’s contents. Hooks are metadata-aware and receive structured info +about what is running. + +**`types.TestInfo` fields**: + +- `name`: leaf test name (`it("...")`) +- `full_name`: group path + test name (e.g. `["outer", "inner", "leaf"]`) +- `tags`: effective tags (includes inherited group tags) +- `kind`: `types.TestKind` (Unit, GherkinScenario(id), etc.) +- `source`: best-effort origin string (see below) + +**`types.SuiteInfo` fields**: + +- `name`: top-level suite/group name +- `tests`: deterministic list of `TestInfo` values that will execute +- `source`: best-effort origin string (see below) + +**Hook signatures**: + +- `before_each_test` / `after_each_test` receive `types.TestInfo`. +- `before_each_suite` / `after_each_suite` receive `types.SuiteInfo`. +- `before_all_suites` / `after_all_suites` receive the full list of `SuiteInfo` after filtering. + +`source` is best-effort: + +- Unit discovery suites use the **module name**. +- Gherkin discovery suites use the **.feature path**. +- Manually constructed suites use `None`. + +```gleam +import dream_test/matchers.{succeed} +import dream_test/runner +import dream_test/types.{type SuiteInfo, type TestInfo} +import dream_test/unit.{describe, it} +import gleam/int +import gleam/io +import gleam/list +import gleam/string + +pub fn tests() { + describe("Example", [ + it("a", fn() { Ok(succeed()) }), + it("b", fn() { Ok(succeed()) }), + ]) +} + +pub fn main() { + runner.new([tests()]) + |> runner.before_all_suites(before_all_suites_hook) + |> runner.before_each_suite(before_each_suite_hook) + |> runner.before_each_test(before_each_test_hook) + |> runner.after_each_test(after_each_test_hook) + |> runner.run() +} + +fn before_all_suites_hook(suites: List(SuiteInfo)) -> Result(Nil, String) { + io.println("suites: " <> int.to_string(list.length(suites))) + Ok(Nil) +} + +fn before_each_suite_hook(suite: SuiteInfo) -> Result(Nil, String) { + io.println("suite: " <> suite.name) + Ok(Nil) +} + +fn before_each_test_hook(info: TestInfo, ctx: Nil) -> Result(Nil, String) { + io.println("test: " <> string.join(info.full_name, " > ")) + Ok(ctx) +} + +fn after_each_test_hook(_info: TestInfo, _ctx: Nil) -> Result(Nil, String) { + Ok(Nil) +} +``` + +🧪 [Tested source](../examples/snippets/test/snippets/runner/runner_hooks.gleam) + ### Choosing a concurrency number (practical guidance) - Start with the default. diff --git a/documentation/README.md b/documentation/README.md index ca59a21..3702383 100644 --- a/documentation/README.md +++ b/documentation/README.md @@ -34,7 +34,7 @@ That’s why you’ll see the same patterns repeated: 4. [Context-aware unit tests](04-context-aware-tests.md) — when “setup returns a value” and you need to pass it into tests. 5. [Assertions & matchers](05-assertions-and-matchers.md) — how and why the `should` pipeline works. 6. [Lifecycle hooks](06-lifecycle-hooks.md) — power tools, and how to keep them from hiding meaning. -7. [Runner & execution model](07-runner-and-execution.md) — concurrency, timeouts, CI, and reliability. +7. [Runner & execution model](07-runner-and-execution.md) — concurrency, timeouts, CI, reliability, and runner hooks. 8. [Reporters](08-reporters.md) — humans vs machines, streaming vs post-run. 9. [Snapshot testing](09-snapshot-testing.md) — when snapshots make tests clearer (and when they make them worse). 10. [Gherkin / BDD](10-gherkin-bdd.md) — scenario testing with a world state and placeholder captures. diff --git a/examples/failure_showcase/manifest.toml b/examples/failure_showcase/manifest.toml index b1b1985..42dde12 100644 --- a/examples/failure_showcase/manifest.toml +++ b/examples/failure_showcase/manifest.toml @@ -2,14 +2,15 @@ # You typically do not need to edit this file packages = [ - { name = "dream_test", version = "2.0.0", build_tools = ["gleam"], requirements = ["gleam_erlang", "gleam_json", "gleam_otp", "gleam_regexp", "gleam_stdlib"], source = "local", path = "../.." }, + { name = "dream_test", version = "2.1.0", build_tools = ["gleam"], requirements = ["gleam_erlang", "gleam_json", "gleam_otp", "gleam_regexp", "gleam_stdlib"], source = "local", path = "../.." }, { name = "gleam_erlang", version = "1.3.0", build_tools = ["gleam"], requirements = ["gleam_stdlib"], otp_app = "gleam_erlang", source = "hex", outer_checksum = "1124AD3AA21143E5AF0FC5CF3D9529F6DB8CA03E43A55711B60B6B7B3874375C" }, { name = "gleam_json", version = "3.1.0", build_tools = ["gleam"], requirements = ["gleam_stdlib"], otp_app = "gleam_json", source = "hex", outer_checksum = "44FDAA8847BE8FC48CA7A1C089706BD54BADCC4C45B237A992EDDF9F2CDB2836" }, { name = "gleam_otp", version = "1.2.0", build_tools = ["gleam"], requirements = ["gleam_erlang", "gleam_stdlib"], otp_app = "gleam_otp", source = "hex", outer_checksum = "BA6A294E295E428EC1562DC1C11EA7530DCB981E8359134BEABC8493B7B2258E" }, { name = "gleam_regexp", version = "1.1.1", build_tools = ["gleam"], requirements = ["gleam_stdlib"], otp_app = "gleam_regexp", source = "hex", outer_checksum = "9C215C6CA84A5B35BB934A9B61A9A306EC743153BE2B0425A0D032E477B062A9" }, - { name = "gleam_stdlib", version = "0.67.1", build_tools = ["gleam"], requirements = [], otp_app = "gleam_stdlib", source = "hex", outer_checksum = "6CE3E4189A8B8EC2F73AB61A2FBDE49F159D6C9C61C49E3B3082E439F260D3D0" }, + { name = "gleam_stdlib", version = "0.68.1", build_tools = ["gleam"], requirements = [], otp_app = "gleam_stdlib", source = "hex", outer_checksum = "F7FAEBD8EF260664E86A46C8DBA23508D1D11BB3BCC6EE1B89B3BC3E5C83FF1E" }, ] [requirements] dream_test = { path = "../.." } +gleam_erlang = { version = ">= 1.3.0 and < 2.0.0" } gleam_stdlib = { version = ">= 0.44.0 and < 2.0.0" } diff --git a/examples/snippets/manifest.toml b/examples/snippets/manifest.toml index d0b9674..22afc5e 100644 --- a/examples/snippets/manifest.toml +++ b/examples/snippets/manifest.toml @@ -2,17 +2,17 @@ # You typically do not need to edit this file packages = [ - { name = "dream_test", version = "2.0.0", build_tools = ["gleam"], requirements = ["gleam_erlang", "gleam_json", "gleam_otp", "gleam_regexp", "gleam_stdlib"], source = "local", path = "../.." }, + { name = "dream_test", version = "2.1.0", build_tools = ["gleam"], requirements = ["gleam_erlang", "gleam_json", "gleam_otp", "gleam_regexp", "gleam_stdlib"], source = "local", path = "../.." }, { name = "gleam_erlang", version = "1.3.0", build_tools = ["gleam"], requirements = ["gleam_stdlib"], otp_app = "gleam_erlang", source = "hex", outer_checksum = "1124AD3AA21143E5AF0FC5CF3D9529F6DB8CA03E43A55711B60B6B7B3874375C" }, { name = "gleam_json", version = "3.1.0", build_tools = ["gleam"], requirements = ["gleam_stdlib"], otp_app = "gleam_json", source = "hex", outer_checksum = "44FDAA8847BE8FC48CA7A1C089706BD54BADCC4C45B237A992EDDF9F2CDB2836" }, { name = "gleam_otp", version = "1.2.0", build_tools = ["gleam"], requirements = ["gleam_erlang", "gleam_stdlib"], otp_app = "gleam_otp", source = "hex", outer_checksum = "BA6A294E295E428EC1562DC1C11EA7530DCB981E8359134BEABC8493B7B2258E" }, { name = "gleam_regexp", version = "1.1.1", build_tools = ["gleam"], requirements = ["gleam_stdlib"], otp_app = "gleam_regexp", source = "hex", outer_checksum = "9C215C6CA84A5B35BB934A9B61A9A306EC743153BE2B0425A0D032E477B062A9" }, - { name = "gleam_stdlib", version = "0.67.1", build_tools = ["gleam"], requirements = [], otp_app = "gleam_stdlib", source = "hex", outer_checksum = "6CE3E4189A8B8EC2F73AB61A2FBDE49F159D6C9C61C49E3B3082E439F260D3D0" }, + { name = "gleam_stdlib", version = "0.68.1", build_tools = ["gleam"], requirements = [], otp_app = "gleam_stdlib", source = "hex", outer_checksum = "F7FAEBD8EF260664E86A46C8DBA23508D1D11BB3BCC6EE1B89B3BC3E5C83FF1E" }, ] [requirements] dream_test = { path = "../.." } gleam_erlang = { version = ">= 1.3.0 and < 2.0.0" } +gleam_otp = { version = ">= 1.2.0 and < 2.0.0" } gleam_regexp = { version = ">= 1.0.0 and < 2.0.0" } gleam_stdlib = { version = ">= 0.44.0 and < 2.0.0" } -gleam_otp = { version = ">= 1.2.0 and < 2.0.0" } diff --git a/examples/snippets/test/snippets/gherkin/gherkin_types.gleam b/examples/snippets/test/snippets/gherkin/gherkin_types.gleam index e4eabbb..4059f6b 100644 --- a/examples/snippets/test/snippets/gherkin/gherkin_types.gleam +++ b/examples/snippets/test/snippets/gherkin/gherkin_types.gleam @@ -82,6 +82,7 @@ pub fn tests() { Feature( name: "Example feature", + source: None, description: None, tags: [], background: None, @@ -91,6 +92,7 @@ pub fn tests() { |> be_equal( Feature( name: "Example feature", + source: None, description: None, tags: [], background: None, diff --git a/examples/snippets/test/snippets/runner/runner_hooks.gleam b/examples/snippets/test/snippets/runner/runner_hooks.gleam new file mode 100644 index 0000000..24f4449 --- /dev/null +++ b/examples/snippets/test/snippets/runner/runner_hooks.gleam @@ -0,0 +1,43 @@ +import dream_test/matchers.{succeed} +import dream_test/runner +import dream_test/types.{type SuiteInfo, type TestInfo} +import dream_test/unit.{describe, it} +import gleam/int +import gleam/io +import gleam/list +import gleam/string + +pub fn tests() { + describe("Example", [ + it("a", fn() { Ok(succeed()) }), + it("b", fn() { Ok(succeed()) }), + ]) +} + +pub fn main() { + runner.new([tests()]) + |> runner.before_all_suites(before_all_suites_hook) + |> runner.before_each_suite(before_each_suite_hook) + |> runner.before_each_test(before_each_test_hook) + |> runner.after_each_test(after_each_test_hook) + |> runner.run() +} + +fn before_all_suites_hook(suites: List(SuiteInfo)) -> Result(Nil, String) { + io.println("suites: " <> int.to_string(list.length(suites))) + Ok(Nil) +} + +fn before_each_suite_hook(suite: SuiteInfo) -> Result(Nil, String) { + io.println("suite: " <> suite.name) + Ok(Nil) +} + +fn before_each_test_hook(info: TestInfo, ctx: Nil) -> Result(Nil, String) { + io.println("test: " <> string.join(info.full_name, " > ")) + Ok(ctx) +} + +fn after_each_test_hook(_info: TestInfo, _ctx: Nil) -> Result(Nil, String) { + Ok(Nil) +} diff --git a/examples/snippets/test/snippets/utils/parallel_with_reporter.gleam b/examples/snippets/test/snippets/utils/parallel_with_reporter.gleam index 6970f1c..7372564 100644 --- a/examples/snippets/test/snippets/utils/parallel_with_reporter.gleam +++ b/examples/snippets/test/snippets/utils/parallel_with_reporter.gleam @@ -38,6 +38,8 @@ pub fn main() { write: write, total: total, completed: completed, + runner_before_each_test: [], + runner_after_each_test: [], ), ) let parallel.RunRootParallelWithReporterResult( diff --git a/gleam.toml b/gleam.toml index e92a2a9..570d06c 100644 --- a/gleam.toml +++ b/gleam.toml @@ -1,15 +1,15 @@ name = "dream_test" -version = "2.0.0" +version = "2.1.0" description = "A testing framework for Gleam that gets out of your way" licences = ["MIT"] repository = { type = "github", user = "TrustBound", repo = "dream_test" } links = [{ title = "Documentation", href = "https://hexdocs.pm/dream_test" }] [dependencies] -gleam_stdlib = ">= 0.60.0 and < 1.0.0" -gleam_otp = ">= 1.1.0 and < 2.0.0" gleam_erlang = ">= 1.0.0 and < 2.0.0" +gleam_json = ">= 3.0.2 and < 4.0.0" +gleam_otp = ">= 1.1.0 and < 2.0.0" gleam_regexp = ">= 1.0.0 and < 2.0.0" -gleam_json = ">= 3.0.1 and < 4.0.0" +gleam_stdlib = ">= 0.60.0 and < 1.0.0" [dev-dependencies] diff --git a/manifest.toml b/manifest.toml index ba2e1f6..321e919 100644 --- a/manifest.toml +++ b/manifest.toml @@ -6,12 +6,12 @@ packages = [ { name = "gleam_json", version = "3.1.0", build_tools = ["gleam"], requirements = ["gleam_stdlib"], otp_app = "gleam_json", source = "hex", outer_checksum = "44FDAA8847BE8FC48CA7A1C089706BD54BADCC4C45B237A992EDDF9F2CDB2836" }, { name = "gleam_otp", version = "1.2.0", build_tools = ["gleam"], requirements = ["gleam_erlang", "gleam_stdlib"], otp_app = "gleam_otp", source = "hex", outer_checksum = "BA6A294E295E428EC1562DC1C11EA7530DCB981E8359134BEABC8493B7B2258E" }, { name = "gleam_regexp", version = "1.1.1", build_tools = ["gleam"], requirements = ["gleam_stdlib"], otp_app = "gleam_regexp", source = "hex", outer_checksum = "9C215C6CA84A5B35BB934A9B61A9A306EC743153BE2B0425A0D032E477B062A9" }, - { name = "gleam_stdlib", version = "0.67.0", build_tools = ["gleam"], requirements = [], otp_app = "gleam_stdlib", source = "hex", outer_checksum = "6368313DB35963DC02F677A513BB0D95D58A34ED0A9436C8116820BF94BE3511" }, + { name = "gleam_stdlib", version = "0.68.1", build_tools = ["gleam"], requirements = [], otp_app = "gleam_stdlib", source = "hex", outer_checksum = "F7FAEBD8EF260664E86A46C8DBA23508D1D11BB3BCC6EE1B89B3BC3E5C83FF1E" }, ] [requirements] gleam_erlang = { version = ">= 1.0.0 and < 2.0.0" } -gleam_json = { version = ">= 3.0.1 and < 4.0.0" } +gleam_json = { version = ">= 3.0.2 and < 4.0.0" } gleam_otp = { version = ">= 1.1.0 and < 2.0.0" } gleam_regexp = { version = ">= 1.0.0 and < 2.0.0" } -gleam_stdlib = { version = ">= 0.60.0 and < 2.0.0" } +gleam_stdlib = { version = ">= 0.60.0 and < 1.0.0" } diff --git a/releases/release-2.1.0.md b/releases/release-2.1.0.md new file mode 100644 index 0000000..feaf820 --- /dev/null +++ b/releases/release-2.1.0.md @@ -0,0 +1,48 @@ +# Dream Test 2.1.0 Release Notes + +**Release Date:** 2026-01-31 + +Dream Test 2.1 adds **runner-level hooks** so you can apply setup/teardown across +all suites without touching each suite’s internal hook structure. + +## Highlights + +### ✅ Runner-level hooks (test/suite/run) + +The runner now supports hooks at three levels: + +- **Per-test**: `before_each_test` / `after_each_test` +- **Per-suite**: `before_each_suite` / `after_each_suite` +- **Per-run**: `before_all_suites` / `after_all_suites` + +Per-test hooks run inside the executor (sandboxed, ordered correctly around suite hooks), +and failures in `before_each_test` yield `SetupFailed` so test bodies are skipped safely. + +### ✅ Metadata passed into hooks + +New metadata types are available for hooks and filtering: + +- `types.TestInfo` includes name, full name, tags, kind, and **source**. +- `types.SuiteInfo` includes suite name, test list, and **source**. + +`source` is best-effort: + +- Unit discovery suites use the **module name**. +- Gherkin discovery suites use the **.feature file path**. +- Manually constructed suites use `None`. + +### ✅ Documentation and examples + +- New **runner hooks** section in the Runner & execution guide. +- Tested snippet added at `examples/snippets/test/snippets/runner/runner_hooks.gleam`. +- README and documentation index updated to highlight runner hooks. +- Compatibility reports refreshed. + +## Files of interest + +- `src/dream_test/runner.gleam` +- `src/dream_test/parallel.gleam` +- `src/dream_test/types.gleam` +- `documentation/07-runner-and-execution.md` +- `examples/snippets/test/snippets/runner/runner_hooks.gleam` + diff --git a/src/dream_test/discover.gleam b/src/dream_test/discover.gleam index ea33ca6..f0f85b0 100644 --- a/src/dream_test/discover.gleam +++ b/src/dream_test/discover.gleam @@ -39,7 +39,7 @@ import dream_test/types.{ AssertionFailure, Group, Root, Test, Unit, } import gleam/list -import gleam/option.{None} +import gleam/option.{None, Some} import gleam/string // ============================================================================ @@ -432,7 +432,11 @@ fn load_suites_from_modules_next( ) -> LoadResult { case call_tests(module_name) { Ok(suite) -> - load_suites_from_modules(rest, [suite, ..suites_rev], errors_rev) + load_suites_from_modules( + rest, + [suite_with_source(suite, module_name), ..suites_rev], + errors_rev, + ) Error(message) -> load_suites_from_modules(rest, suites_rev, [ format_load_error(module_name, message), @@ -547,6 +551,47 @@ fn root_to_group(suite: TestSuite(Nil)) -> Node(Nil) { tree } +fn suite_with_source(suite: TestSuite(Nil), source: String) -> TestSuite(Nil) { + let Root(seed, tree) = suite + Root(seed: seed, tree: node_with_source(tree, source)) +} + +fn node_with_source(node: Node(Nil), source: String) -> Node(Nil) { + case node { + Group(name, tags, children) -> + Group( + name: name, + tags: tags, + children: children_with_source(children, source, []), + ) + Test(name, tags, kind, run, timeout_ms, _source) -> + Test( + name: name, + tags: tags, + kind: kind, + run: run, + timeout_ms: timeout_ms, + source: Some(source), + ) + other -> other + } +} + +fn children_with_source( + children: List(Node(Nil)), + source: String, + acc_rev: List(Node(Nil)), +) -> List(Node(Nil)) { + case children { + [] -> list.reverse(acc_rev) + [child, ..rest] -> + children_with_source(rest, source, [ + node_with_source(child, source), + ..acc_rev + ]) + } +} + fn error_to_node(error: String) -> Node(Nil) { Test( name: "Discovery Error: " <> error, @@ -554,6 +599,7 @@ fn error_to_node(error: String) -> Node(Nil) { kind: Unit, run: discovery_error_run, timeout_ms: None, + source: None, ) } diff --git a/src/dream_test/gherkin/discover.gleam b/src/dream_test/gherkin/discover.gleam index db4a447..227dfd7 100644 --- a/src/dream_test/gherkin/discover.gleam +++ b/src/dream_test/gherkin/discover.gleam @@ -66,6 +66,7 @@ import dream_test/types.{ } import gleam/list import gleam/option.{type Option, None, Some} +import gleam/string // ============================================================================ // Types @@ -323,9 +324,17 @@ fn error_to_node(error: String) -> Node(Nil) { kind: Unit, run: parse_error_test_run, timeout_ms: None, + source: source_from_parse_error(error), ) } +fn source_from_parse_error(error: String) -> Option(String) { + case string.split_once(error, ": ") { + Ok(#(path, _message)) -> Some(path) + Error(_) -> None + } +} + fn parse_error_test_run(_nil: Nil) -> Result(AssertionResult, String) { Ok(parse_error_assertion()) } diff --git a/src/dream_test/gherkin/feature.gleam b/src/dream_test/gherkin/feature.gleam index 7631c1f..63d4f4d 100644 --- a/src/dream_test/gherkin/feature.gleam +++ b/src/dream_test/gherkin/feature.gleam @@ -227,6 +227,7 @@ fn build_scenario_test_node( Ok(execute_scenario(scenario_id, all_steps, config.step_registry)) }, timeout_ms: None, + source: feature.source, ) } @@ -534,6 +535,7 @@ pub fn feature( let parsed_feature = gherkin_types.Feature( name: name, + source: None, description: None, tags: [], background: None, @@ -738,6 +740,7 @@ pub fn feature_with_background( let parsed_feature = gherkin_types.Feature( name: name, + source: None, description: None, tags: [], background: Some(gherkin_types.Background(steps: background_steps)), diff --git a/src/dream_test/gherkin/parser.gleam b/src/dream_test/gherkin/parser.gleam index d827269..bd5d17f 100644 --- a/src/dream_test/gherkin/parser.gleam +++ b/src/dream_test/gherkin/parser.gleam @@ -43,12 +43,35 @@ import gleam/string /// pub fn parse_file(path path: String) -> Result(Feature, String) { case file.read(path) { - Ok(content) -> parse_string(content) + Ok(content) -> + case parse_string(content) { + Ok(feature) -> Ok(feature_with_source(feature, path)) + Error(message) -> Error(message) + } Error(error) -> Error("Failed to read feature file: " <> file.error_to_string(error)) } } +fn feature_with_source(feature: Feature, source: String) -> Feature { + let Feature( + name: name, + source: _source, + description: description, + tags: tags, + background: background, + scenarios: scenarios, + ) = feature + Feature( + name: name, + source: Some(source), + description: description, + tags: tags, + background: background, + scenarios: scenarios, + ) +} + /// Parse Gherkin content from a string. /// /// Parses the provided string as Gherkin syntax. @@ -554,6 +577,7 @@ fn finalize_feature(state: ParserState) -> Result(Feature, String) { Some(name) -> Ok(Feature( name: name, + source: None, description: final_state.feature_description, tags: final_state.feature_tags, background: final_state.background, diff --git a/src/dream_test/gherkin/types.gleam b/src/dream_test/gherkin/types.gleam index c687f00..ed8ea97 100644 --- a/src/dream_test/gherkin/types.gleam +++ b/src/dream_test/gherkin/types.gleam @@ -232,6 +232,8 @@ pub type Feature { Feature( /// The feature name (from the Feature: line) name: String, + /// Optional source identifier (typically the .feature file path) + source: Option(String), /// Optional description text below the feature name description: Option(String), /// Tags applied to the feature (inherited by all scenarios) diff --git a/src/dream_test/parallel.gleam b/src/dream_test/parallel.gleam index c2cc47a..ee5ea09 100644 --- a/src/dream_test/parallel.gleam +++ b/src/dream_test/parallel.gleam @@ -41,9 +41,9 @@ import dream_test/sandbox import dream_test/timing import dream_test/types.{ type AssertionFailure, type AssertionResult, type Node, type Status, - type TestKind, type TestResult, type TestSuite, AfterAll, AfterEach, - AssertionFailed, AssertionFailure, AssertionOk, AssertionSkipped, BeforeAll, - BeforeEach, Failed, Group, Passed, Root, SetupFailed, Skipped, Test, + type TestInfo, type TestKind, type TestResult, type TestSuite, AfterAll, + AfterEach, AssertionFailed, AssertionFailure, AssertionOk, AssertionSkipped, + BeforeAll, BeforeEach, Failed, Group, Passed, Root, SetupFailed, Skipped, Test, TestResult, TimedOut, Unit, } import gleam/erlang/process.{ @@ -109,6 +109,10 @@ pub type RunRootParallelWithReporterConfig(context) { write: fn(String) -> Nil, total: Int, completed: Int, + runner_before_each_test: List( + fn(TestInfo, context) -> Result(context, String), + ), + runner_after_each_test: List(fn(TestInfo, context) -> Result(Nil, String)), ) } @@ -132,6 +136,10 @@ type ExecuteNodeConfig(context) { context: context, inherited_before_each: List(fn(context) -> Result(context, String)), inherited_after_each: List(fn(context) -> Result(Nil, String)), + runner_before_each_test: List( + fn(TestInfo, context) -> Result(context, String), + ), + runner_after_each_test: List(fn(TestInfo, context) -> Result(Nil, String)), node: Node(context), progress_reporter: Option(progress.ProgressReporter), write: fn(String) -> Nil, @@ -164,6 +172,10 @@ type RunParallelLoopConfig(context) { scope: List(String), inherited_tags: List(String), context: context, + runner_before_each_test: List( + fn(TestInfo, context) -> Result(context, String), + ), + runner_after_each_test: List(fn(TestInfo, context) -> Result(Nil, String)), before_each_hooks: List(fn(context) -> Result(context, String)), after_each_hooks: List(fn(context) -> Result(Nil, String)), failures_rev: List(AssertionFailure), @@ -187,6 +199,10 @@ type StartWorkersUpToLimitConfig(context) { scope: List(String), inherited_tags: List(String), context: context, + runner_before_each_test: List( + fn(TestInfo, context) -> Result(context, String), + ), + runner_after_each_test: List(fn(TestInfo, context) -> Result(Nil, String)), before_each_hooks: List(fn(context) -> Result(context, String)), after_each_hooks: List(fn(context) -> Result(Nil, String)), failures_rev: List(AssertionFailure), @@ -310,6 +326,8 @@ pub fn run_root_parallel( context: seed, inherited_before_each: [], inherited_after_each: [], + runner_before_each_test: [], + runner_after_each_test: [], node: tree, progress_reporter: None, write: discard_write, @@ -400,6 +418,8 @@ pub fn run_root_parallel_with_reporter( write: write, total: total, completed: completed, + runner_before_each_test: runner_before_each_test, + runner_after_each_test: runner_after_each_test, ) = config let Root(seed, tree) = suite @@ -411,6 +431,8 @@ pub fn run_root_parallel_with_reporter( context: seed, inherited_before_each: [], inherited_after_each: [], + runner_before_each_test: runner_before_each_test, + runner_after_each_test: runner_after_each_test, node: tree, progress_reporter: progress_reporter, write: write, @@ -441,6 +463,8 @@ fn execute_node( context: context, inherited_before_each: inherited_before_each, inherited_after_each: inherited_after_each, + runner_before_each_test: runner_before_each_test, + runner_after_each_test: runner_after_each_test, node: node, progress_reporter: progress_reporter, write: write, @@ -522,6 +546,8 @@ fn execute_node( group_scope, combined_tags, ctx2, + runner_before_each_test, + runner_after_each_test, combined_before_each, combined_after_each, tests, @@ -538,6 +564,8 @@ fn execute_node( group_scope, combined_tags, ctx2, + runner_before_each_test, + runner_after_each_test, combined_before_each, combined_after_each, groups, @@ -588,7 +616,7 @@ fn fail_test_nodes( ) -> List(TestResult) { case nodes { [] -> acc_rev - [Test(name, tags, kind, _run, _timeout_ms), ..rest] -> { + [Test(name, tags, kind, _run, _timeout_ms, _source), ..rest] -> { let full_name = list.append(scope, [name]) let all_tags = list.append(inherited_tags, tags) let failures = list.reverse(failures_rev) @@ -768,6 +796,10 @@ fn run_tests_in_group( scope: List(String), inherited_tags: List(String), context: context, + runner_before_each_test: List( + fn(TestInfo, context) -> Result(context, String), + ), + runner_after_each_test: List(fn(TestInfo, context) -> Result(Nil, String)), before_each_hooks: List(fn(context) -> Result(context, String)), after_each_hooks: List(fn(context) -> Result(Nil, String)), tests: List(Node(context)), @@ -786,6 +818,8 @@ fn run_tests_in_group( scope, inherited_tags, context, + runner_before_each_test, + runner_after_each_test, before_each_hooks, after_each_hooks, tests, @@ -802,6 +836,8 @@ fn run_tests_in_group( scope, inherited_tags, context, + runner_before_each_test, + runner_after_each_test, before_each_hooks, after_each_hooks, tests, @@ -819,6 +855,10 @@ fn run_tests_parallel( scope: List(String), inherited_tags: List(String), context: context, + runner_before_each_test: List( + fn(TestInfo, context) -> Result(context, String), + ), + runner_after_each_test: List(fn(TestInfo, context) -> Result(Nil, String)), before_each_hooks: List(fn(context) -> Result(context, String)), after_each_hooks: List(fn(context) -> Result(Nil, String)), tests: List(Node(context)), @@ -839,6 +879,8 @@ fn run_tests_parallel( scope: scope, inherited_tags: inherited_tags, context: context, + runner_before_each_test: runner_before_each_test, + runner_after_each_test: runner_after_each_test, before_each_hooks: before_each_hooks, after_each_hooks: after_each_hooks, failures_rev: failures_rev, @@ -877,6 +919,8 @@ fn run_parallel_loop( scope: scope, inherited_tags: inherited_tags, context: context, + runner_before_each_test: runner_before_each_test, + runner_after_each_test: runner_after_each_test, before_each_hooks: before_each_hooks, after_each_hooks: after_each_hooks, failures_rev: failures_rev, @@ -899,6 +943,8 @@ fn run_parallel_loop( scope: scope, inherited_tags: inherited_tags, context: context, + runner_before_each_test: runner_before_each_test, + runner_after_each_test: runner_after_each_test, before_each_hooks: before_each_hooks, after_each_hooks: after_each_hooks, failures_rev: failures_rev, @@ -984,6 +1030,8 @@ fn start_workers_up_to_limit( scope: scope, inherited_tags: inherited_tags, context: context, + runner_before_each_test: runner_before_each_test, + runner_after_each_test: runner_after_each_test, before_each_hooks: before_each_hooks, after_each_hooks: after_each_hooks, failures_rev: failures_rev, @@ -1006,6 +1054,8 @@ fn start_workers_up_to_limit( scope, inherited_tags, context, + runner_before_each_test, + runner_after_each_test, before_each_hooks, after_each_hooks, failures_rev, @@ -1039,7 +1089,7 @@ fn running_test_metadata( node: Node(context), ) -> #(String, List(String), List(String), TestKind) { case node { - Test(name, tags, kind, _run, _timeout_ms) -> #( + Test(name, tags, kind, _run, _timeout_ms, _source) -> #( name, list.append(scope, [name]), list.append(inherited_tags, tags), @@ -1055,6 +1105,10 @@ fn spawn_test_worker( scope: List(String), inherited_tags: List(String), context: context, + runner_before_each_test: List( + fn(TestInfo, context) -> Result(context, String), + ), + runner_after_each_test: List(fn(TestInfo, context) -> Result(Nil, String)), before_each_hooks: List(fn(context) -> Result(context, String)), after_each_hooks: List(fn(context) -> Result(Nil, String)), failures_rev: List(AssertionFailure), @@ -1072,6 +1126,8 @@ fn spawn_test_worker( scope, inherited_tags, context, + runner_before_each_test, + runner_after_each_test, before_each_hooks, after_each_hooks, failures_rev, @@ -1093,7 +1149,7 @@ fn test_timeout_ms(config: ParallelConfig, node: Node(context)) -> Int { let ParallelConfig(default_timeout_ms: default_timeout_ms, max_concurrency: _) = config case node { - Test(_, _, _, _, Some(ms)) -> ms + Test(_, _, _, _, Some(ms), _) -> ms _ -> default_timeout_ms } } @@ -1444,6 +1500,10 @@ fn execute_one_test_node( scope: List(String), inherited_tags: List(String), context: context, + runner_before_each_test: List( + fn(TestInfo, context) -> Result(context, String), + ), + runner_after_each_test: List(fn(TestInfo, context) -> Result(Nil, String)), before_each_hooks: List(fn(context) -> Result(context, String)), after_each_hooks: List(fn(context) -> Result(Nil, String)), failures_rev: List(AssertionFailure), @@ -1451,33 +1511,68 @@ fn execute_one_test_node( ) -> TestResult { // Fallback to sequential single-test logic by reusing existing code path. case node { - Test(name, tags, kind, run, timeout_ms) -> { + Test(name, tags, kind, run, timeout_ms, source) -> { let full_name = list.append(scope, [name]) let all_tags = list.append(inherited_tags, tags) + let info = + types.TestInfo( + name: name, + full_name: full_name, + tags: all_tags, + kind: kind, + source: source, + ) let start = timing.now_ms() - let #(ctx_after_setup, setup_status, setup_failures) = - run_before_each_list(config, scope, context, before_each_hooks, []) + let #(ctx_after_runner_setup, runner_setup_status, runner_setup_failures) = + run_runner_before_each_list( + config, + scope, + info, + context, + runner_before_each_test, + [], + ) - let assertion = case setup_status { - SetupFailed -> AssertionFailed(head_failure_or_unknown(setup_failures)) + let #(ctx_after_setup, setup_status, setup_failures) = case + runner_setup_status + { + SetupFailed -> #( + ctx_after_runner_setup, + SetupFailed, + runner_setup_failures, + ) _ -> - run_in_sandbox(config, timeout_ms, fn() { - case run(ctx_after_setup) { - Ok(a) -> a - Error(message) -> AssertionFailed(hook_failure("error", message)) - } - }) + run_before_each_list( + config, + scope, + ctx_after_runner_setup, + before_each_hooks, + runner_setup_failures, + ) } - let #(status, failures) = - assertion_to_status_and_failures( - assertion, - failures_rev, - setup_failures, - ) + let #(status, failures) = case setup_status { + SetupFailed -> + setup_failed_status_and_failures(failures_rev, setup_failures) + _ -> { + let assertion = + run_in_sandbox(config, timeout_ms, fn() { + case run(ctx_after_setup) { + Ok(a) -> a + Error(message) -> + AssertionFailed(hook_failure("error", message)) + } + }) + assertion_to_status_and_failures( + assertion, + failures_rev, + setup_failures, + ) + } + } - let #(final_status, final_failures) = + let #(status_after_suite, failures_after_suite) = run_after_each_list( config, scope, @@ -1487,6 +1582,17 @@ fn execute_one_test_node( failures, ) + let #(final_status, final_failures) = + run_runner_after_each_list( + config, + scope, + info, + ctx_after_setup, + runner_after_each_test, + status_after_suite, + failures_after_suite, + ) + let duration = timing.now_ms() - start TestResult( @@ -1517,6 +1623,10 @@ fn run_tests_sequentially( scope: List(String), inherited_tags: List(String), context: context, + runner_before_each_test: List( + fn(TestInfo, context) -> Result(context, String), + ), + runner_after_each_test: List(fn(TestInfo, context) -> Result(Nil, String)), before_each_hooks: List(fn(context) -> Result(context, String)), after_each_hooks: List(fn(context) -> Result(Nil, String)), tests: List(Node(context)), @@ -1529,33 +1639,68 @@ fn run_tests_sequentially( ) -> #(List(TestResult), Int) { case tests { [] -> #(acc_rev, completed) - [Test(name, tags, kind, run, timeout_ms), ..rest] -> { + [Test(name, tags, kind, run, timeout_ms, source), ..rest] -> { let full_name = list.append(scope, [name]) let all_tags = list.append(inherited_tags, tags) + let info = + types.TestInfo( + name: name, + full_name: full_name, + tags: all_tags, + kind: kind, + source: source, + ) let start = timing.now_ms() - let #(ctx_after_setup, setup_status, setup_failures) = - run_before_each_list(config, scope, context, before_each_hooks, []) + let #(ctx_after_runner_setup, runner_setup_status, runner_setup_failures) = + run_runner_before_each_list( + config, + scope, + info, + context, + runner_before_each_test, + [], + ) - let assertion = case setup_status { - SetupFailed -> AssertionFailed(head_failure_or_unknown(setup_failures)) + let #(ctx_after_setup, setup_status, setup_failures) = case + runner_setup_status + { + SetupFailed -> #( + ctx_after_runner_setup, + SetupFailed, + runner_setup_failures, + ) _ -> - run_in_sandbox(config, timeout_ms, fn() { - case run(ctx_after_setup) { - Ok(a) -> a - Error(message) -> AssertionFailed(hook_failure("error", message)) - } - }) + run_before_each_list( + config, + scope, + ctx_after_runner_setup, + before_each_hooks, + runner_setup_failures, + ) } - let #(status, failures) = - assertion_to_status_and_failures( - assertion, - failures_rev, - setup_failures, - ) + let #(status, failures) = case setup_status { + SetupFailed -> + setup_failed_status_and_failures(failures_rev, setup_failures) + _ -> { + let assertion = + run_in_sandbox(config, timeout_ms, fn() { + case run(ctx_after_setup) { + Ok(a) -> a + Error(message) -> + AssertionFailed(hook_failure("error", message)) + } + }) + assertion_to_status_and_failures( + assertion, + failures_rev, + setup_failures, + ) + } + } - let #(final_status, final_failures) = + let #(status_after_suite, failures_after_suite) = run_after_each_list( config, scope, @@ -1565,6 +1710,17 @@ fn run_tests_sequentially( failures, ) + let #(final_status, final_failures) = + run_runner_after_each_list( + config, + scope, + info, + ctx_after_setup, + runner_after_each_test, + status_after_suite, + failures_after_suite, + ) + let duration = timing.now_ms() - start let result = @@ -1592,6 +1748,8 @@ fn run_tests_sequentially( scope, inherited_tags, context, + runner_before_each_test, + runner_after_each_test, before_each_hooks, after_each_hooks, rest, @@ -1610,6 +1768,8 @@ fn run_tests_sequentially( scope, inherited_tags, context, + runner_before_each_test, + runner_after_each_test, before_each_hooks, after_each_hooks, rest, @@ -1628,6 +1788,10 @@ fn run_child_groups_sequentially( scope: List(String), inherited_tags: List(String), context: context, + runner_before_each_test: List( + fn(TestInfo, context) -> Result(context, String), + ), + runner_after_each_test: List(fn(TestInfo, context) -> Result(Nil, String)), before_each_hooks: List(fn(context) -> Result(context, String)), after_each_hooks: List(fn(context) -> Result(Nil, String)), groups: List(Node(context)), @@ -1648,6 +1812,8 @@ fn run_child_groups_sequentially( context: context, inherited_before_each: before_each_hooks, inherited_after_each: after_each_hooks, + runner_before_each_test: runner_before_each_test, + runner_after_each_test: runner_after_each_test, node: group_node, progress_reporter: progress_reporter, write: write, @@ -1660,6 +1826,8 @@ fn run_child_groups_sequentially( scope, inherited_tags, context, + runner_before_each_test, + runner_after_each_test, before_each_hooks, after_each_hooks, rest, @@ -1803,19 +1971,124 @@ fn run_after_each_list( } } -fn hook_failure(operator: String, message: String) -> AssertionFailure { - AssertionFailure(operator: operator, message: message, payload: None) +fn run_runner_before_each_list( + config: ParallelConfig, + scope: List(String), + info: TestInfo, + context: context, + hooks: List(fn(TestInfo, context) -> Result(context, String)), + failures_rev: List(AssertionFailure), +) -> #(context, Status, List(AssertionFailure)) { + case hooks { + [] -> #(context, Passed, failures_rev) + [hook, ..rest] -> + case run_runner_hook_transform(config, scope, info, context, hook) { + Ok(next) -> + run_runner_before_each_list( + config, + scope, + info, + next, + rest, + failures_rev, + ) + Error(message) -> #(context, SetupFailed, [ + hook_failure("before_each_test", message), + ..failures_rev + ]) + } + } } -fn head_failure_or_unknown( +fn run_runner_after_each_list( + config: ParallelConfig, + scope: List(String), + info: TestInfo, + context: context, + hooks: List(fn(TestInfo, context) -> Result(Nil, String)), + status: Status, failures_rev: List(AssertionFailure), -) -> AssertionFailure { - case failures_rev { - [f, ..] -> f - [] -> hook_failure("before_each", "setup failed") +) -> #(Status, List(AssertionFailure)) { + case hooks { + [] -> #(status, failures_rev) + [hook, ..rest] -> + case run_runner_hook_teardown(config, scope, info, context, hook) { + Ok(_) -> + run_runner_after_each_list( + config, + scope, + info, + context, + rest, + status, + failures_rev, + ) + Error(message) -> + run_runner_after_each_list( + config, + scope, + info, + context, + rest, + Failed, + [hook_failure("after_each_test", message), ..failures_rev], + ) + } + } +} + +fn run_runner_hook_transform( + config: ParallelConfig, + scope: List(String), + info: TestInfo, + context: context, + hook: fn(TestInfo, context) -> Result(context, String), +) -> Result(context, String) { + let ParallelConfig(default_timeout_ms: default_timeout_ms, max_concurrency: _) = + config + let sandbox_config = + sandbox.SandboxConfig( + timeout_ms: default_timeout_ms, + show_crash_reports: False, + ) + + case sandbox.run_isolated(sandbox_config, fn() { hook(info, context) }) { + sandbox.SandboxCompleted(result) -> result + sandbox.SandboxTimedOut -> + Error("hook timed out in " <> string.join(scope, " > ")) + sandbox.SandboxCrashed(reason) -> + Error("hook crashed in " <> string.join(scope, " > ") <> ": " <> reason) + } +} + +fn run_runner_hook_teardown( + config: ParallelConfig, + scope: List(String), + info: TestInfo, + context: context, + hook: fn(TestInfo, context) -> Result(Nil, String), +) -> Result(Nil, String) { + let ParallelConfig(default_timeout_ms: default_timeout_ms, max_concurrency: _) = + config + let sandbox_config = + sandbox.SandboxConfig( + timeout_ms: default_timeout_ms, + show_crash_reports: False, + ) + + case sandbox.run_isolated(sandbox_config, fn() { hook(info, context) }) { + sandbox.SandboxCompleted(result) -> result + sandbox.SandboxTimedOut -> + Error("hook timed out in " <> string.join(scope, " > ")) + sandbox.SandboxCrashed(reason) -> + Error("hook crashed in " <> string.join(scope, " > ") <> ": " <> reason) } } +fn hook_failure(operator: String, message: String) -> AssertionFailure { + AssertionFailure(operator: operator, message: message, payload: None) +} + fn assertion_to_status_and_failures( result: AssertionResult, inherited_failures_rev: List(AssertionFailure), @@ -1837,6 +2110,13 @@ fn assertion_to_status_and_failures( } } +fn setup_failed_status_and_failures( + inherited_failures_rev: List(AssertionFailure), + setup_failures_rev: List(AssertionFailure), +) -> #(Status, List(AssertionFailure)) { + #(SetupFailed, list.append(setup_failures_rev, inherited_failures_rev)) +} + fn run_in_sandbox( config: ParallelConfig, timeout_override: Option(Int), diff --git a/src/dream_test/runner.gleam b/src/dream_test/runner.gleam index ef3d775..3103479 100644 --- a/src/dream_test/runner.gleam +++ b/src/dream_test/runner.gleam @@ -55,8 +55,9 @@ import dream_test/reporters/json import dream_test/reporters/progress import dream_test/reporters/types as reporter_types import dream_test/types.{ - type Node, type TestKind, type TestResult, type TestSuite, AfterAll, AfterEach, - BeforeAll, BeforeEach, Failed, Group, Root, SetupFailed, Test, TimedOut, + type AssertionFailure, type Node, type TestResult, type TestSuite, AfterAll, + AfterEach, BeforeAll, BeforeEach, Failed, Group, Root, SetupFailed, Test, + TimedOut, Unit, } import gleam/io import gleam/list @@ -64,58 +65,9 @@ import gleam/option.{type Option, None, Some} /// Lightweight information about a test, used for filtering what runs. /// -/// ## Fields -/// -/// - `name`: the test’s local name (the one passed to `it("...", ...)`) -/// - `full_name`: the group path + test name (useful for fully-qualified filters) -/// - `tags`: effective tags (includes inherited group tags) -/// - `kind`: the `types.TestKind` (Unit, Gherkin, etc.) -/// -/// ## Example -/// -/// ```gleam -/// import dream_test/matchers.{be_equal, or_fail_with, should, succeed} -/// import dream_test/reporters/bdd -/// import dream_test/reporters/progress -/// import dream_test/runner.{type TestInfo} -/// import dream_test/unit.{describe, it, with_tags} -/// import gleam/list -/// -/// pub fn tests() { -/// describe("Filtering tests", [ -/// it("smoke", fn() { -/// 1 + 1 -/// |> should -/// |> be_equal(2) -/// |> or_fail_with("math should work") -/// }) -/// |> with_tags(["smoke"]), -/// it("slow", fn() { Ok(succeed()) }) -/// |> with_tags(["slow"]), -/// ]) -/// } -/// -/// pub fn only_smoke(info: TestInfo) -> Bool { -/// list.contains(info.tags, "smoke") -/// } -/// -/// pub fn main() { -/// runner.new([tests()]) -/// |> runner.filter_tests(only_smoke) -/// |> runner.progress_reporter(progress.new()) -/// |> runner.results_reporters([bdd.new()]) -/// |> runner.exit_on_failure() -/// |> runner.run() -/// } -/// ``` -pub type TestInfo { - TestInfo( - name: String, - full_name: List(String), - tags: List(String), - kind: TestKind, - ) -} +/// See `dream_test/types.TestInfo` for fields and details. +pub type TestInfo = + types.TestInfo /// Builder for configuring and running suites. /// @@ -161,6 +113,18 @@ pub opaque type RunBuilder(ctx) { progress_reporter: Option(progress.ProgressReporter), results_reporters: List(reporter_types.ResultsReporter), output: Option(Output), + runner_before_each_test: List( + fn(types.TestInfo, ctx) -> Result(ctx, String), + ), + runner_after_each_test: List(fn(types.TestInfo, ctx) -> Result(Nil, String)), + runner_before_each_suite: List(fn(types.SuiteInfo) -> Result(Nil, String)), + runner_after_each_suite: List(fn(types.SuiteInfo) -> Result(Nil, String)), + runner_before_all_suites: List( + fn(List(types.SuiteInfo)) -> Result(Nil, String), + ), + runner_after_all_suites: List( + fn(List(types.SuiteInfo)) -> Result(Nil, String), + ), ) } @@ -233,6 +197,12 @@ pub fn new(suites suites: List(TestSuite(ctx))) -> RunBuilder(ctx) { progress_reporter: None, results_reporters: [], output: None, + runner_before_each_test: [], + runner_after_each_test: [], + runner_before_each_suite: [], + runner_after_each_suite: [], + runner_before_all_suites: [], + runner_after_all_suites: [], ) } @@ -579,6 +549,112 @@ pub fn filter_tests( RunBuilder(..builder, test_filter: Some(predicate)) } +/// Register a runner-level hook that runs before each test. +/// +/// The hook receives `types.TestInfo` and the current context value. +/// If it returns `Error("...")`, the test is marked `SetupFailed` +/// and the test body does not run. +/// +/// ## Example +/// +/// ```gleam +/// import dream_test/runner +/// import dream_test/types.{type TestInfo} +/// +/// fn log_test(info: TestInfo, ctx: Nil) { +/// io.println("starting " <> string.join(info.full_name, " > ")) +/// Ok(ctx) +/// } +/// +/// runner.new([suite]) +/// |> runner.before_each_test(log_test) +/// ``` +pub fn before_each_test( + builder builder: RunBuilder(ctx), + hook hook: fn(types.TestInfo, ctx) -> Result(ctx, String), +) -> RunBuilder(ctx) { + RunBuilder( + ..builder, + runner_before_each_test: list.append(builder.runner_before_each_test, [hook]), + ) +} + +/// Register a runner-level hook that runs after each test. +/// +/// The hook receives `types.TestInfo` and the current context value. +/// If it returns `Error("...")`, the test is marked `Failed`. +pub fn after_each_test( + builder builder: RunBuilder(ctx), + hook hook: fn(types.TestInfo, ctx) -> Result(Nil, String), +) -> RunBuilder(ctx) { + RunBuilder( + ..builder, + runner_after_each_test: list.append(builder.runner_after_each_test, [hook]), + ) +} + +/// Register a runner-level hook that runs before each suite. +/// +/// The hook receives `types.SuiteInfo` for the suite about to run. +/// If it returns `Error("...")`, all tests in that suite are marked `Failed` +/// and the suite is skipped. +pub fn before_each_suite( + builder builder: RunBuilder(ctx), + hook hook: fn(types.SuiteInfo) -> Result(Nil, String), +) -> RunBuilder(ctx) { + RunBuilder( + ..builder, + runner_before_each_suite: list.append(builder.runner_before_each_suite, [ + hook, + ]), + ) +} + +/// Register a runner-level hook that runs after each suite. +/// +/// The hook receives `types.SuiteInfo` for the suite that just ran. +/// If it returns `Error("...")`, a synthetic failure result is appended. +pub fn after_each_suite( + builder builder: RunBuilder(ctx), + hook hook: fn(types.SuiteInfo) -> Result(Nil, String), +) -> RunBuilder(ctx) { + RunBuilder( + ..builder, + runner_after_each_suite: list.append(builder.runner_after_each_suite, [hook]), + ) +} + +/// Register a runner-level hook that runs before all suites. +/// +/// The hook receives the full list of `SuiteInfo` after filtering. +/// If it returns `Error("...")`, all selected tests are marked `Failed` +/// and the run short-circuits. +pub fn before_all_suites( + builder builder: RunBuilder(ctx), + hook hook: fn(List(types.SuiteInfo)) -> Result(Nil, String), +) -> RunBuilder(ctx) { + RunBuilder( + ..builder, + runner_before_all_suites: list.append(builder.runner_before_all_suites, [ + hook, + ]), + ) +} + +/// Register a runner-level hook that runs after all suites. +/// +/// The hook receives the full list of `SuiteInfo` after filtering. +/// If it returns `Error("...")`, a synthetic failure result is appended. +pub fn after_all_suites( + builder builder: RunBuilder(ctx), + hook hook: fn(List(types.SuiteInfo)) -> Result(Nil, String), +) -> RunBuilder(ctx) { + RunBuilder( + ..builder, + runner_after_all_suites: list.append(builder.runner_after_all_suites, [hook]), + ) +} + /// Run all suites and return a list of `TestResult`. /// /// If a progress reporter is attached, the runner will emit progress output during @@ -623,6 +699,7 @@ pub fn filter_tests( pub fn run(builder builder: RunBuilder(ctx)) -> List(TestResult) { let selected_runs = apply_test_filter(builder.suite_runs, builder.test_filter) let total = count_total_tests(selected_runs) + let suite_infos = suite_infos_from_runs(selected_runs, []) let output = case builder.output { Some(output) -> output @@ -630,7 +707,8 @@ pub fn run(builder builder: RunBuilder(ctx)) -> List(TestResult) { } let completed0 = 0 - let completed1 = case builder.progress_reporter { + let progress_reporter0 = builder.progress_reporter + let completed1 = case progress_reporter0 { None -> completed0 Some(progress_reporter) -> { write_progress_event( @@ -642,18 +720,61 @@ pub fn run(builder builder: RunBuilder(ctx)) -> List(TestResult) { } } - let #(results, completed2) = - run_suites_with_progress( - selected_runs, - builder.config, - builder.progress_reporter, - output, - total, - completed1, - [], - ) - - case builder.progress_reporter { + let before_all_result = + run_before_all_suites_hooks(builder.runner_before_all_suites, suite_infos) + + let #(results, completed2, progress_reporter1) = case before_all_result { + Ok(_) -> + run_suites_with_hooks( + selected_runs, + suite_infos, + builder.config, + progress_reporter0, + output, + total, + completed1, + builder.runner_before_each_suite, + builder.runner_after_each_suite, + builder.runner_before_each_test, + builder.runner_after_each_test, + [], + ) + Error(message) -> { + let failures = + build_failed_results_from_suites( + suite_infos, + "before_all_suites", + message, + ) + let completed_after_failures = + emit_test_finished_progress_results( + progress_reporter0, + output, + completed1, + total, + failures, + ) + let results_with_after_all = + apply_after_all_suites_hooks( + builder.runner_after_all_suites, + suite_infos, + failures, + ) + #(results_with_after_all, completed_after_failures, progress_reporter0) + } + } + + let final_results = case before_all_result { + Ok(_) -> + apply_after_all_suites_hooks( + builder.runner_after_all_suites, + suite_infos, + results, + ) + Error(_) -> results + } + + case progress_reporter1 { None -> Nil Some(progress_reporter) -> write_progress_event( @@ -661,17 +782,507 @@ pub fn run(builder builder: RunBuilder(ctx)) -> List(TestResult) { reporter_types.RunFinished( completed: completed2, total: total, - results: results, + results: final_results, ), output, ) } - write_results_reporters(builder.results_reporters, results, output) + write_results_reporters(builder.results_reporters, final_results, output) + + maybe_exit_on_failure(builder.should_exit_on_failure, final_results) + + final_results +} + +fn suite_infos_from_runs( + suite_runs: List(SuiteRun(ctx)), + acc_rev: List(types.SuiteInfo), +) -> List(types.SuiteInfo) { + case suite_runs { + [] -> list.reverse(acc_rev) + [SuiteRun(suite: suite, config_override: _), ..rest] -> + suite_infos_from_runs(rest, [suite_info_from_suite(suite), ..acc_rev]) + } +} + +fn suite_info_from_suite(suite: TestSuite(ctx)) -> types.SuiteInfo { + let Root(seed: _seed, tree: tree) = suite + let tests_rev = collect_test_infos(tree, [], [], []) + let tests = list.reverse(tests_rev) + let source = suite_source_from_tests(tests) + types.SuiteInfo( + name: suite_name_from_tree(tree), + tests: tests, + source: source, + ) +} + +fn suite_name_from_tree(tree: Node(ctx)) -> String { + case tree { + Group(name: name, tags: _tags, children: _children) -> name + _ -> "" + } +} + +fn collect_test_infos( + node: Node(ctx), + scope: List(String), + inherited_tags: List(String), + acc_rev: List(types.TestInfo), +) -> List(types.TestInfo) { + case node { + Test( + name: name, + tags: tags, + kind: kind, + run: _run, + timeout_ms: _timeout_ms, + source: source, + ) -> { + let full_name = list.append(scope, [name]) + let effective_tags = list.append(inherited_tags, tags) + let info = + types.TestInfo( + name: name, + full_name: full_name, + tags: effective_tags, + kind: kind, + source: source, + ) + [info, ..acc_rev] + } + Group(name: name, tags: tags, children: children) -> { + let next_scope = list.append(scope, [name]) + let next_tags = list.append(inherited_tags, tags) + collect_test_infos_children(children, next_scope, next_tags, acc_rev) + } + _ -> acc_rev + } +} + +fn collect_test_infos_children( + children: List(Node(ctx)), + scope: List(String), + inherited_tags: List(String), + acc_rev: List(types.TestInfo), +) -> List(types.TestInfo) { + case children { + [] -> acc_rev + [child, ..rest] -> + collect_test_infos_children( + rest, + scope, + inherited_tags, + collect_test_infos(child, scope, inherited_tags, acc_rev), + ) + } +} + +fn suite_source_from_tests(tests: List(types.TestInfo)) -> Option(String) { + case tests { + [] -> None + [ + types.TestInfo( + source: source, + name: _name, + full_name: _full_name, + tags: _tags, + kind: _kind, + ), + ..rest + ] -> suite_source_from_tests_with_first(source, rest) + } +} + +fn suite_source_from_tests_with_first( + first: Option(String), + rest: List(types.TestInfo), +) -> Option(String) { + case first { + None -> None + Some(value) -> + case all_sources_match(value, rest) { + True -> Some(value) + False -> None + } + } +} + +fn all_sources_match(value: String, rest: List(types.TestInfo)) -> Bool { + case rest { + [] -> True + [ + types.TestInfo( + source: source, + name: _name, + full_name: _full_name, + tags: _tags, + kind: _kind, + ), + ..next + ] -> + case source { + Some(s) -> + case s == value { + True -> all_sources_match(value, next) + False -> False + } + None -> False + } + } +} + +fn run_before_all_suites_hooks( + hooks: List(fn(List(types.SuiteInfo)) -> Result(Nil, String)), + suites: List(types.SuiteInfo), +) -> Result(Nil, String) { + case hooks { + [] -> Ok(Nil) + [hook, ..rest] -> + case hook(suites) { + Ok(_) -> run_before_all_suites_hooks(rest, suites) + Error(message) -> Error(message) + } + } +} + +fn run_before_each_suite_hooks( + hooks: List(fn(types.SuiteInfo) -> Result(Nil, String)), + suite: types.SuiteInfo, +) -> Result(Nil, String) { + case hooks { + [] -> Ok(Nil) + [hook, ..rest] -> + case hook(suite) { + Ok(_) -> run_before_each_suite_hooks(rest, suite) + Error(message) -> Error(message) + } + } +} + +fn run_after_each_suite_hooks( + hooks: List(fn(types.SuiteInfo) -> Result(Nil, String)), + suite: types.SuiteInfo, +) -> Result(Nil, String) { + case hooks { + [] -> Ok(Nil) + [hook, ..rest] -> + case hook(suite) { + Ok(_) -> run_after_each_suite_hooks(rest, suite) + Error(message) -> Error(message) + } + } +} + +fn run_after_all_suites_hooks( + hooks: List(fn(List(types.SuiteInfo)) -> Result(Nil, String)), + suites: List(types.SuiteInfo), +) -> Result(Nil, String) { + case hooks { + [] -> Ok(Nil) + [hook, ..rest] -> + case hook(suites) { + Ok(_) -> run_after_all_suites_hooks(rest, suites) + Error(message) -> Error(message) + } + } +} + +fn run_suites_with_hooks( + suite_runs: List(SuiteRun(ctx)), + suite_infos: List(types.SuiteInfo), + default_config: parallel.ParallelConfig, + progress_reporter: Option(progress.ProgressReporter), + output: Output, + total: Int, + completed: Int, + before_each_suite_hooks: List(fn(types.SuiteInfo) -> Result(Nil, String)), + after_each_suite_hooks: List(fn(types.SuiteInfo) -> Result(Nil, String)), + runner_before_each_test: List(fn(types.TestInfo, ctx) -> Result(ctx, String)), + runner_after_each_test: List(fn(types.TestInfo, ctx) -> Result(Nil, String)), + acc: List(TestResult), +) -> #(List(TestResult), Int, Option(progress.ProgressReporter)) { + case suite_runs, suite_infos { + [], [] -> #(acc, completed, progress_reporter) + [SuiteRun(suite: suite, config_override: override), ..rest], + [suite_info, ..rest_infos] + -> { + let before_result = + run_before_each_suite_hooks(before_each_suite_hooks, suite_info) + case before_result { + Error(message) -> { + let failures = + build_failed_results_from_tests( + suite_info.tests, + "before_each_suite", + message, + ) + let completed_after_failures = + emit_test_finished_progress_results( + progress_reporter, + output, + completed, + total, + failures, + ) + let acc_with_failures = list.append(acc, failures) + let acc_after_after_each = + apply_after_each_suite_hooks( + after_each_suite_hooks, + suite_info, + acc_with_failures, + ) + run_suites_with_hooks( + rest, + rest_infos, + default_config, + progress_reporter, + output, + total, + completed_after_failures, + before_each_suite_hooks, + after_each_suite_hooks, + runner_before_each_test, + runner_after_each_test, + acc_after_after_each, + ) + } + Ok(_) -> { + let suite_config = suite_run_config(default_config, override) + let parallel_result = + parallel.run_root_parallel_with_reporter( + parallel.RunRootParallelWithReporterConfig( + config: suite_config, + suite: suite, + progress_reporter: progress_reporter, + write: output_out(output), + total: total, + completed: completed, + runner_before_each_test: runner_before_each_test, + runner_after_each_test: runner_after_each_test, + ), + ) + let parallel.RunRootParallelWithReporterResult( + results: results, + completed: completed_after_suite, + progress_reporter: next_progress_reporter, + ) = parallel_result + let acc_with_results = list.append(acc, results) + let acc_after_after_each = + apply_after_each_suite_hooks( + after_each_suite_hooks, + suite_info, + acc_with_results, + ) + run_suites_with_hooks( + rest, + rest_infos, + default_config, + next_progress_reporter, + output, + total, + completed_after_suite, + before_each_suite_hooks, + after_each_suite_hooks, + runner_before_each_test, + runner_after_each_test, + acc_after_after_each, + ) + } + } + } + _, _ -> #(acc, completed, progress_reporter) + } +} + +fn apply_after_each_suite_hooks( + hooks: List(fn(types.SuiteInfo) -> Result(Nil, String)), + suite_info: types.SuiteInfo, + results: List(TestResult), +) -> List(TestResult) { + case run_after_each_suite_hooks(hooks, suite_info) { + Ok(_) -> results + Error(message) -> + list.append(results, [ + after_each_suite_failure_result(suite_info, message), + ]) + } +} + +fn apply_after_all_suites_hooks( + hooks: List(fn(List(types.SuiteInfo)) -> Result(Nil, String)), + suites: List(types.SuiteInfo), + results: List(TestResult), +) -> List(TestResult) { + case run_after_all_suites_hooks(hooks, suites) { + Ok(_) -> results + Error(message) -> + list.append(results, [after_all_suites_failure_result(message)]) + } +} + +fn build_failed_results_from_suites( + suites: List(types.SuiteInfo), + operator: String, + message: String, +) -> List(TestResult) { + build_failed_results_from_suites_rev(suites, operator, message, []) + |> list.reverse() +} + +fn build_failed_results_from_suites_rev( + suites: List(types.SuiteInfo), + operator: String, + message: String, + acc_rev: List(TestResult), +) -> List(TestResult) { + case suites { + [] -> acc_rev + [suite, ..rest] -> + build_failed_results_from_suites_rev( + rest, + operator, + message, + build_failed_results_from_tests_rev( + suite.tests, + operator, + message, + acc_rev, + ), + ) + } +} + +fn build_failed_results_from_tests( + tests: List(types.TestInfo), + operator: String, + message: String, +) -> List(TestResult) { + build_failed_results_from_tests_rev(tests, operator, message, []) + |> list.reverse() +} + +fn build_failed_results_from_tests_rev( + tests: List(types.TestInfo), + operator: String, + message: String, + acc_rev: List(TestResult), +) -> List(TestResult) { + case tests { + [] -> acc_rev + [test_info, ..rest] -> + build_failed_results_from_tests_rev(rest, operator, message, [ + failed_result_for_test_info(test_info, operator, message), + ..acc_rev + ]) + } +} + +fn failed_result_for_test_info( + info: types.TestInfo, + operator: String, + message: String, +) -> TestResult { + let types.TestInfo( + name: name, + full_name: full_name, + tags: tags, + kind: kind, + source: _source, + ) = info + types.TestResult( + name: name, + full_name: full_name, + status: Failed, + duration_ms: 0, + tags: tags, + failures: [hook_failure(operator, message)], + kind: kind, + ) +} + +fn after_all_suites_failure_result(message: String) -> TestResult { + types.TestResult( + name: "", + full_name: [""], + status: Failed, + duration_ms: 0, + tags: [], + failures: [hook_failure("after_all_suites", message)], + kind: Unit, + ) +} - maybe_exit_on_failure(builder.should_exit_on_failure, results) +fn after_each_suite_failure_result( + suite_info: types.SuiteInfo, + message: String, +) -> TestResult { + let types.SuiteInfo(name: name, tests: _tests, source: _source) = suite_info + types.TestResult( + name: "", + full_name: [name, ""], + status: Failed, + duration_ms: 0, + tags: [], + failures: [hook_failure("after_each_suite", message)], + kind: Unit, + ) +} - results +fn hook_failure(operator: String, message: String) -> AssertionFailure { + types.AssertionFailure(operator: operator, message: message, payload: None) +} + +fn emit_test_finished_progress( + progress_reporter: Option(progress.ProgressReporter), + output: Output, + completed: Int, + total: Int, + result: TestResult, +) -> Int { + let next_completed = completed + 1 + case progress_reporter { + None -> next_completed + Some(reporter) -> { + write_progress_event( + reporter, + reporter_types.TestFinished( + completed: next_completed, + total: total, + result: result, + ), + output, + ) + next_completed + } + } +} + +fn emit_test_finished_progress_results( + progress_reporter: Option(progress.ProgressReporter), + output: Output, + completed: Int, + total: Int, + results: List(TestResult), +) -> Int { + case results { + [] -> completed + [result, ..rest] -> + emit_test_finished_progress_results( + progress_reporter, + output, + emit_test_finished_progress( + progress_reporter, + output, + completed, + total, + result, + ), + total, + rest, + ) + } } fn default_output() -> Output { @@ -744,15 +1355,23 @@ fn filter_node( predicate: fn(TestInfo) -> Bool, ) -> #(Option(Node(ctx)), Bool) { case node { - Test(name: name, tags: tags, kind: kind, run: run, timeout_ms: timeout_ms) -> { + Test( + name: name, + tags: tags, + kind: kind, + run: run, + timeout_ms: timeout_ms, + source: source, + ) -> { let full_name = list.append(scope, [name]) let effective_tags = list.append(inherited_tags, tags) let info = - TestInfo( + types.TestInfo( name: name, full_name: full_name, tags: effective_tags, kind: kind, + source: source, ) case predicate(info) { True -> #( @@ -762,6 +1381,7 @@ fn filter_node( kind: kind, run: run, timeout_ms: timeout_ms, + source: source, )), True, ) @@ -828,49 +1448,6 @@ fn filter_children( } } -fn run_suites_with_progress( - suite_runs: List(SuiteRun(ctx)), - default_config: parallel.ParallelConfig, - progress_reporter: Option(progress.ProgressReporter), - output: Output, - total: Int, - completed: Int, - acc: List(TestResult), -) -> #(List(TestResult), Int) { - case suite_runs { - [] -> #(acc, completed) - [SuiteRun(suite: suite, config_override: override), ..rest] -> { - let suite_config = suite_run_config(default_config, override) - let parallel_result = - parallel.run_root_parallel_with_reporter( - parallel.RunRootParallelWithReporterConfig( - config: suite_config, - suite: suite, - progress_reporter: progress_reporter, - write: output_out(output), - total: total, - completed: completed, - ), - ) - let parallel.RunRootParallelWithReporterResult( - results: results, - completed: completed_after_suite, - progress_reporter: next_progress_reporter, - ) = parallel_result - - run_suites_with_progress( - rest, - default_config, - next_progress_reporter, - output, - total, - completed_after_suite, - list.append(acc, results), - ) - } - } -} - fn suite_run_config( default_config: parallel.ParallelConfig, override: Option(parallel.ParallelConfig), diff --git a/src/dream_test/types.gleam b/src/dream_test/types.gleam index 03b6516..dabc75a 100644 --- a/src/dream_test/types.gleam +++ b/src/dream_test/types.gleam @@ -324,6 +324,44 @@ pub type TestResult { ) } +/// Lightweight test identity information. +/// +/// This is used by the runner for filtering and hook metadata. +/// +/// ## Fields +/// +/// - `name`: the leaf test name (the `it("...")` label) +/// - `full_name`: group path + test name (deterministic, good for identifiers) +/// - `tags`: effective tags (includes inherited group tags) +/// - `kind`: `TestKind` (Unit, Integration, or GherkinScenario(id)) +/// - `source`: best-effort origin string (module name or `.feature` path) +/// +/// ### Source behavior +/// +/// - Unit discovery uses the **module name** +/// - Gherkin discovery uses the **.feature file path** +/// - Manually constructed suites use `None` +pub type TestInfo { + TestInfo( + name: String, + full_name: List(String), + tags: List(String), + kind: TestKind, + source: Option(String), + ) +} + +/// Suite metadata for runner-level hooks. +/// +/// ## Fields +/// +/// - `name`: top-level suite/group name +/// - `tests`: deterministic list of `TestInfo` values that will execute +/// - `source`: best-effort origin string (see `TestInfo.source`) +pub type SuiteInfo { + SuiteInfo(name: String, tests: List(TestInfo), source: Option(String)) +} + /// A complete test suite in the unified execution model. /// /// A `Root(context)` stores the initial `seed` context value and the top-level @@ -346,6 +384,8 @@ pub type Node(context) { kind: TestKind, run: fn(context) -> Result(AssertionResult, String), timeout_ms: Option(Int), + /// Best-effort origin string for this test (module name or .feature path). + source: Option(String), ) /// Group-scoped hooks. diff --git a/src/dream_test/unit.gleam b/src/dream_test/unit.gleam index 9e4e7c5..e20942d 100644 --- a/src/dream_test/unit.gleam +++ b/src/dream_test/unit.gleam @@ -255,6 +255,7 @@ pub fn it( kind: Unit, run: fn(_nil: Nil) { run() }, timeout_ms: None, + source: None, ) } @@ -327,13 +328,21 @@ pub fn skip( ) -> UnitNode { let node = it(name, run) case node { - Test(name: name, tags: tags, kind: kind, run: _run, timeout_ms: timeout_ms) -> + Test( + name: name, + tags: tags, + kind: kind, + run: _run, + timeout_ms: timeout_ms, + source: source, + ) -> Test( name: name, tags: tags, kind: kind, run: skipped_test_run, timeout_ms: timeout_ms, + source: source, ) other -> other } @@ -737,8 +746,15 @@ pub fn with_tags(node node: UnitNode, tags tags: List(String)) -> UnitNode { case node { Group(name, _, children) -> Group(name: name, tags: tags, children: children) - Test(name, _, kind, run, timeout_ms) -> - Test(name: name, tags: tags, kind: kind, run: run, timeout_ms: timeout_ms) + Test(name, _, kind, run, timeout_ms, source) -> + Test( + name: name, + tags: tags, + kind: kind, + run: run, + timeout_ms: timeout_ms, + source: source, + ) other -> other } } diff --git a/src/dream_test/unit_context.gleam b/src/dream_test/unit_context.gleam index a74c221..5620dfa 100644 --- a/src/dream_test/unit_context.gleam +++ b/src/dream_test/unit_context.gleam @@ -243,7 +243,14 @@ pub fn it( name name: String, run run: fn(context) -> Result(AssertionResult, String), ) -> ContextNode(context) { - Test(name: name, tags: [], kind: Unit, run: run, timeout_ms: None) + Test( + name: name, + tags: [], + kind: Unit, + run: run, + timeout_ms: None, + source: None, + ) } /// Define a skipped context-aware test. @@ -310,6 +317,7 @@ pub fn skip( kind: Unit, run: skipped_test_run, timeout_ms: None, + source: None, ) } @@ -396,8 +404,15 @@ pub fn with_tags( case node { Group(name, _, children) -> Group(name: name, tags: tags, children: children) - Test(name, _, kind, run, timeout_ms) -> - Test(name: name, tags: tags, kind: kind, run: run, timeout_ms: timeout_ms) + Test(name, _, kind, run, timeout_ms, source) -> + Test( + name: name, + tags: tags, + kind: kind, + run: run, + timeout_ms: timeout_ms, + source: source, + ) other -> other } } diff --git a/test/dream_test/gherkin/parser_api_test.gleam b/test/dream_test/gherkin/parser_api_test.gleam index 3f12e5f..17230b9 100644 --- a/test/dream_test/gherkin/parser_api_test.gleam +++ b/test/dream_test/gherkin/parser_api_test.gleam @@ -4,7 +4,7 @@ import dream_test/gherkin/types as gtypes import dream_test/matchers.{fail_with} import dream_test/types as test_types import dream_test/unit.{describe, it} -import gleam/option.{None} +import gleam/option.{None, Some} pub fn tests() { describe("dream_test/gherkin/parser", [ @@ -24,6 +24,7 @@ pub fn tests() { case result { Ok(gtypes.Feature( name: "Demo", + source: None, description: None, tags: ["smoke"], background: None, @@ -58,14 +59,15 @@ pub fn tests() { let _ = file.write(path, content) case parser.parse_file(path) { - Ok(gtypes.Feature( - name: "From File", - description: _, - tags: _, - background: _, - scenarios: [_], - )) -> Ok(test_types.AssertionOk) - Ok(_) -> Ok(fail_with("unexpected parse result")) + Ok(feature) -> + case feature.source { + Some(source) -> + case source == path && feature.name == "From File" { + True -> Ok(test_types.AssertionOk) + False -> Ok(fail_with("unexpected parse result")) + } + None -> Ok(fail_with("expected source to be set")) + } Error(msg) -> Ok(fail_with(msg)) } }), diff --git a/test/dream_test/runner_hooks_test.gleam b/test/dream_test/runner_hooks_test.gleam new file mode 100644 index 0000000..f74ba33 --- /dev/null +++ b/test/dream_test/runner_hooks_test.gleam @@ -0,0 +1,384 @@ +import dream_test/discover +import dream_test/file +import dream_test/gherkin/discover as gherkin_discover +import dream_test/gherkin/steps.{type StepContext, new, step} +import dream_test/matchers.{be_equal, fail_with, or_fail_with, should} +import dream_test/runner +import dream_test/types.{ + type AssertionResult, type TestResult, AssertionFailure, AssertionOk, Failed, + Passed, SetupFailed, Unit, +} +import dream_test/unit.{describe, group, it, with_tags} +import dream_test/unit_context.{ + after_all, after_each, before_all, before_each, describe as ctx_describe, + it as ctx_it, +} +import gleam/option.{type Option, None, Some} + +pub fn tests() { + describe("runner hooks", [ + it("runner per-test hooks wrap suite hooks", fn() { + let log_path = "test/fixtures/file/temp/runner_hooks_order.txt" + + let suite = + ctx_describe("root", Nil, [ + before_all(fn(_ctx: Nil) { write_string(log_path, "") }), + before_each(fn(_ctx: Nil) { append_line(log_path, "suite_before\n") }), + after_each(fn(_ctx: Nil) { append_line(log_path, "suite_after\n") }), + ctx_it("passes", fn(_ctx: Nil) { Ok(AssertionOk) }), + ]) + + let results = + runner.new([suite]) + |> runner.max_concurrency(1) + |> runner.before_each_test(fn(_info, ctx: Nil) { + log_and_keep_context(log_path, "runner_before\n", ctx) + }) + |> runner.after_each_test(fn(_info, _ctx: Nil) { + append_line(log_path, "runner_after\n") + }) + |> runner.run() + + let assert [_r] = results + + file.read(log_path) + |> should + |> be_equal(Ok("runner_before\nsuite_before\nsuite_after\nrunner_after\n")) + |> or_fail_with("runner hooks should wrap suite hooks") + }), + + it("runner before_each_test sees full_name, tags, and kind", fn() { + let suite = + describe("root", [ + group("group", [ + it("leaf", fn() { Ok(AssertionOk) }) + |> with_tags(["leaf"]), + ]) + |> with_tags(["group"]), + ]) + + let results = + runner.new([suite]) + |> runner.before_each_test(fn(info, ctx: Nil) { + case + info.name == "leaf" + && info.full_name == ["root", "group", "leaf"] + && info.tags == ["group", "leaf"] + && info.kind == Unit + { + True -> Ok(ctx) + False -> Error("metadata mismatch") + } + }) + |> runner.run() + + let assert [r1] = results + + let _ = + r1.status + |> should + |> be_equal(Passed) + |> or_fail_with("expected metadata to be correct") + }), + + it("runner per-suite hooks wrap suite before_all/after_all", fn() { + let log_path = "test/fixtures/file/temp/runner_hooks_suite_order.txt" + let _ = write_string(log_path, "") + + let suite = + ctx_describe("suite_one", Nil, [ + before_all(fn(_ctx: Nil) { + append_line(log_path, "suite_before_all\n") + }), + after_all(fn(_ctx: Nil) { append_line(log_path, "suite_after_all\n") }), + ctx_it("passes", fn(_ctx: Nil) { Ok(AssertionOk) }), + ]) + + let results = + runner.new([suite]) + |> runner.max_concurrency(1) + |> runner.before_each_suite(fn(_suite) { + append_line(log_path, "runner_before_suite\n") + }) + |> runner.after_each_suite(fn(_suite) { + append_line(log_path, "runner_after_suite\n") + }) + |> runner.run() + + let assert [_r] = results + + file.read(log_path) + |> should + |> be_equal(Ok( + "runner_before_suite\nsuite_before_all\nsuite_after_all\nrunner_after_suite\n", + )) + |> or_fail_with("runner suite hooks should wrap suite hooks") + }), + + it( + "before_all_suites failure skips tests and records after_all_suites failure", + fn() { + let suite_one = + describe("suite_one", [ + it("a", fn() { panic as "should not run" }), + ]) + + let suite_two = + describe("suite_two", [ + it("b", fn() { panic as "should not run" }), + ]) + + let results = + runner.new([suite_one, suite_two]) + |> runner.before_all_suites(fn(_suites) { Error("nope") }) + |> runner.after_all_suites(fn(_suites) { Error("cleanup") }) + |> runner.run() + + let assert [r1, r2, r3] = results + + let _ = + #(r1.status, first_failure_operator(r1)) + |> should + |> be_equal(#(Failed, Some("before_all_suites"))) + |> or_fail_with("first test should fail due to before_all_suites") + + let _ = + #(r2.status, first_failure_operator(r2)) + |> should + |> be_equal(#(Failed, Some("before_all_suites"))) + |> or_fail_with("second test should fail due to before_all_suites") + + let _ = + #(r3.name, first_failure_operator(r3)) + |> should + |> be_equal(#("", Some("after_all_suites"))) + |> or_fail_with("after_all_suites failure should be reported") + }, + ), + + it( + "before_each_suite failure skips suite tests and runs after_each_suite", + fn() { + let suite_one = + describe("suite_one", [ + it("boom", fn() { panic as "should not run" }), + ]) + + let suite_two = + describe("suite_two", [ + it("ok", fn() { Ok(AssertionOk) }), + ]) + + let results = + runner.new([suite_one, suite_two]) + |> runner.before_each_suite(fn(suite) { + case suite.name == "suite_one" { + True -> Error("boom") + False -> Ok(Nil) + } + }) + |> runner.after_each_suite(fn(suite) { + case suite.name == "suite_one" { + True -> Error("cleanup") + False -> Ok(Nil) + } + }) + |> runner.run() + + let assert [r1, r2, r3] = results + + let _ = + #(r1.status, first_failure_operator(r1)) + |> should + |> be_equal(#(Failed, Some("before_each_suite"))) + |> or_fail_with("suite_one test should fail due to before_each_suite") + + let _ = + #(r2.name, first_failure_operator(r2)) + |> should + |> be_equal(#("", Some("after_each_suite"))) + |> or_fail_with("after_each_suite failure should be reported") + + let _ = + r3.status + |> should + |> be_equal(Passed) + |> or_fail_with("suite_two test should pass") + }, + ), + + it( + "before_each_test failure yields SetupFailed under parallel execution", + fn() { + let suite = + describe("parallel_suite", [ + it("ok", fn() { Ok(AssertionOk) }), + it("fail", fn() { Ok(AssertionOk) }), + ]) + + let results = + runner.new([suite]) + |> runner.max_concurrency(2) + |> runner.before_each_test(fn(info, ctx: Nil) { + case info.name == "fail" { + True -> Error("nope") + False -> Ok(ctx) + } + }) + |> runner.run() + + case find_result_by_name(results, "fail") { + Some(result) -> + result.status + |> should + |> be_equal(SetupFailed) + |> or_fail_with("failed test should be SetupFailed") + None -> Ok(fail_with("expected test named 'fail'")) + } + }, + ), + + it("after_each_test failure marks test Failed", fn() { + let suite = + describe("after_suite", [ + it("after", fn() { Ok(AssertionOk) }), + ]) + + let results = + runner.new([suite]) + |> runner.after_each_test(fn(info, _ctx: Nil) { + case info.name == "after" { + True -> Error("boom") + False -> Ok(Nil) + } + }) + |> runner.run() + + let assert [r1] = results + let _ = + #(r1.status, first_failure_operator(r1)) + |> should + |> be_equal(#(Failed, Some("after_each_test"))) + |> or_fail_with("after_each_test failure should mark test Failed") + }), + + it("unit discovery populates source with module name", fn() { + let discovery = discover.tests("dream_test/runner_api_test.gleam") + + case discover.list_modules(discovery) { + Ok([module_name]) -> { + let log_path = + "test/fixtures/file/temp/runner_hooks_source_module.txt" + let _ = write_string(log_path, "") + let suites = discover.to_suites(discovery) + + let _ = + runner.new(suites) + |> runner.before_each_suite(fn(suite) { + case suite.source { + Some(value) -> write_string(log_path, value) + None -> write_string(log_path, "") + } + }) + |> runner.run() + + file.read(log_path) + |> should + |> be_equal(Ok(module_name)) + |> or_fail_with("suite source should match module name") + } + _ -> Ok(fail_with("expected one discovered module")) + } + }), + + it("gherkin discovery populates source with .feature path", fn() { + let feature_path = "test/fixtures/file/temp/runner_hooks_source.feature" + let content = + "Feature: Source\n" <> "\n" <> "Scenario: One\n" <> " Given ok\n" + + let _ = file.write(feature_path, content) + + let registry = + new() + |> step("ok", step_ok) + + let suite = + gherkin_discover.features(feature_path) + |> gherkin_discover.with_registry(registry) + |> gherkin_discover.to_suite("gherkin_source") + + let log_path = "test/fixtures/file/temp/runner_hooks_source_gherkin.txt" + let _ = write_string(log_path, "") + + let _ = + runner.new([suite]) + |> runner.before_each_test(fn(info, ctx: Nil) { + case info.source { + Some(value) -> + case write_string(log_path, value) { + Ok(_) -> Ok(ctx) + Error(e) -> Error(e) + } + None -> Error("missing source") + } + }) + |> runner.run() + + file.read(log_path) + |> should + |> be_equal(Ok(feature_path)) + |> or_fail_with("gherkin test source should be .feature path") + }), + ]) +} + +fn append_line(path: String, line: String) -> Result(Nil, String) { + case file.read(path) { + Ok(existing) -> write_string(path, existing <> line) + Error(e) -> Error(file.error_to_string(e)) + } +} + +fn write_string(path: String, content: String) -> Result(Nil, String) { + case file.write(path, content) { + Ok(_) -> Ok(Nil) + Error(e) -> Error(file.error_to_string(e)) + } +} + +fn log_and_keep_context( + path: String, + line: String, + ctx: Nil, +) -> Result(Nil, String) { + case append_line(path, line) { + Ok(_) -> Ok(ctx) + Error(e) -> Error(e) + } +} + +fn find_result_by_name( + results: List(TestResult), + name: String, +) -> Option(TestResult) { + case results { + [] -> None + [r, ..rest] -> + case r.name == name { + True -> Some(r) + False -> find_result_by_name(rest, name) + } + } +} + +fn first_failure_operator(result: TestResult) -> Option(String) { + case result.failures { + [] -> None + [AssertionFailure(operator: operator, message: _message, payload: _), ..] -> + Some(operator) + } +} + +fn step_ok(_context: StepContext) -> Result(AssertionResult, String) { + Ok(AssertionOk) +} From 58979559dc968889f29fdec6b7690917ae52d290 Mon Sep 17 00:00:00 2001 From: Dara Rockwell Date: Sun, 1 Feb 2026 15:33:24 -0700 Subject: [PATCH 2/2] chore: update CI Gleam version ## Why This Change Was Made - CI was failing because gleam_stdlib now requires Gleam >= 1.14.0, but the workflow was pinned to 1.13.0. ## What Was Changed - Bumped the GitHub Actions workflow to install Gleam 1.14.0. ## Note to Future Engineer - If CI suddenly breaks with a version mismatch, check this pin before blaming the universe. - Yes, another version bump. You're welcome, future you. --- .github/workflows/ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 2b65ae7..519367b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -24,7 +24,7 @@ jobs: uses: erlef/setup-beam@v1 with: otp-version: "28" - gleam-version: "1.13.0" + gleam-version: "1.14.0" rebar3-version: "3" - name: Download dependencies