RFC: Test runner foundations

[ivov]

Tip

This is the first iteration I have in mind for a test runner, in the form of user-facing documentation. Scoped as foundations: writing and running test files and functions, assertions, and the test context. Later iterations might tackle test coverage, test labels, snapshot testing, etc. as needed. Open for feedback!

Testing

Lisette's test runner invokes #[test] functions in test files via the lis test command.

lis test [options]

Run the project's tests.

Options:
    --filter, -f <pattern>    Run only tests whose name matches
    --go-flags "<flags>"      Pass flags through to `go test`

Examples:
    lis test                           Run all tests in project
    lis test -f parse                  Only test names containing "parse"
    lis test --go-flags "-failfast"    Stop at the first failing test

On success:

· Testing `my_project` v0.1.0

  math
    ✓ addition
    ✓ subtraction
    ✓ multiplication
    ✓ division

✓ 4 passed (12ms)

On failure:

· Testing `my_project` v0.1.0

  math
    ✗ addition
    ✓ subtraction
    ✓ multiplication
    ✓ division

  [fail] addition
    ╭─[math/math.test.lis:5:8]
  5 │ assert 2 + 2 == 5
    ·        ────┬─────
    ·            ╰── actual: 4, expected: 5
    ╰────

✖ 1 failed, 3 passed (12ms)

Test types

All test files are suffixed with .test.lis

Internal tests sit in the same module as the file under test, so they can access private symbols in the module.

External tests sit in a tests directory at project root, so they can only exercise modules through public APIs. Integration tests typically live here.

src/
├── main.lis
└── math/
    ├── math.lis         # file under test
    └── math.test.lis    # internal tests
tests/
└── arithmetic.test.lis  # external tests

• lis test runs all tests, internal and external.
• lis build excludes all test files from the binary.
• lis check includes all files, production and tests.

Test functions

In a test file, a function marked #[test] will be found and invoked by the test runner. A test function usually takes no parameters and omits the return type annotation.

#[test]
fn addition() {
  // ...
}

Optionally, a test can carry a title and description:

// csv.test.lis

/// Trims surrounding whitespace before splitting each field.
#[test("splits and trims CSV fields")]
fn splits_csv() {
  // ...
}

In the report, the title replaces the function name, and the description follows beneath:

· Testing `my_project` v0.1.0

  csv
    ✓ splits and trims CSV fields
      Trims surrounding whitespace before splitting each field.

✓ 1 passed (7ms)

The --filter flag in lis test matches strings in both function names and function titles.

Test assertions

In a test function, assert checks if a boolean expression is true or fails the test.

#[test]
fn basics() {
  assert is_email("name@email.com") // ok
  assert slugify("Hello World") == "hello-world" // ok

  let a = 1
  let b = 2
  assert a == b // fail
}

Diagnostic on failure:

[fail] basics
  ╭─[math/math.test.lis:5:10]
5 │   assert a == b
  ·          ──┬──
  ·            ╰── actual: 1, expected: 2
  ╰────

For types that cannot be natively compared, use the #[equality] attribute and the equals method:

#[equality]
struct Order {
  id: int,
  tags: Slice<string>,
}

#[test]
fn orders_match() {
  let a = Order { id: 1, tags: ["a"] }
  let b = Order { id: 9, tags: ["z"] }
  assert a.equals(b)
}

Diagnostic on failure:

[fail] orders_match
  ╭─[orders/orders.test.lis:7:10]
7 │   assert a.equals(b)
  ·          ─────┬────
  ·               ╰── assertion failed
  ╰────
    actual:   `a` is `Order { id: 1, tags: ["a"] }`
    expected: `b` is `Order { id: 9, tags: ["z"] }`

Use let assert to assert by pattern matching:

#[test]
fn parses_header() {
  let bytes = [0x02, 0x00]
  let assert Ok(h) = parse_header(bytes) // mismatch fails the test
  assert h.version == 2
}

To assert that an operation panics, recover from it and match the Err:

#[test]
fn panics_on_out_of_bounds() {
  let xs = [1, 2, 3]
  let assert Err(_) = recover { xs[9] } // non-panic fails the test
}

Use Result to assert via the ? operator:

#[test]
fn round_trips() -> Result<(), error> {
  let point = Point { x: 1, y: 2 }
  let bytes = encode(point)? // `Err` fails the test
  let restored = decode(bytes)? // `Err` fails the test
  assert restored == point
  Ok(())
}

Test context

Optionally, Go's testing.T is available as a t parameter in every test function. t is of type TestContext (from Lisette's prelude) and works like Go's testing.T.

fn cases() -> Slice<Case> {
  [
    Case { name: "a", input: 1, expected: 2 },
    Case { name: "b", input: 2, expected: 4 },
  ]
}

#[test]
fn basics(t: TestContext) {
  for case in cases() {   
    t.run(case.name, || {
      assert compute(case.input) == case.expected
    })
  }
}

t is also available as a lambda parameter. In the following example, the test function's t creates subtests, and each lambda's t makes the subtest run concurrently.

#[test]
fn basics(t: TestContext) {
  for case in cases() {
    t.run(case.name, |t| {
      t.parallel()
      assert compute(case.input) == case.expected
    })
  }
}

[ekisu]

Would code located in .test.lis files be able to "see" code in other .test.lis files? Say I had something like the following structure:

src/
├── main.lis
└── models/
    └── common.test.lis # Some common helpers for testing, like `cleanup_db`
    ├── order.lis
    └── order.test.lis
    ├── user.lis
    └── user.test.lis

Would the order/user tests be able to see the common.test.lis file?

[ivov]

Exactly! The same rule that applies today continues to apply. All files in a module are visible to each other, so tests in order.test.lis and user.test.lis can call symbols from common.test.lis. The compiler will enforce that production files do not use symbols defined in test files.

[stormpat]

This looks awesome!

Im curious, is assert just an placeholder in this proposal, or an actual new keyword?

Go historically lacks assertions, so you end up with just having if <bool> checks. So any builtin assertions would be a really nice DX improvement, specially for testing. This also leads to the question, will assert be scoped only for tests, or also available in normal lis code? (like previously the value enum were only a typedef thing).

The assert failure diagnostic looks good (i like the actual vs expected message).

But having said that, one thing to note is that the compiler wont be able to force the fact that the assert is positional. Meaning left=actual and right=expected can be a lie.

Without knowing the assert implementation, these are just cases that came to mind

The user reversed the order

assert 5 == add(2, 2)

would yield:

actual: 5, expected: 4

Also, it can lead to confusion in computed values:

assert cache.length() == result.length()

would yield:

actual: 3, expected: 4 // which one is actually expected?

Also this can be confusing:

assert user.id != order.user_id

would yield:

actual: 7, expected: 7

So i propose changing the terminology to something more general like left and right. This would pretty much cover all the cases above.

As an example:

assert 5 == add(2, 2)

[fail] addition
  ╭─[math.test.lis:3:10]
3 │   assert 5 == add(2, 2)
  ·          ──────┬────────
  ·                ╰── this is false
  ╰────
    left:  `5`         = 5
    right: `add(2, 2)` = 4

[ivov]

Re: assert - Yes, this'd be a new reserved keyword, not a function. The diagnostic wants the operands and their source text, but a function would operate on the already evaluated boolean, so a function would have to be multi-param like assert_eq(a, b) else we'd be back to the explicit Go style:

got := add(2, 2)
want := 4
if got != want {
    t.Errorf("got %v, want %v", got, want)
}

The expression-based form fits Lisette better. Initially assert is intended for tests only, but we could later consider production assertions if need be. I considered also assert <expr> as "custom message" but descoped it for now.

Re: vocabulary - Great point! The order of actual/expected is a convention the compiler cannot enforce, and across languages and testing libs the order is inconsistent. Left/right is a better fit for expression-based assertions.

[zapu]

Just as a data point here, I really enjoy writing golang tests using https://pkg.go.dev/github.com/stretchr/testify/require

[rsyring]

I like assert as an expression. In the Python world, two big players are pytest and the standard lib's unittest. Pytest overrides the assert statements to give helpful output when the assert fails. I find this to be a really productive enhancement because I can write the test very concise but still not sacrifice useful diagnostic information.

Having to type Something.Equals(...) or Something.NotEquals(..), with all the possible variants, to me, feels like a lot of extra code and noise for so little value. assert x == y or assert x !=y is just so clean IMO, especially when the test runner is going to enhance that failure to help debugging.