Skip to content

mise recipe for benchmark baseline comparison and creation is broken #762

@RobertJacobsonCDC

Description

@RobertJacobsonCDC

The mise recipes for the bench:create and bench:compare tasks are not expanding as expected. When invoking the task using key=value syntax (for example, name=sample baseline=main), the resulting command passed the literal strings "name=sample" and "baseline=main" to cargo bench, rather than just "sample" and "main".

The current task definitions look like this:

[tasks.'bench:create']
description = "Compare benchmarks against a Criterion baseline"
run = "cargo bench -p ixa-bench {{arg(name='name', default='')}} -- --save-baseline {{arg(name='baseline')}}"

[tasks.'bench:compare']
description = "Compare benchmarks against a Criterion baseline"
run = [
    "cargo bench -p ixa-bench {{arg(name='name', default='')}} -- --baseline {{arg(name='baseline')}}",
    "cargo run -q -p ixa-bench --bin check_criterion_regressions"
]

Running:

mise run bench:create name=sample baseline=main

expands to:

cargo bench -p ixa-bench 'name=sample' -- --save-baseline 'baseline=main'

and similarly for bench:compare. The intended behavior, however, is to run:

cargo bench -p ixa-bench 'sample' -- --save-baseline 'main'

The root cause appears to be that the tasks use {{arg(...)}} without defining explicit args. When arguments are supplied as key=value, mise treats them as literal positional strings rather than parsing them into named arguments, so the template receives the entire key=value text.

The fix is to define explicit task arguments and reference them directly. For example:

[tasks.'bench:create']
description = "Create a Criterion baseline"
args = [
  { name = "filter", optional = true },
  { name = "baseline" }
]
run = "cargo bench -p ixa-bench {{filter}} -- --save-baseline {{baseline}}"

[tasks.'bench:compare']
description = "Compare benchmarks against a Criterion baseline"
args = [
  { name = "filter", optional = true },
  { name = "baseline" }
]
run = [
  "cargo bench -p ixa-bench {{filter}} -- --baseline {{baseline}}",
  "cargo run -q -p ixa-bench --bin check_criterion_regressions"
]

With this change, the correct usage becomes:

mise run bench:create sampl main
mise run bench:compare sampl main

which expands to the intended Cargo commands.

Metadata

Metadata

Assignees

No one assigned

    Labels

    small taskGood for new contributors

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions