[Feature]: Additional configuration options for fully qualified test case title

[Aaron-Pool]

The Problem

My team often writes a significant amount of detail the Rule definitions in our feature files. We keep our Feature and Scenario descriptions very succinct and instead put business rules underneath the Rule headings. However, since each Rule generates a describe(...textContents) block, it makes the cli output virtually unreadable and very hard to parse. Is it possible to have some configuration option to manage how a test suite is structured for a given feature file hierarchy.

Proposed Solution

I was thinking you could expand the concept currently seen in examplesTitleFormat? to be just titleFormat?, and it would take the following object:

interface TitleFormat {
  feature?: string | null,
  rule?: string | null,
  scenarioOutline?: string | null, 
  scenario?: string,
  example?: string,
}

This would allow anything that maps to a describe block to be provided null, which would mean that it would map to an anonymous describe group.

Anything not provided would use the default behavior. To clarify, neither scenario nor example could be provided null, as playwright does not allow the test block itself to be anonymous.

This way I could accomplish what I wanted by providing:

{
  titleFormat: {
    rule: null,
  }
}

But if another company instead used the Feature line to hold business requirements and the Rule/Scenario to distinguish to test case, they could instead do:

{
  titleFormat: {
    feature: null
  }
}

Additional Context

No response

Priority Level

Medium

[vitalets]

@Aaron-Pool could you show an example feature file that you use, and the actual / expected output?

[Aaron-Pool]

@vitalets
Sure! So, currently a feature file like this:

Feature: Resource Links
  
  Rule: Resource Links must be in the specifiied format, otherwise inline error message will be displayed

    Scenario: Activating the format validation
      When I provide the "URL" field with "<url>"
      Then the link row <error> show an error of "URL is not valid"

      Examples:
        | url               | error    |
        | test.org/         | will     |
        | https://test.org/ | will not |

Will generate the following test blocks:

test.describe("Resource Links", () => {
  test.describe("Resource Links must be in the specifiied format, otherwise inline error message will be displayed", () => {
    test("Activating the format validation", { tag: ["@done"] }, async ({ When, page, section, Then }) => {
      await When("I provide the \"URL\" field with \"test.org/\"", null, { page, section });
      await Then("the link row will show an error of \"URL is not valid\"");
    });
  });
});

And that causes the line reporter in the terminal to display a super long message as each test executes and its pretty difficult to read/follow.

I would like to be able to configure it to instead generate:

test.describe("Resource Links", () => {
  test.describe(() => {
    test("Activating the format validation", { tag: ["@done"] }, async ({ When, page, section, Then }) => {
      await When("I provide the \"URL\" field with \"test.org/\"", null, { page, section });
      await Then("the link row will show an error of \"URL is not valid\"");
    });
  });
});

The middle describe block is an anonymous (unnamed) describe, which is supported by the playwright test runner.

[vitalets]

Got it.
Do you need this long definition appear in the other reports? Or other reasons of using Rule over plain gherkin comments?

I mean you can write this way:

Feature: Resource Links
  
    Resource Links must be in the specified format, otherwise inline error message will be displayed

    Scenario: Activating the format validation
      When I provide the "URL" field with "<url>"
      Then the link row <error> show an error of "URL is not valid"

      Examples:
        | url               | error    |
        | test.org/         | will     |
        | https://test.org/ | will not |

Then it will be generated without extra nesting at all:

test.describe("Resource Links", () => {
  test("Activating the format validation", { tag: ["@done"] }, async ({ When, page, section, Then }) => {
    await When("I provide the \"URL\" field with \"test.org/\"", null, { page, section });
    await Then("the link row will show an error of \"URL is not valid\"");
  });
});

[Aaron-Pool]

Yeah, we have a few reasons:

1. Rules allow Background to be included, which we frequently do.
2. for people actually reading the gherkin as a requirements document, the Rule groupings specify the actual "requirement" that is being described by the scenario, so having them be highlighted an emphasized is important.

So, yes, the use of Rule is important from a documentation perspective, but from the perspective of a dev viewing the output of tests running in the CLI, they're not important for distinguishing which test is running/failing.

[vitalets]

Fair enough. So you mean you would like to see this output:

instead of:

---

I still have doubts, how to better resolve it.

Replacing with anonymous describe block may have it's own problems: the same scenario names in different rules will throw an error, although from the gherkin structure they are different:

test.describe(() => {
  test('scenario 1', async ({ Given, page, When, Then }) => { ... });
});

test.describe(() => {
  test('scenario 1', async ({ Given, page, When, Then }) => { ... });
});

Produces Playwright error:

Error: duplicate test title "Resource Links › scenario 1", first declared in features/homepage.feature.spec.js:8

I'm thinking about dealing with it on the Playwright level: even non-bdd Playwright tests may have very long titles that could be stripped from the output. There are already open issues, related to a long output due to tags auto-appending: microsoft/playwright#33947, microsoft/playwright#17375.

Actually, we are facing the same problem.
My proposal is to open a new, more general issue in the Playwright repo for better truncation of the CLI output.
Currently, Playwright truncates lines from the left to fit the screen width. Potentially, it can be improved to truncate only describe block internal part:

✓  1 .features-gen/homepage.feature.spec.js:6:7 › Resource Links › Resource … displayed › Activating the format validation (1.1s)

instead of:

✓  1 … Links › Resource Links must be in the specifiied format, otherwise inline error message will be displayed › Activating the format validation (1.1s)

What do you think?

[Aaron-Pool]

@vitalets

What do you think?
In general, I find it safe to assume the maintainer has a better grasp on how the feature set of their library should/shouldn't be expanded, so I would certainly defer to your expertise on this. It is slightly worrisome that the issue in you linked, even given its limited scope, hasn't gained much traction on the playwright side—and this request would be even larger.

For what it's worth, I'll respond to this comment:

Replacing with anonymous describe block may have it's own problems: the same scenario names in different rules will throw an error, although from the gherkin structure they are different:

That seems like an issue that the user would (theoretically) understand to be a consequence of setting a given output configuration to null , and something they would have to account for in naming their scenarios.

A possibly more robust approach would be the following configuration instead:

type ItemType = 'feature' | 'rule' | 'outline' | 'scenario' | 'example';

interface ProvidedMetadata {
  type: ItemType,
  raw: string,
  index: number,
  filename: string
}

interface TitleFormat {
  (meta: Metadata): string | null
}

Which would allow people a great deal of flexibility in the title format.

But like, I stated initially, you're the expert in this domain, and I understand if you're not interested in extending the API in this way 👌

[vitalets]

Okay! So, would this output be acceptable in your case?

✓  1 .features-gen/homepage.feature.spec.js:6:7 › Resource Links › Resource … displayed › Activating the format validation (1.1s)

[Aaron-Pool]

It would be a significant improvement, for sure.

Though, would the file path eventually receive some truncation treatment as well? I could see the path getting long enough that the other truncation doesn't even really matter.

[vitalets]

I've created an issue in the Playwright repo: microsoft/playwright#35204
Let's wait for the thoughts from the PW team.

[vitalets]

Another option to copy this tiny reporter that outputs tests as a tree: https://github.com/WhereJuly/70-playwright-hierarchy-reporter

[Aaron-Pool]

Another option to copy this tiny reporter that outputs tests as a tree: https://github.com/WhereJuly/70-playwright-hierarchy-reporter

Thanks. I'll take a look at this one!