Skip to content

feat(parsers): ingest Allure results directories - #17

Open
ibrahimsn98 wants to merge 1 commit into
mainfrom
feat/allure-results-parser
Open

ibrahimsn98 wants to merge 1 commit into
mainfrom
feat/allure-results-parser

Conversation

@ibrahimsn98

Copy link
Copy Markdown
Contributor

Why

Teams leaving Allure TestOps already have Allure adapters wired into every framework. With this they can point qf at the results directory and change nothing else. It also gives the landing page's Allure TestOps alternative its strongest honest migration claim, which until now it couldn't make.

pytest --alluredir=allure-results
qf myapp collect allure-results/

How

  • A new generic/allure parser, implemented as a PathAwareParser (the same extension point .xcresult uses). The whole directory is one input because a retry is a second *-result.json with the same historyId, so attempts can only be grouped with every file in view.
  • expandDirectories keeps a directory that holds *-result.json whole instead of splitting it into its .json files.
  • Detection: the directory is detected by its contents, whatever it's called. A single result file is detected by content (uuid, historyId, status, start and stop together).
  • Passthrough format, like CTRF. The launch and suite are labelled with the producing framework from the framework label (pytest → python, junit5 → junit, cucumberjvm → cucumber, …). An unmapped producer falls back to generic, because an unknown category would 400 the launch server-side.

Mapped

  • Status: broken and unknown map to error.
  • Suite path: parentSuite > suite > subSuite, carried in ClassName.
  • Metadata: labels kept verbatim, plus tags, severity → priority, and links.
  • Steps: nested steps keep their tree via parentIndex.
  • Parameters: masked keeps the name but not the value; hidden is dropped.
  • Retries: they become attempts, retryCount is set, and failed-then-passed counts as flaky.
  • Attachments: images, videos and Playwright traces go through the existing artifact upload, gated by --upload-artifacts; other files up to 1 MiB are inlined. A source that isn't a plain file name in the directory is dropped, so ../ can't read files elsewhere.

Not in this PR: *-container.json fixtures. A failing fixture already makes Allure mark its test broken, so statuses stay correct.

Tests

  • Fixture: a realistic allure-results directory with passed, failed, broken and skipped tests, a retry, nested steps, every parameter mode, an attachment, and a container, executor.json, categories.json and environment.properties that must all be ignored.
  • 15 parser tests, 3 factory tests and 1 expandDirectories test, plus the existing category contract test, which now includes allure.
  • go test ./..., go vet and gofmt all pass.
  • End to end: qf e2e collect testdata/allure-results --dry-run --output json reads it as one input, labels the launch python, and reports 5 cases with 1 flaky test.

`qf <id> collect allure-results/` now uploads Allure 2 results written by any
framework's Allure adapter — no reporter change for teams moving off Allure.

The directory is ONE input (a ports.PathAwareParser, like .xcresult): a retry
is a second *-result.json with the same historyId, so attempts can only be
grouped with every file in view. expandDirectories keeps a directory holding
*-result.json whole instead of splitting it into its .json files, and the
factory detects it by contents (adapters write wherever they are told), plus a
single result file by content.

Allure is a FORMAT, so it is registered as a passthrough like CTRF: the suite
records the producing framework from the `framework` label (pytest -> python,
junit5 -> junit, cucumberjvm -> cucumber, …) and takes its category; an
unmapped producer falls back to generic, since an unknown category would 400
the launch server-side.

Mapped: status (broken -> error), parentSuite > suite > subSuite into
ClassName, labels, tags, severity, links, nested steps, parameters honouring
masked/hidden, retries/flaky, and attachments — images, videos and Playwright
traces through the artifact upload, other small files inline; a source that is
not a plain file name in the directory is dropped. Containers are not read
yet: a failing fixture already marks its test broken.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant