WIP: Consumer tests

[MaximilianSoerenPollak]

Developing the consumer tests here for docs-as-code.

Just want to gather some feedback here on the general appraoch etc.

[github-actions]

License Check Results

🚀 The license check job ran with the Bazel command:

bazel run //src:license-check

Status: ✅ Passed

Click to expand output

[License Check Output]
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Invocation ID: 1b88a96a-2f40-43ea-b22f-7b2fc8a0e47c
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Loading: 
Loading: 0 packages loaded
Loading: 0 packages loaded
    currently loading: src
Loading: 0 packages loaded
    currently loading: src
Analyzing: target //src:license-check (1 packages loaded, 0 targets configured)
Analyzing: target //src:license-check (1 packages loaded, 0 targets configured)

Analyzing: target //src:license-check (103 packages loaded, 10 targets configured)

Analyzing: target //src:license-check (146 packages loaded, 1353 targets configured)

Analyzing: target //src:license-check (153 packages loaded, 2652 targets configured)

Analyzing: target //src:license-check (158 packages loaded, 2679 targets configured)

Analyzing: target //src:license-check (160 packages loaded, 2692 targets configured)

Analyzing: target //src:license-check (162 packages loaded, 4820 targets configured)

INFO: Analyzed target //src:license-check (163 packages loaded, 4946 targets configured).
[11 / 13] [Prepa] JavaToolchainCompileBootClasspath external/rules_java~/toolchains/platformclasspath.jar
[12 / 13] [Prepa] Building src/license.check.license_check.jar ()
INFO: Found 1 target...
Target //src:license.check.license_check up-to-date:
  bazel-bin/src/license.check.license_check
  bazel-bin/src/license.check.license_check.jar
INFO: Elapsed time: 22.028s, Critical Path: 2.93s
INFO: 13 processes: 9 internal, 3 processwrapper-sandbox, 1 worker.
INFO: Build completed successfully, 13 total actions
INFO: Running command line: bazel-bin/src/license.check.license_check src/formatted.txt -review -project automotive.score -repo https://github.com/eclipse-score/docs-as-code -token otyhZ4eaRYK1tKLNNF-Y
[main] INFO Querying Eclipse Foundation for license data for 84 items.
[main] INFO Found 58 items.
[main] INFO Querying ClearlyDefined for license data for 26 items.
[main] INFO Found 26 items.
[main] INFO Vetted license information was found for all content. No further investigation is required.

[github-actions]

The created documentation from the pull request is available at: docu-html

[Copilot]

Pull Request Overview

This draft adds end-to-end consumer tests that clone sample repositories, override their Bazel dependencies (locally or via Git), and verify build success.

• Introduces a ConsumerRepo dataclass and a list of repos to test
• Implements helper functions to patch MODULE.bazel for local and Git overrides
• Adds a single parametrized test (test_and_clone_repos) to clone repos, apply overrides, and run Bazel commands

Comments suppressed due to low confidence (1)

src/tests/test_consumer.py:24

• [nitpick] Attribute names should follow snake_case; rename to local_override_result for consistency.

    LocalOverrideResult: bool

[nicu1989]

why not check the subprocess for errors also here? with check=True. I feel like if some future maintainer “optimises” the processing of BR and forgets to look at returncode failures silently become successes.

[MaximilianSoerenPollak]

The issue is, that if I do 'check=True' here, if bazel exits with a non 0 exit code (which it does when sphinx errors for example) it just stops execution immediatly there.
At least to my testing. That's why I thought it better to not do this here.

[nicu1989]

understood

[AlexanderLanin]

it throws an exception. you could catch it. It's safer - but more boilerplate than checking return code.

[MaximilianSoerenPollak]

Also note.

This is on purpose not a Bazel test target, as this is executed via direct call.
.venv/bin/python -m pytest <args> src/tests

Additionally, the placement of this is obviously not finaly, I just put it there for now, if you can think of where it would fit better, let me know.

[MaximilianSoerenPollak]

@AlexanderLanin Where should this test file be located btw? Not really sure if this is a good place for it, where I put it currently.

[MaximilianSoerenPollak]

One thing I do struggle with this, and have not even attempted is. How can this be tested beyond just manually testing it and making sure it does what it should?
Seems extremely difficult to me.

[Aymen-Soussi-01]

Don't forget to correct the formatting check

[MaximilianSoerenPollak]

Don't forget to correct the formatting check

Thanks. Just pushed the formatting.