ci: add Cook-based Verilator Tier 1 and Tier 2 workflows for GitHub Actions

[AlexChenIC]

Summary

This PR extends the existing Tier CI flow used on the master branch to master_candidate and adapts it to the Cook-based compilation and regression flow introduced on master_candidate by Thales contributors.

Motivation and architecture

The existing master Tier CI provides the overall model for organizing regression coverage into separate tiers. The master_candidate branch uses the Thales Cook infrastructure as the basis of its software compilation and verification flow.

This PR connects these two approaches. It preserves the Tier 1 and Tier 2 organization established by the master CI while adapting the execution path to the Cook-based infrastructure used by master_candidate.

The longer-term goal is for the public GitHub Actions CI and the Thales GitLab CI to share the same underlying execution architecture wherever practical. Platform-specific concerns, such as job scheduling, caching, and artifact publication, remain implemented by the respective GitHub Actions and GitLab CI layers.

What this PR adds

The implementation:

• compiles selected testlists through cook.py
• consumes the compiled ELFs through a Cook recipe adapted for the existing Verilator TestHarness
• runs live Spike tandem checking
• adds a reusable GitHub Actions environment setup for the RISC-V GCC toolchain, Verilator, Spike, and pinned Cook Python dependencies
• generates the Cook compiler configuration from the installed GCC toolchain
• propagates compilation, simulation, and tandem-comparison failures to GitHub Actions
• handles non-zero child processes, missing simulation results, tandem reports, and failure patterns in generated logs
• supports both the existing Verilator log format and the cycle-prefixed TestHarness format
• retains toolchain metadata, reports, logs, compiled software, simulation outputs, and generated configuration as artifacts

Initial validation scope

This is the first scoped integration of the proposed architecture on master_candidate. Its purpose is to validate the complete end-to-end structure with a controlled set of configurations before expanding the regression coverage. It is not intended to provide the final testcase or core-configuration coverage in this PR.

The current validation scope covers two RV32 AXI core configurations.

Tier 1 — hosted run, 2/2 matrix jobs passed:

• cv32a60x_axi with base-rv32-p
• cv32a65x_axi with base-rv32-p

Tier 2 — hosted run, 3/3 matrix jobs passed:

• cv32a60x_axi with base-rv32-p
• cv32a65x_axi with base-rv32-p
• cv32a65x_axi with base-pmp

All five hosted matrix jobs completed successfully, the targeted Verilator TestHarness safety tests passed, and artifacts were uploaded for every matrix job. This demonstrates the end-to-end path across both core configurations.

Dependency

This integration requires the uvm_warning compatibility fix proposed in #3478. Without that fix, the Verilator build used by the Cook-to-TestHarness path fails because the master_candidate version of the macro does not match the standard two-argument UVM signature. For validation, this branch temporarily includes the equivalent two-line fix.

Follow-up work

Once the initial architecture is accepted, follow-up work is expected to include:

• reviewing and expanding the testcase coverage for the existing configurations
• adding further master_candidate core configurations, using the Thales GitLab CI coverage as a reference
• aligning the GitHub Actions and GitLab CI result metadata and test intent where practical
• adding a regression dashboard similar to the existing master Tier CI dashboard

These extensions are intentionally kept outside the scope of this first PR so that the Cook-based integration architecture can be reviewed and validated independently.

[yanicasa]

Thank you for this contribution! The testharness support is definitely useful. However, I don't think the current implementation fits well with the design philosophy of cook.py.

The main concern is that, in this PR, verilator_testlist.py is currently a monolithic wrapper around the existing cva6.py flow. One of the goals of cook.py is to avoid carrying this kind of legacy orchestration into the new recipe structure.

cook.py philosophy:

• Atomic recipes: each recipe should have a single responsibility (e.g. compile or run, rather than doing both).
• Minimal external dependencies: recipes should invoke the simulator directly rather than going through Makefiles or legacy scripts whenever possible.
• Composability: recipes should be independently usable and composable into higher-level flows.
• Consistency: the same overall structure should be used across simulators and testbench types.

Proposed Architecture

Based on the existing UVM flows for VCS, Xcelium, and Questa, I would suggest splitting the testharness support into separate compilation, execution, and testlist recipes.

With this architecture, users would:

# HW Compile testharness with Verilator
./cook.py verilator-testharness-comp -t cv32a60x

### Single test:
# SW Compile hello-world test
./cook.py hello-world -t cv32a60x -c llvm-20-1-8  # this recipe already exist!
# Run a single test
./cook.py verilator-testharness-run -t cv32a60x -n riscv-tests-rv32ui-p-add # TOBEDONE

### Run a testlist
# SW Compile all tests in testlist
./cook.py sw-compile-testlist -t cv32a60x -c llvm-20-1-8 -l custom/regression.yml # this recipe already exist!
# Run all tests in testlist
./cook.py testharness-run-testlist --simulator verilator -t cv32a60x -l tests.yml # TOBEDONE

1. Compilation Recipes (per simulator)

flows/recipes/verilator_testharness_comp.py
flows/recipes/vcs_testharness_comp.py (further work)
flows/recipes/questa_testharness_comp.py  (further work)
flows/recipes/xcelium_testharness_comp.py  (further work)

Responsibilities:

• Compile RTL with testharness testbench (using appropriate Flist)
• Handle trace modes (notrace/fast/gui/compact)
• Support compilation modes (rtl/coverage/gate)
• Generate compiled binary/database

Key differences from UVM:

• Different Flist
• Different top module
• Different testbench hierarchy

Reference implementation: vcs_uvm_comp.py, xcelium_uvm_comp.py, questa_uvm_comp.py

For example, the Verilator recipe could expose an interface along these lines:

def verilator_testharness_comp(
    target: str = typer.Option(..., "--target", "-t"),
    comp_mode: CompMode = typer.Option(CompMode.rtl),
    trace_mode: TraceMode = typer.Option(TraceMode.notrace),
    stats: bool = typer.Option(False, help="Enable RTL perf tracer"),
    quiet: bool = typer.Option(False, "--quiet", "-q"),
):
    """Verilator testharness compilation flow"""

2. Run Recipes (per simulator)

flows/recipes/verilator_testharness_run.py
flows/recipes/vcs_testharness_run.py  (further work)
flows/recipes/questa_testharness_run.py  (further work)
flows/recipes/xcelium_testharness_run.py  (further work)

Responsibilities:

• Run a SINGLE test with the compiled testbench
• Handle reference model comparison (ISS)
• Generate logs and reports
• Extract pass/fail status

Key differences from UVM:

• Different mechanism for detecting and reporting PASS/FAIL

Reference implementation: vcs_uvm_run.py, xcelium_uvm_run.py, questa_uvm_run.py

For example, the Verilator recipe could expose an interface along these lines:

def verilator_testharness_run(
    target: str = typer.Option(..., "--target", "-t"),
    test_name: str = typer.Option(..., "--testname", "-n"),
    comp_mode: CompMode = typer.Option(CompMode.rtl),
    trace_mode: TraceMode = typer.Option(TraceMode.notrace),
    iss_enabled: bool = typer.Option(False, help="Enable ISS comparison"),
    interactive_gui: bool = typer.Option(False),
    quiet: bool = typer.Option(False, "--quiet", "-q"),
):
    """Verilator testharness run simulation flow"""

3. TestHarness Testlist Runner (multi-simulator)

flows/recipes/testharness_run_testlist.py

Responsibilities:

• Parse testlist YAML
• Call appropriate simulator run recipe for each test
• Aggregate results
• Generate summary report

Reference implementation: uvm_run_testlist.py (already supports VCS/Xcelium/Questa)

Signature:

def testharness_run_testlist(
    simulator: str = typer.Option(..., "--simulator", help="vcs/xcelium/questa/verilator"),
    target: str = typer.Option(..., "--target", "-t"),
    testlist: str = typer.Option(..., "--testlist", "-l"),
    comp_mode: CompMode = typer.Option(CompMode.rtl),
    trace_mode: TraceMode = typer.Option(TraceMode.notrace),
    iss_enabled: bool = typer.Option(False),
    quiet: bool = typer.Option(False, "--quiet", "-q"),
):
    """Run testharness testlist with specified simulator"""
    # Dispatch to appropriate simulator run recipe
    if simulator == "verilator":
        verilator_testharness_run(...)
    elif simulator == "vcs":
        vcs_testharness_run(...)
    # etc.

--

Would you be willing to consider refactoring the implementation along these lines?

I’d be happy to help with:

• designing the recipe interfaces,
• reviewing incremental changes for each component

The goal is to make TestHarness support fully integrated into cook.py, with the same level of consistency and maintainability as the existing UVM flows.

[AlexChenIC]

Hi @yanicasa, thank you for the detailed review and for proposing a clearer architecture.

At present, this PR adds a single TestHarness testlist recipe, primarily used by the Tier CI path, that delegates each compiled test to the legacy cva6.py flow. This approach has been useful for validating the Cook software compilation, Verilator TestHarness, and Spike tandem path. However, I agree that it does not fully align with the Cook design goals you described and should be refactored.

My understanding is that the TestHarness support should be reorganized into three recipe layers:

• verilator_testharness_comp.py for simulator-specific TestHarness compilation;
• verilator_testharness_run.py for running one compiled test with the Verilator TestHarness;
• testharness_run_testlist.py for simulator selection, testlist iteration, result aggregation, and reporting.

The simulator-specific compilation and run recipes should invoke Verilator TestHarness directly where practical, rather than delegating to cva6.py or Makefiles. The Tier CI path would then compose these Cook recipes instead of relying on the current monolithic recipe around the legacy flow. Please correct me if I have misunderstood the proposed architecture.

I would appreciate your help with both the recipe interface design and the incremental review. Before revising the implementation, could you please confirm the following points?

• I understand that the VCS, Questa, and Xcelium TestHarness backends are follow-up work. For the first Verilator implementation, is RTL compilation and execution a sufficient initial scope, with coverage and gate modes deferred?
• For the testlist layer, should I initially follow the direct-dispatch pattern used by uvm_run_testlist.py, or would you prefer a shared runner/result interface to be introduced now? Is there an existing result/report contract that should be reused?
• Would you prefer me to update this PR incrementally in place, or would it be cleaner to submit the atomic TestHarness Cook support as a prerequisite PR and keep the Tier CI workflows in a follow-up PR?

Once these points are clarified, I can begin implementing the Verilator compilation recipe based on the existing work and share that component for review.

[yanicasa]

Hi @AlexChenIC
For the first iteration, I would keep the scope focused on two recipes:

1. verilator_testharness_comp.py
   Compile the RTL with Verilator and the TestHarness testbench.
   No dependency on cva6.py, Makefiles, or other legacy.
2. verilator_testharness_run.py
   Run a single test using the compiled Verilator TestHarness.
   Again, no dependency on cva6.py, Makefiles, or other external.

Let's start with the compilation recipe first and review it incrementally. Once that is in place, we can move on to the run recipe and then the testlist integration.

Coverage, gate modes, and the other simulator backends can be addressed later.