feat: traceability metrics (#484)

Signed-off-by: Frank Scholter Peres(MBTI) <145544737+FScholPer@users.noreply.github.com>
Co-authored-by: Maximilian Sören Pollak <maximilian.pollak@qorix.com>
Co-authored-by: Andreas Zwinkau <95761648+a-zw@users.noreply.github.com>
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Alexander Lanin <Alexander.Lanin@etas.com>

Author: 5 people

.gitignore

@@ -26,3 +26,4 @@ __pycache__/
 
 # bug: This file is created in repo root on test discovery.
 /consumer_test.log
+.clwb

docs.bzl

@@ -304,16 +304,24 @@ def docs(source_dir = "docs", data = [], deps = [], scan_code = [], known_good =
             "--jobs",
             "auto",
             "--define=external_needs_source=" + str(data),
-        ] + metamodel_opts,
+            "--define=score_sourcelinks_json=$(location :sourcelinks_json)",
+            "--define=score_source_code_linker_plain_links=1",
+        ],
         formats = ["needs"],
         sphinx = ":sphinx_build",
-        tools = data + metamodel_data,
+        tools = data + [":sourcelinks_json"],
         visibility = ["//visibility:public"],
         # Persistent workers cause stale symlinks after dependency version
         # changes, corrupting the Bazel cache.
         allow_persistent_workers = False,
     )
 
+    native.alias(
+        name = "traceability_gate",
+        actual = "@score_docs_as_code//scripts_bazel:traceability_gate",
+        tags = ["cli_help=Enforce traceability coverage thresholds:\nbazel run //:traceability_gate -- --metrics-json bazel-bin/needs_json/_build/needs/metrics.json"],
+    )
+
 def _sourcelinks_json(name, srcs):
     """
     Creates a target that generates a JSON file with source code links.

docs/how-to/dashboards_and_quality_gates.rst

@@ -0,0 +1,145 @@
+..
+   # *******************************************************************************
+   # Copyright (c) 2026 Contributors to the Eclipse Foundation
+   #
+   # See the NOTICE file(s) distributed with this work for additional
+   # information regarding copyright ownership.
+   #
+   # This program and the accompanying materials are made available under the
+   # terms of the Apache License Version 2.0 which is available at
+   # https://www.apache.org/licenses/LICENSE-2.0
+   #
+   # SPDX-License-Identifier: Apache-2.0
+   # *******************************************************************************
+
+Build Dashboards and Quality Gates
+==================================
+
+Use this guide in repositories that consume docs-as-code as a Bazel
+dependency.
+
+Goals:
+
+1. Publish traceability dashboards from repository needs.
+2. Export machine-readable metrics.
+3. Enforce CI thresholds with ``traceability_gate``.
+
+What You Get
+------------
+
+With the ``docs(...)`` macro and ``score_metamodel`` extension enabled, your
+repository can:
+
+- build an HTML dashboard from its own Sphinx needs,
+- include external needs from other repositories when desired,
+- export ``needs.json`` and ``metrics.json`` for machine-readable reporting,
+- gate CI on traceability thresholds via ``traceability_gate``.
+
+Typical Setup
+-------------
+
+For details, see :ref:`setup`.
+
+Minimal Configuration Example
+-----------------------------
+
+In ``docs/conf.py``:
+
+.. code-block:: python
+
+   score_metamodel_requirement_types = "feat_req,comp_req,aou_req"
+   score_metamodel_include_external_needs = False
+
+Use ``score_metamodel_include_external_needs = True`` only in repositories that
+intentionally aggregate requirements across module dependencies, such as
+integration repositories. Use ``False`` for module repositories to gate only on
+local traceability.
+
+Building the Dashboard
+----------------------
+
+After building/running any docs command (i.e. ``bazel build //:needs_json`` or ``bazel run //:docs_check`` are the fastest):
+
+The documentation build writes ``metrics.json`` via ``score_metamodel``, and the ``needs_json`` artifact contains:
+
+- ``bazel-bin/needs_json/_build/needs/needs.json``
+- ``bazel-bin/needs_json/_build/needs/metrics.json``
+
+The dashboard charts and the CI gate both use the same computed metrics.
+
+Inputs for Linkage Metrics
+--------------------------
+
+To get meaningful dashboard and gate values, consumer repositories typically
+need three inputs:
+
+1. Requirement and architecture needs in the documentation itself.
+2. Source code references via :doc:`source_to_doc_links`.
+3. Test metadata via :doc:`test_to_doc_links`.
+
+If one of those inputs is missing, the related chart or gate metric will remain
+empty or low.
+
+Choosing Local vs Aggregated Views
+----------------------------------
+
+There are two common modes:
+
+**Module repository**
+
+- Set ``score_metamodel_include_external_needs = False``.
+- Gate only on the needs owned by the repository itself.
+- Use this for per-module implementation progress and traceability.
+
+**Integration repository**
+
+- Set ``score_metamodel_include_external_needs = True``.
+- Aggregate requirements across module dependencies when that is the intended
+  repository purpose.
+- Use this for system or integration-level dashboards.
+
+CI Quality Gate
+---------------
+
+Any docs build (``bazel run //:docs``, ``bazel run //:docs_check``, etc.)
+writes ``metrics.json`` alongside the build output. Run the gate on the
+exported metrics:
+
+.. code-block:: bash
+
+   bazel run //:docs && \
+   bazel run //:traceability_gate -- \
+      --metrics-json bazel-bin/needs_json/_build/needs/metrics.json \
+      --min-req-code 70 \
+      --min-req-test 70 \
+      --min-req-fully-linked 60 \
+      --min-tests-linked 70
+
+In CI, wire targets through Bazel dependencies so test execution and
+docs generation happen before the gate target.
+
+In larger repositories, define a dedicated wrapper target for your standard
+gate thresholds so CI calls a single Bazel target.
+
+Useful flags:
+
+- ``--require-all-links`` for strict 100 percent gating
+
+Recommended Rollout
+-------------------
+
+For a new consumer repository:
+
+1. Start with local-only metrics.
+2. Enable ``scan_code`` and verify ``source_code_link`` coverage first.
+3. Add test metadata and verify ``testlink`` coverage.
+4. Introduce modest thresholds in CI.
+5. Raise thresholds over time as the repository matures.
+
+Related Guides
+--------------
+
+- :ref:`setup`
+- :doc:`other_modules`
+- :doc:`source_to_doc_links`
+- :doc:`test_to_doc_links`

docs/how-to/get_started.rst

@@ -24,3 +24,6 @@ In an existing S-CORE repository, you can build the documentation using Bazel:
 Open the generated site at ``_build/index.html`` in your browser.
 
 In a new S-CORE repository, see :ref:`setup`.
+
+After the initial setup, continue with :doc:`dashboards_and_quality_gates` to
+build a repository dashboard and enforce CI quality gates.

docs/how-to/index.rst

@@ -27,6 +27,7 @@ Here you find practical guides on how to use docs-as-code.
    write_docs
    faq
    other_modules
+   dashboards_and_quality_gates
    source_to_doc_links
    test_to_doc_links
    add_extensions

docs/how-to/setup.md

@@ -71,6 +71,7 @@ The `docs()` macro accepts the following arguments:
 | `data` | List of `needs_json` targets that should be included in the documentation | No |
 | `deps` | Additional Bazel Python dependencies | No |
 | `scan_code` | Source code targets to scan for traceability tags | No |
+| `known_good` | Label to a "known good" JSON file for source links | No |
 | `metamodel` | Label to a custom `metamodel.yaml` that replaces the default metamodel | No |
 
 ### 4. Copy conf.py
@@ -88,3 +89,9 @@ bazel run //:docs
 #### 6. Access your documentation at
 
 `/_build/index.html`
+
+## Next Step
+
+After basic setup, see {doc}`dashboards_and_quality_gates` to configure
+traceability dashboards, export `metrics.json`, and enforce CI quality gates in
+consumer repositories.

docs/how-to/test_to_doc_links.rst

@@ -12,6 +12,8 @@
    # SPDX-License-Identifier: Apache-2.0
    # *******************************************************************************
 
+   # Assisted-by: GitHub Copilot
+
 Reference Docs in Tests
 =======================
 
@@ -53,3 +55,9 @@ Limitations
 - Partial properties will lead to no Testlink creation.
   If you want a test to be linked, please ensure all requirement properties are provided.
 - Tests must be executed by Bazel first so `test.xml` files exist.
+
+Related
+-------
+
+For end-to-end dashboard and CI threshold setup, see
+:doc:`dashboards_and_quality_gates`.

docs/internals/requirements/implementation_state.rst

@@ -11,21 +11,58 @@
    #
    # SPDX-License-Identifier: Apache-2.0
    # *******************************************************************************
+
+   # Assisted-by: GitHub Copilot
 .. _docs_statistics:
 
-Implementation State Statistics
-================================
+Tooling Coverage
+================
+
+This page shows how the docs-as-code tooling covers process and tool
+requirements. It focuses on tooling capabilities offered to downstream
+repositories rather than on product-specific traceability inside those
+repositories.
 
 Overview
 --------
 
-.. needpie:: Requirements Status
-   :labels: not implemented, implemented but not tested, implemented and tested
+.. needpie:: Tool Requirements Status
+   :labels: not implemented, implemented but incomplete traceability, fully linked
    :colors: red,yellow, green
+   :filter-func: src.extensions.score_metamodel.checks.traceability_dashboard.pie_requirements_status(tool_req)
+
+Jump to evidence tables:
+
+- :ref:`Tool Requirement Implementation and Links table <tooling_coverage_table_impl_links>`
+- :ref:`Process Requirement to Tool Requirement mapping table <tooling_coverage_table_process_mapping>`
+
+How To Read These Levels
+------------------------
+
+The overview pie combines implementation state and traceability evidence:
+
+- ``not implemented``:
+   requirement has ``implemented == NO``.
+- ``implemented but incomplete traceability``:
+   requirement has ``implemented == YES`` or ``implemented == PARTIAL``,
+   but is missing at least one traceability link (code link and/or test link).
+- ``fully linked``:
+   requirement is implemented and has both ``source_code_link`` and ``testlink``.
 
-   type == 'tool_req' and implemented == 'NO'
-   type == 'tool_req' and testlink == '' and (implemented == 'YES' or implemented == 'PARTIAL')
-   type == 'tool_req' and testlink != '' and (implemented == 'YES' or implemented == 'PARTIAL')
+Implementation labels used on this page:
+
+- ``NO``: requirement is not implemented.
+- ``PARTIAL``: requirement is partly implemented.
+- ``YES``: requirement is implemented.
+
+Why multiple pies are shown:
+
+- ``Requirements with Codelinks`` shows requirement-to-implementation traceability.
+- ``Requirements with linked tests`` shows requirement-to-verification traceability.
+- ``Requirements fully linked`` is the strict roll-up (both links present).
+
+These are intentionally separate because they answer different diagnostics:
+missing code links, missing test links, or both.
 
 In Detail
 ---------
@@ -48,78 +85,43 @@ In Detail
       .. needpie:: Requirements with Codelinks
          :labels: no codelink, with codelink
          :colors: red, green
-
-         type == 'tool_req' and source_code_link == ''
-         type == 'tool_req' and source_code_link != ''
+         :filter-func: src.extensions.score_metamodel.checks.traceability_dashboard.pie_requirements_with_code_links(tool_req)
 
    .. grid-item-card::
 
-      .. needpie:: Test Results
-         :labels: passed, failed, skipped
-         :colors: green, red, orange
-
-         type == 'testcase' and result == 'passed'
-         type == 'testcase' and result == 'failed'
-         type == 'testcase' and result == 'skipped'
-
-.. grid:: 2
+      .. needpie:: Requirements with linked tests
+         :labels: no test link, with test link
+         :colors: red, green
+         :filter-func: src.extensions.score_metamodel.checks.traceability_dashboard.pie_requirements_with_test_links(tool_req)
 
    .. grid-item-card::
 
-      Failed Tests
-
-      *Hint: this table is empty by definition, as PRs with failing tests are not allowed to be merged in docs-as-code repo.*
-
-      .. needtable:: FAILED TESTS
-         :filter: result == "failed"
-         :tags: TEST
-         :columns: name as "testcase";result;fully_verifies;partially_verifies;test_type;derivation_technique;id as "link"
+      .. needpie:: Requirements fully linked (code + tests)
+         :labels: not fully linked, fully linked
+         :colors: orange, green
+         :filter-func: src.extensions.score_metamodel.checks.traceability_dashboard.pie_requirements_fully_linked(tool_req)
 
    .. grid-item-card::
 
-      Skipped / Disabled Tests
-
-      *Hint: this table is empty by definition, as we do not allow skipped or disabled tests in docs-as-code repo.*
-
-      .. needtable:: SKIPPED/DISABLED TESTS
-         :filter: result != "failed" and result != "passed"
-         :tags: TEST
-         :columns: name as "testcase";result;fully_verifies;partially_verifies;test_type;derivation_technique;id as "link"
-
-
-
-
-All passed Tests
------------------
-
-.. needtable:: SUCCESSFUL TESTS
-   :filter: result == "passed"
-   :tags: TEST
-   :columns: name as "testcase";result;fully_verifies;partially_verifies;test_type;derivation_technique;id as "link"
-
+      .. needpie:: Process requirements linked by tool requirements
+         :labels: not linked, linked
+         :colors: red, green
+         :filter-func: src.extensions.score_metamodel.checks.traceability_dashboard.pie_process_requirements_linked(tool_req,true)
 
-Details About Testcases
-------------------------
-*Data is not filled out yet within the test cases.*
 
-.. needpie:: Test Types Used In Testcases
-   :labels: fault-injection, interface-test, requirements-based, resource-usage
-   :legend:
+Process-to-Tool Mapping
+-----------------------
 
-   type == 'testcase' and test_type == 'fault-injection'
-   type == 'testcase' and test_type == 'interface-test'
-   type == 'testcase' and test_type == 'requirements-based'
-   type == 'testcase' and test_type == 'resource-usage'
+.. _tooling_coverage_table_process_mapping:
 
+.. needtable:: Process requirement -> tool requirement mapping
+   :types: tool_req
+   :columns: satisfies as "Process Requirement";id as "Tool Requirement"
+   :style: table
 
-.. needpie:: Derivation Techniques Used In Testcases
-   :labels: requirements-analysis, design-analysis, boundary-values, equivalence-classes, fuzz-testing, error-guessing, explorative-testing
-   :legend:
+.. _tooling_coverage_table_impl_links:
 
-   type == 'testcase' and derivation_technique == 'requirements-analysis'
-   type == 'testcase' and derivation_technique == 'design-analysis'
-   type == 'testcase' and derivation_technique == 'boundary-values'
-   type == 'testcase' and derivation_technique == 'equivalence-classes'
-   type == 'testcase' and derivation_technique == 'fuzz-testing'
-   type == 'testcase' and derivation_technique == 'error-guessing'
-   type == 'testcase' and derivation_technique == 'explorative-testing'
+.. needtable:: Tool requirement implementation and links
+   :types: tool_req
+   :columns: id as "Tool Requirement";implemented;source_code_link;testlink
+   :style: table