Skip to content

Commit 67d2f1b

Browse files
clanghansclanghans
committed
docs(decisions): add DR-001 for unit test design
Documents the four design decisions made in PR #94: - py_itf_unittest macro (thin wrapper, no plugin machinery) - surgical Bazel target splitting for atomic deps - pytest bootstrap via shared main.py - pytest-mock over unittest.mock References: #94
1 parent 1d7fa01 commit 67d2f1b

3 files changed

Lines changed: 170 additions & 0 deletions

File tree

‎docs/decisions/DR-001-infra.md‎

Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
<!--
2+
*******************************************************************************
3+
Copyright (c) 2026 Contributors to the Eclipse Foundation
4+
5+
See the NOTICE file(s) distributed with this work for additional
6+
information regarding copyright ownership.
7+
8+
This program and the accompanying materials are made available under the
9+
terms of the Apache License Version 2.0 which is available at
10+
https://www.apache.org/licenses/LICENSE-2.0
11+
12+
SPDX-License-Identifier: Apache-2.0
13+
*******************************************************************************
14+
-->
15+
16+
# DR-001-Infra: Unit Test Infrastructure Design
17+
18+
**Date:** 2026-05-11
19+
**Status:** Accepted
20+
**PR:** [eclipse-score/itf#94](https://github.com/eclipse-score/itf/pull/94)
21+
**Discussion:** [eclipse-score/discussions#2867](https://github.com/orgs/eclipse-score/discussions/2867)
22+
23+
> This record follows the Decision Record convention established by the
24+
> Eclipse S-CORE project:
25+
> [eclipse-score/score — docs/design_decisions](https://github.com/eclipse-score/score/tree/main/docs/design_decisions).
26+
27+
## Overview
28+
29+
This decision record documents the infrastructure design for unit testing in
30+
ITF. It covers the Bazel macro, dependency scoping strategy, pytest bootstrap
31+
pattern, and mocking library choice, all accepted as part of PR #94.
32+
33+
## Problem Statement
34+
35+
ITF previously had only integration tests: tests that start a real target
36+
(Docker or QEMU) and exercise the system end-to-end. Adding unit tests raised
37+
four concrete questions that each had multiple viable answers:
38+
39+
1. Should unit tests reuse `py_itf_test` or have a dedicated macro?
40+
2. How should Bazel dependencies be scoped to keep tests atomic?
41+
3. How does pytest run inside Bazel, and what does that mean for test
42+
structure?
43+
4. Which mocking library should be used?
44+
45+
## Options Evaluated
46+
47+
### Macro design
48+
49+
**Option A — Reuse `py_itf_test` with empty `plugins`.**
50+
The macro would not crash with an empty plugin list, but it would still
51+
generate the launcher script and resolve `PyItfPluginInfo` providers. The
52+
BUILD file would not communicate that no target is involved.
53+
54+
**Option B — Dedicated `py_itf_unittest` macro (chosen).**
55+
A thin wrapper around `py_test` with no plugin machinery. The name makes
56+
intent explicit. `pytest-mock` is included as a default dep. JUnit XML
57+
reporting is baked in via `$XML_OUTPUT_FILE`.
58+
59+
### Dependency scoping
60+
61+
**Option A — One large Bazel target per package.**
62+
Simple to maintain, but pulls in all transitive dependencies as runfiles.
63+
Bazel measures coverage over all files in the runfiles tree, so the coverage
64+
denominator grows with every transitive dep, even ones not under test.
65+
66+
**Option B — Surgical target splitting (chosen).**
67+
Split Bazel targets along cohesion boundaries so each unit test can declare
68+
only the module it actually exercises. Example: `score/itf/plugins/qemu/BUILD`
69+
was split into `:config` (Pydantic schema only) and `:qemu` (full plugin). The
70+
unit test for schema validation depends only on `:config`, excluding process
71+
management, SSH, and QEMU binary wrappers from its runfiles tree.
72+
73+
### Pytest bootstrap
74+
75+
**Option A — `score_py_pytest` from `@score_tooling`.**
76+
The tooling repository provides a `score_py_pytest` rule, but it bundles a
77+
full Python development environment including `basedpyright` and
78+
`nodejs-wheel-binaries`. These are unrelated to the code under test and expand
79+
the runfiles tree significantly, inflating the coverage denominator and
80+
increasing build time.
81+
82+
**Option B — Shared `main.py` entry point (chosen).**
83+
`py_test` requires an executable Python module. A minimal `main.py` that calls
84+
`pytest.main(sys.argv[1:])` is the de facto standard for Bazel + pytest. The
85+
same bootstrap file is shared across integration and unit test rules, keeping
86+
the approach consistent. This was confirmed as the community standard in the
87+
GitHub discussion linked above.
88+
89+
### Mocking library
90+
91+
**Option A — `unittest.mock.patch` via context managers.**
92+
Part of the standard library, no extra dep. Context manager nesting becomes
93+
verbose when multiple objects need patching.
94+
95+
**Option B — `pytest-mock` via the `mocker` fixture (chosen).**
96+
Patches are registered and torn down automatically through the pytest fixture
97+
lifecycle, removing context manager nesting. Cleaner for tests that mock
98+
several collaborators:
99+
100+
```python
101+
def test_ping_reachable(mocker):
102+
mocker.patch("score.itf.core.com.ping.shutil.which", return_value="/usr/bin/ping")
103+
mocker.patch("score.itf.core.com.ping.os.system", return_value=0)
104+
assert ping("127.0.0.1") is True
105+
```
106+
107+
## Decision & Rationale
108+
109+
All four decisions favour the option that minimises coupling and maximises
110+
clarity in the BUILD file:
111+
112+
- **Dedicated `py_itf_unittest` macro** — the name signals "no target" and
113+
the macro carries no plugin machinery.
114+
- **Surgical Bazel target splitting** — dep declarations in BUILD files become
115+
a lightweight design signal: a test that can only list `:config` as a dep
116+
proves that the schema module is cohesive and has no hidden coupling.
117+
- **Shared `main.py` bootstrap** — consistent with integration tests and
118+
aligned with community practice.
119+
- **`pytest-mock`** — included as a default dep in `py_itf_unittest`; test
120+
authors get `mocker` without an explicit declaration.
121+
122+
Coverage uses Bazel-native LCOV (`configure_coverage_tool = True` in
123+
`MODULE.bazel`) rather than `pytest-cov`, for consistency across all test
124+
types and compatibility with Bazel's `--combined_report`.
125+
126+
## Key Implications
127+
128+
- Unit tests live in `test/unit/` and integration tests in `test/integration/`.
129+
The split is enforced by directory layout and BUILD files, not just naming.
130+
- Adding unit tests for a new module may require splitting its Bazel target if
131+
the current target has a large transitive dep set. This is intentional:
132+
splitting is a design signal that the module has a cohesion opportunity.
133+
- `py_itf_unittest` does not support the `plugins` attribute. A test that
134+
needs a real target belongs in `test/integration/` and uses `py_itf_test`.
135+
- The `mocker` fixture preference applies project-wide; `unittest.mock` context
136+
managers should not be introduced in new tests.

‎docs/decisions/index.rst‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
..
2+
# *******************************************************************************
3+
# Copyright (c) 2026 Contributors to the Eclipse Foundation
4+
#
5+
# See the NOTICE file(s) distributed with this work for additional
6+
# information regarding copyright ownership.
7+
#
8+
# This program and the accompanying materials are made available under the
9+
# terms of the Apache License Version 2.0 which is available at
10+
# https://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# SPDX-License-Identifier: Apache-2.0
13+
# *******************************************************************************
14+
15+
.. _itf_decisions:
16+
17+
Decisions
18+
=========
19+
20+
Design decisions and their rationale, following the convention established by
21+
the `Eclipse S-CORE project <https://github.com/eclipse-score/score/tree/main/docs/design_decisions>`_.
22+
23+
Each record is named ``DR-{number}-{category}.md`` where category is one of
24+
``arch``, ``infra``, ``proc``, or ``strat``.
25+
26+
Infrastructure
27+
--------------
28+
29+
.. toctree::
30+
:maxdepth: 1
31+
:glob:
32+
33+
DR-*-infra*

‎docs/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,7 @@ Integration Test Framework for ECU testing in automotive domains.
4949
how-to/index
5050
reference/index
5151
concepts/index
52+
decisions/index
5253
manual/index
5354
release/index
5455
safety_mgt/index

0 commit comments

Comments
 (0)