Eclipse S-CORE
Test use cases and environments #2875
Replies: 2 comments
-
|
Just fyi: we have a current PR eclipse-score/os_images#5 for the "Generic QNX IFS" available which would currently support QEMU options 3 and 4, and if virtio9p is added, option 2 as well. |
Beta Was this translation helpful? Give feedback.
-
|
Since this discussion goes into direction what test levels do we have and which tools do we use on what level, I've steered Claude to make some analysis. TLDR;
Full analysis:1. SummaryThe discussion proposes a taxonomy of S-CORE test use cases (UT, CIT/FIT, PIT) mapped to four target environments (Host Ubuntu, QEMU QNX aarch64, QEMU QNX x86_64, Hardware QNX aarch64). It then presents architectural options for how to structure the QEMU and hardware test environments, varying how the OS image, software-under-test (SUT), and test scripts are packaged and delivered. 2. Inconsistencies and Issues Found2.1 Typo: "enviornment"In the "Basic test environment requirements" table, the header reads "Test enviornment" instead of "Test environment". 2.2 Typo: "Generix" (appears twice)QEMU options 3 and 4 are described as "Generix QNX IFS" — should be "Generic QNX IFS". Option 2 spells it correctly as "Generic". 2.3 ENV1 use-case scope is inconsistent with the test-use-case table
This is a direct contradiction. Either the first table is too restrictive (Host should be listed for CIT/FIT/PIT) or the second table is too permissive (ENV1 should only list UT). The discussion provides no rationale for the expansion. 2.4 ENV3 use-case scope is inconsistent with the test-use-case table
Two problems:
2.5 Hardware options table numbering is misleadingThe hardware options are numbered 1, 2, 3 while the QEMU options are numbered 1, 2, 3, 4. Both sequences start at 1, but these are conceptually different option sets. The comment by the author references "QEMU options 3 and 4" which is clear enough, but having two distinct "Option 1", "Option 2", "Option 3" in the same document without explicit qualification is error-prone when referencing them. 2.6 Missing cross-reference between QEMU and Hardware optionsQEMU Option 3 and Hardware Option 2 are functionally equivalent (Generic IFS + qnx6fs partition images), and QEMU Option 4 and Hardware Option 3 are functionally equivalent (Generic IFS + tarballs). This relationship is not explicitly stated, which makes it harder to reason about a coherent strategy that spans both QEMU and hardware environments. 2.7 No mention of ENV1 architecture at allThe discussion describes QEMU-based environment architecture (ENV2, ENV3) and Hardware environment architecture (ENV4), but ENV1 (Host Ubuntu x86_64) has no architectural options presented. This is a significant gap — how are tests run natively on the host? Are the SUT binaries compiled natively? How does this interact with the QNX-specific functionality? This is left completely unaddressed. 2.8 The
|
| Aspect | Assessment |
|---|---|
| Simplicity | Single artifact, simple deployment |
| Flexibility | Very poor — read-only filesystem, must rebuild IFS per test case |
| Licensing | Requires QNX SDP license for every build |
| Hardware portability | Questionable |
| Verdict | Not viable for any non-trivial use case. The need to rebuild a complete IFS for every test scenario is a showstopper. |
Option 2: Generic IFS + virtio9p
| Aspect | Assessment |
|---|---|
| Simplicity | Moderate — requires custom virtio9p driver on QNX |
| Flexibility | Excellent — shared directory, fast iteration, no image rebuilds |
| Licensing | Avoids QNX license dependency for test builds |
| Hardware portability | Not portable — virtio9p is QEMU-only |
| Maturity | Already implemented in qnx_unit_tests repo |
| Verdict | Best option for QEMU-only testing (UT, development CIT/FIT). Fast feedback loop. Cannot serve as the sole approach if hardware testing parity is needed. |
Option 3: Generic IFS + qnx6fs partition image
| Aspect | Assessment |
|---|---|
| Simplicity | More complex — requires building qnx6fs images |
| Flexibility | Good — reusable OS image, separate test content |
| Licensing | Requires QNX license for qnx6fs tools at build time |
| Hardware portability | Excellent — same image works on QEMU and hardware |
| Verdict | Best option for cross-environment parity (PIT, hardware CIT/FIT). License dependency is a significant practical concern. |
Option 4: Generic IFS + tarballs
| Aspect | Assessment |
|---|---|
| Simplicity | Moderate — requires runtime unpacking, blank qnx6fs image |
| Flexibility | Good — avoids license dependency |
| Licensing | Avoids QNX license for qnx6fs tools |
| Hardware portability | Works on both, but adds boot-time overhead |
| Verdict | Pragmatic compromise when license constraints prevent Option 3. Runtime unpacking adds latency and complexity. |
4. Analysis of Hardware Options
Option 1: All-in-IFS
Same problems as QEMU Option 1, compounded by BSP dependency. Not viable.
Option 2: IFS + qnx6fs partition
Direct analog of QEMU Option 3. Preferred for hardware if licensing permits.
Option 3: IFS + tarballs
Direct analog of QEMU Option 4. Fallback for license-constrained environments.
5. Recommended Strategy
Tiered approach: Option 2 (QEMU dev) + Option 3 or 4 (cross-environment)
The discussion implicitly points toward a two-tier strategy, but doesn't state it:
| Testing tier | Environment | Recommended option | Rationale |
|---|---|---|---|
| Developer inner loop (UT, local CIT/FIT) | QEMU only (ENV2/ENV3) | Option 2 (virtio9p) | Fastest iteration, already built, no image rebuild. Use qnx_unit_tests repo. |
| Pipeline / integration (CIT/FIT, PIT) | QEMU + Hardware (ENV2/ENV4) | Option 3 (qnx6fs) or Option 4 (tarballs) | Cross-environment portability required. Use os_images repo. |
| Host testing (UT) | Ubuntu x86_64 (ENV1) | Native compilation | Needs its own story — not addressed. |
Key decision point: licensing
The choice between Option 3 and Option 4 for the pipeline tier hinges entirely on whether QNX license for qnx6fs tooling is available in CI/CD environments. This is a business/legal constraint, not a technical one, and should be resolved before committing to an architecture.
6. The Role of ITF (eclipse-score/itf) — and its Absence from the Discussion
6.1 What ITF is
The Integration Test Framework (ITF) is a pytest-based testing framework already present in the eclipse-score ecosystem. It provides:
- A
Targetabstraction layer — an abstract base class withexecute(),upload(),download(),restart()methods and a capability system (has_capability(),@requires_capabilitiesdecorator). - Plugin architecture — concrete target implementations are loaded via
py_itf_pluginBazel macros. Existing plugins:- Docker plugin (
DockerTarget): runs commands via Docker exec API, supports file transfer, optional SSH. - QEMU plugin (
QemuTarget): boots a QNX IFS image in QEMU, interacts via SSH/SFTP, supports bridge networking and port forwarding, async command execution, ping/restart lifecycle. - DLT plugin: Diagnostic Log and Trace message capture and analysis (can be composed with Docker or QEMU plugins).
- Docker plugin (
- Bazel-native integration —
py_itf_testmacro creates hermetic test targets that carry their full dependency graph (images, configs) and integrate with Bazel's test caching and JUnit reporting. - Write-once, run-anywhere test model — the same test code (
target.execute("uname -a")) works against Docker, QEMU, or (with a custom plugin) hardware, with capability guards to skip inapplicable tests.
6.2 ITF already solves the test execution layer
The discussion in #2875 focuses on how to package/deliver the OS image and test content (IFS structure, partition images, tarballs), but never mentions how tests are actually orchestrated and executed against the target. This is precisely what ITF does.
ITF's QEMU plugin already:
- Takes a QNX IFS image as input (via
--qemu-imagearg). - Boots it in QEMU with configurable networking (bridge or port-forwarding), CPU cores, and RAM.
- Connects via SSH to execute test commands and transfer files.
- Collects results and tears down the VM.
This means the environment options from the discussion map to ITF as follows:
| Discussion option | ITF compatibility | Notes |
|---|---|---|
| QEMU Option 1 (all-in-IFS) | Partially compatible | ITF can boot the image but has limited ability to interact if no SSH/writable filesystem is available. |
| QEMU Option 2 (IFS + virtio9p) | Not currently supported | ITF's QEMU plugin uses SSH, not virtio9p. The qnx_unit_tests repo's virtio9p approach is entirely separate from ITF. |
| QEMU Option 3 (IFS + qnx6fs partition) | Fully compatible | ITF boots the IFS image; the OS mounts the qnx6fs partition at /opt/score. Tests interact via SSH. This is exactly how ITF's QEMU plugin expects to work. |
| QEMU Option 4 (IFS + tarballs) | Compatible with caveats | ITF can boot the image. Tarball unpacking would need to happen in the startup script before ITF begins SSH interaction. |
| Hardware Option 2/3 | Compatible with new plugin | ITF's plugin architecture is designed for this — a HardwareTarget plugin that connects to physical hardware via SSH would reuse the same Target interface. No such plugin exists yet but it's an intended extension point. |
6.3 ITF can be used for CIT/FIT — and is arguably designed for it
The discussion lists three test use case categories:
| Test type | Can ITF be used? | Assessment |
|---|---|---|
| Unit Tests (UT) | Possible but heavy | ITF boots a full QEMU VM or Docker container per test session. For fast unit test iteration, the qnx_unit_tests repo's microvm + virtio9p approach (boot in <1s, shared filesystem) is more appropriate. ITF adds overhead via SSH connection setup, pytest infrastructure, and VM lifecycle management that isn't justified for simple UT. |
| Component/Feature Integration Tests (CIT/FIT) | Yes — this is ITF's sweet spot | CIT/FIT tests need to deploy modules, start services, verify inter-component behavior, and possibly capture DLT traces. ITF provides all of this: target.execute() to start services, target.upload() to deploy binaries, @requires_capabilities("ssh", "sftp") for full-system interaction, DLT plugin for trace verification. The Docker plugin enables CIT/FIT on host (ENV1) without needing QNX at all. |
| Platform Integration Tests (PIT) | Yes — with environment abstraction | PIT tests run against "collections of modules" on QEMU and hardware. ITF's target abstraction means PIT tests written against target.execute() can run on QEMU today and on hardware when a hardware plugin is implemented. The restart() and ping() capabilities are specifically useful for PIT scenarios testing system resilience. |
6.4 The critical gap: the discussion ignores ITF
Discussion #2875 and ITF appear to be developed in silos:
-
The discussion never mentions ITF. It focuses entirely on image packaging, not on test orchestration. Yet the two are tightly coupled — the choice of image packaging determines what ITF plugins can interact with.
-
ITF's QEMU plugin and
qnx_unit_testsuse different approaches. ITF boots a full QEMU VM and connects via SSH (similar to Options 3/4).qnx_unit_testsboots a microvm and shares files via virtio9p (Option 2). These are complementary approaches for different test tiers, but no one has stated this. -
There is no hardware plugin for ITF yet, but the architecture explicitly anticipates one. The environment discussion should be planning for ITF hardware plugin compatibility rather than designing hardware test execution independently.
-
ITF already addresses ENV1 (Host Ubuntu) via the Docker plugin. The discussion lists ENV1 as a test environment but provides no architecture for it. ITF's Docker plugin provides a ready-made solution: CIT/FIT tests can run in Docker containers on the host without any QNX dependency, using the same test code that runs on QEMU.
6.5 Recommended ITF integration
The three repos (itf, qnx_unit_tests, os_images) should be positioned as a coherent stack:
┌─────────────────────────────────────────────────────────────┐
│ Test Code (pytest) │
├─────────────────────────────────────────────────────────────┤
│ ITF Target Abstraction (target.execute, etc.) │
├──────────────┬──────────────┬───────────────┬───────────────┤
│ Docker Plugin│ QEMU Plugin │ Hardware Plugin│ (future) │
│ (ENV1) │ (ENV2/ENV3) │ (ENV4) │ │
├──────────────┼──────────────┼───────────────┼───────────────┤
│ Docker image │ OS image from│ OS image from │ │
│ (ubuntu:*) │ os_images │ os_images │ │
│ │ repo │ repo + BSP │ │
└──────────────┴──────────────┴───────────────┴───────────────┘
UT (fast): qnx_unit_tests (microvm + virtio9p) — bypasses ITF
CIT/FIT: ITF + Docker (ENV1) or ITF + QEMU (ENV2/ENV3)
PIT: ITF + QEMU (ENV2) or ITF + Hardware (ENV4)
This means:
os_imagesprovides the generic QNX IFS images (Options 3/4) that ITF's QEMU plugin and the future hardware plugin consume.itfprovides the test orchestration layer for CIT/FIT/PIT across all environments.qnx_unit_testsremains the fast-path for unit tests where ITF's overhead is not justified.- The environment option choice (IFS+partition vs IFS+tarball) should be made with ITF's QEMU plugin expectations in mind — specifically, the OS must have SSH running and a writable filesystem accessible before ITF connects.
6.6 Specific concerns for ITF + discussion options
-
Option 2 (virtio9p) is incompatible with ITF's current QEMU plugin. ITF uses SSH for all interaction; virtio9p is a different transport entirely. If Option 2 is chosen for CIT/FIT (not just UT), ITF would need a new transport mechanism or a new plugin — this is a significant architectural decision that should be discussed.
-
Option 3/4 images must launch
sshd. Theos_imagesPR already does this (the README mentions "Launchessshdservice for remote control (userroot, empty password by default)"). This is good — it meansos_imagesoutput is ITF-QEMU-plugin-compatible by default. -
Test content delivery needs alignment. ITF's QEMU plugin currently expects test binaries to be accessible on the target. With Option 3, they'd be on the qnx6fs partition mounted at
/opt/score. With Option 4, they'd need to be unpacked from tarballs first. ITF test code would need to know where binaries live — this path convention (/opt/scorevs/tmp/unpackedvs wherever) should be standardized. -
Hardware plugin design. When a hardware ITF plugin is built, it will likely connect via SSH to the target board. The same
os_imagesIFS (with sshd) running on hardware should work, but the test content delivery (partition flash vs tarball upload via SSH) needs to be decided in advance, and it should mirror the QEMU approach for test portability.
7. Open Questions Not Addressed
-
ENV1 (Host Ubuntu) architecture: How do QNX-targeted modules get tested natively? Is there an abstraction layer? POSIX compatibility shim? This is listed as a first-class environment but has zero architectural discussion.
-
Relationship between
qnx_unit_testsandos_imagesrepos: Are they complementary (different tiers), competing (converge to one), or transitional (one replaces the other)? -
ITF integration: The
eclipse-score/itfrepo already provides a mature test orchestration framework with QEMU and Docker target plugins (see Section 6). The discussion makes no reference to ITF despite it being the natural execution layer for CIT/FIT/PIT. How do the proposed environment options map to ITF's target abstraction and plugin architecture? -
ENV3 (QEMU QNX x86_64) justification: The discussion hedges on whether this is even needed ("limited, only for development use cases", "maybe"). If x86_64 QEMU offers no advantage over aarch64 QEMU (both require cross-compilation from a host perspective), what is the concrete developer use case? Is it KVM acceleration on x86_64 hosts?
-
Test result extraction: None of the hardware options discuss how test results get back to the CI pipeline. The
qnx_unit_testsrepo solves this via virtio9p shared directories, but for hardware (ENV4), there's no mention of SSH, serial console, or other result collection mechanisms. -
Disk/partition sizing: Option 3 mentions qnx6fs partition images and the
os_imagesPR creates 512MB QCOW2 disks. Is this sufficient for PIT scenarios with "collections of modules"? No sizing analysis is provided. -
Security considerations: The
os_imagesPR ships with hardcoded SSH host keys and root login with empty password. This is noted in the README but not discussed as a concern for any environment beyond local QEMU development.
8. Overall Assessment
The discussion is a reasonable starting point but has internal inconsistencies in the use-case-to-environment mapping (sections 2.3 and 2.4 above) that need resolution before the community can make an informed decision. The options are well-enumerated but the evaluation criteria are ad-hoc (+ / - / o notation) rather than systematically weighted.
The biggest gap is the lack of a stated recommendation or convergence proposal. The discussion presents four QEMU options and three hardware options without ranking them or proposing a path forward. Combined with three existing repos (qnx_unit_tests, os_images, and itf) that implement different layers of the testing stack, this risks fragmentation if a clear strategy isn't established.
The complete omission of ITF from the discussion is the most significant oversight. ITF already provides the test orchestration layer (target abstraction, capability system, pytest integration, Bazel macros) that the discussion's environment options will need to integrate with. The environment packaging decision (Options 2/3/4) directly affects which ITF plugins can be used and how test content is delivered. Making this decision without considering ITF compatibility risks choosing an approach that either doesn't work with ITF or requires ITF to be significantly reworked.
The ENV1 (host native) gap is partially addressed by ITF's Docker plugin, which the discussion appears unaware of. Connecting these pieces — os_images for QNX base images, ITF for test orchestration, qnx_unit_tests for fast unit test iteration — into a coherent, layered testing architecture should be the primary outcome of this discussion.
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi - we are thinking about test use cases and test environments and came up with some options we would like to present/discuss. Idea is to limit ourselves to a subset of the listed QEMU/Hardware options we would like to support in S-CORE.
S-CORE Test use cases
Which test use cases do exist?
- QEMU (QNX8 aarch64, quality requirement: target architecture),
- QEMU (QNX8 x86_64, limited, only for development use cases)
- QEMU QNX8 x86_64, (maybe, only for development use cases)
- Hardware QNX8 aarch64
Basic test environment requirements
Out of the test use cases we can derive these test environments:
Test environment architecture
QEMU-only testing: ENV1,ENV2,ENV3 (UT/CIT,FIT/PIT)
How should the QEMU OS environment be designed for supporting the test cases?
- Build individual IFS for all test cases?
- QEMU-only test environment? Hardware usability questioned.
- Test build environment (ITF) requires dependency to QNX SDP and QNX license to build IFS.
+ only one target to build
+ only one build target
+ quick test execution
- test cases may be limited to QEMU only(?). "Generic" PIT test cases which should also run on hardware potentially not supported.
+ can avoid need for QNX license for qnx6fs tools in pipeline
- less convenient: have to create additional qnx6fs image
+ qnx6fs image works on QEMU and on hardware, if flashed as a partition image (->PIT use case)
o Test env requires a blank writeable qnx6fs image.
- test scripts have to unpack tarballs before running tests
+ creating tarballs instead of qnx6fs images can avoid need for QNX license for qnx6fs tools at ITF stage
Hardware testing: ENV4 (PIT/CIT,FIT)
How should the Hardware OS environment be designed for supporting the test cases?
- Build individual IFS for all test cases?
- Test build environment (ITF) requires dependency to hardware BSP and QNX license.
+ only one build target
+ qnx6fs image works on QEMU and on hardware, if flashed as a partition image.
- qnx6fs tools require QNX license at test build environment stage (ITF)
o Test env requires a blank writeable qnx6fs partition
- test scripts have to unpack tarballs before running tests
+ unpacking tarballs may be faster than flashing a complete partition
+ creating tarballs instead of qnx6fs images can avoid need for QNX license for qnx6fs tools at ITF stage
Beta Was this translation helpful? Give feedback.
All reactions