macOS QA Test Suite Reorganization

[tallpsmith]

Problem Statement

The current macOS test suite in dev/darwin/ needs reorganization to align with PCP's standard QA framework and address PR feedback from Nathan Scott and Ken McDonell. The current structure has grown organically during development but needs to integrate with PCP's established testing infrastructure before being merged.

Current State

Directory structure:

dev/darwin/
├── dev/                          # Build infrastructure (~5-10s rebuild)
│   ├── GNUmakefile
│   ├── build-quick.sh
│   └── setup-local-pcp.sh
├── test/
│   ├── quick-test.sh            # Orchestrator: build→unit→integration
│   ├── unit/                    # DSO validation tests (13 test files)
│   └── integration/             # pmcd-based integration tests
└── README.md

Test coverage:

• darwin PMDA (domain 78, DSO, system metrics): 12 unit tests covering CPU, memory, disk, VFS, network protocols (TCP/UDP/ICMP), socket stats, memory compression, vmstat64 upgrade
• darwin_proc PMDA (domain 3, pipe, process metrics): 1 unit test covering proc.nprocs, proc.io, proc.fd, proc.memory, proc.runq
• Integration tests: pmstat and pmrep :macstat views (testing both PMDAs via pmcd)

Key characteristics:

• Fast iteration: 20-30s for full build+test cycle
• Tests organized by feature area rather than by PMDA
• No dependencies on qa/ directory currently

Why This Matters

1. PR feedback alignment: Need to address Nathan's preference for distributed test placement and Ken's emphasis on qa/ integration
2. Fast iteration preservation: Critical 20-30s dev cycle enables rapid testing during PMDA development
3. QA framework integration: Path to using qa/check -g pmda.darwin for comprehensive testing
4. Maintainability: Clear separation between darwin (system metrics) and darwin_proc (process metrics) PMDAs
5. Developer experience: New contributors need to easily discover and run tests

Two Competing Approaches

Option 1: Fully Distributed (Nathan's Stated Preference)

Philosophy: Tests live near the code they test, no centralized orchestration

Structure:

build/mac/dev/           # Build tools (GNUmakefile, build-quick.sh)
src/pmdas/darwin/test/   # darwin PMDA unit tests (12 files)
src/pmdas/darwin_proc/test/  # darwin_proc PMDA unit tests (1 file)
qa/darwin/               # Integration tests (cross-PMDA)
qa/2100-2119            # Numbered QA tests (eventual)

Pros:

• Aligns with Nathan's stated preference
• Tests co-located with source code
• Clear separation between darwin and darwin_proc
• Follows "tests near code" principle

Cons:

• Slower iteration (30-45s, navigating multiple directories)
• Less discoverable for new contributors
• Integration tests separated from unit tests
• No simple "run all tests" command
• Hard to run "all macOS tests" without tooling

Fast dev loop: cd build/mac/dev && ./build-quick.sh then manually navigate to multiple test directories

---

Option 2: Distributed Tests with Centralized Orchestration (Recommended)

Philosophy: Tests live near the code they test, with simple wrapper scripts in build/mac/ for fast "run everything" iteration

Structure:

src/pmdas/darwin/test/       # darwin PMDA unit tests (12 files)
│   ├── unit/
│   │   ├── run-unit-tests.sh
│   │   └── test-*.txt       # 12 test files
│   └── README.md

src/pmdas/darwin_proc/test/  # darwin_proc PMDA unit tests (1 file)
│   ├── unit/
│   │   ├── run-unit-tests.sh
│   │   └── test-proc.txt
│   └── README.md

qa/darwin/                   # Integration tests (cross-PMDA)
│   ├── test-pmstat.sh
│   └── test-pmrep-macstat.sh

build/mac/
├── dev/                     # Build tools (GNUmakefile, build-quick.sh)
└── test/
    ├── run-all-tests.sh     # Orchestrator: build → darwin → darwin_proc → integration
    ├── run-darwin-tests.sh  # Wrapper: runs src/pmdas/darwin/test/unit/run-unit-tests.sh
    └── run-integration-tests.sh  # Wrapper: runs qa/darwin/*.sh

qa/2100-2119                 # Numbered QA tests (eventual)
qa/group                     # Add pmda.darwin, pmda.darwin_proc

Pros:

• Tests co-located with source code (Nathan's stated preference)
• Fast iteration via wrapper scripts (20-30s for full cycle)
• Simple orchestration: build/mac/ scripts know where to find all tests
• No symlinks required
• Clear separation between darwin and darwin_proc
• Natural fit for QA integration
• Developers can run individual PMDA tests OR all tests easily

Cons:

• Tests spread across multiple directories (navigation requires knowing locations)
• Wrapper scripts need to maintain paths to test locations
• Integration tests in qa/darwin/ separate from unit tests

Fast dev loop:

• Run all tests: cd build/mac && ./test/run-all-tests.sh (20-30s)
• Run specific PMDA: cd src/pmdas/darwin && ./test/unit/run-unit-tests.sh
• Run just one test file: cd src/pmdas/darwin/test/unit && ./run-unit-tests.sh test-cpu.txt

Key advantage: Tests live near code (satisfying "tests near code" principle) while maintaining fast "run everything" workflow via simple wrapper scripts in build/mac/

Comparison Matrix

Aspect	Option 1: Fully Distributed	Option 2: Distributed + Orchestration
Fast iteration	30-45s (manual navigation)	20-30s (via wrapper scripts)
Tests near code	Yes (native)	Yes (native)
Nathan's preference	High match	High match
Ken's QA integration	Natural fit	Natural fit (qa/group mechanism)
Run all tests easily	No (manual process)	Yes (wrapper scripts)
Migration complexity	High (4+ locations)	Medium (3 locations + wrappers)
Maintenance burden	Medium	Low (simple wrapper scripts)
New contributor UX	Must know 3+ locations	Wrappers in build/mac/ provide entry point
Implementation	Move files only	Move files + create wrapper scripts

QA Integration Path (Phase 2 - Separate Work)

Both options will eventually integrate with PCP's QA framework:

Test numbering: qa/2100-2119 (20 tests allocated)

• 2100-2104: darwin PMDA core system (CPU, memory, disk, system basics, filesystem)
• 2105-2109: darwin PMDA network (TCP, UDP, ICMP, connection states, socket stats)
• 2110-2114: darwin_proc PMDA (process enumeration, I/O, file descriptors, memory, run queue)
• 2115-2119: Integration & tools (pmrep views, pmstat)

QA groups: Add to qa/group:

# macOS Darwin PMDA (system metrics)
pmda.darwin

# macOS Darwin process PMDA (per-process metrics)
pmda.darwin_proc

This matches existing patterns like pmda.proc (line 268) and pmda.linux (line 306).

Questions for Maintainers

1. Primary question: Which approach do you prefer - Option 1 (fully distributed) or Option 2 (distributed with orchestration wrappers)?

2. Integration test placement: Should integration tests (which test BOTH darwin and darwin_proc PMDAs) live in qa/darwin/ or elsewhere?

3. Test numbering: Is 2100-2119 (20 tests) the right allocation, or should we reserve more (e.g., 2100-2149)?

4. Migration approach: Single large PR for Phase 1 restructuring, or split into smaller PRs?

Implementation Plan

Regardless of which option is chosen, the work breaks down into:

Phase 1: Restructure tests (3-5 hours)

• Move test files to distributed locations (src/pmdas/darwin/test, src/pmdas/darwin_proc/test, qa/darwin/)
• Create wrapper scripts in build/mac/test/ (Option 2 only)
• Update script paths
• Create documentation
• Verify 20-30s dev cycle preserved

Phase 2: QA integration (8-16 hours, can be incremental)

• Add numbered tests to qa/2100-2119
• Update qa/group with pmda.darwin and pmda.darwin_proc
• Create platform-specific expected output files

Phase 3: CI integration (2-3 hours)

• Update .cirrus.yml with QA test steps
• Document CI expectations

Additional Context

Full reorganization plan with detailed analysis: build/mac/MACOS_QA_REORGANIZATION.md in the branch

---

Request: Please provide guidance on the preferred approach so we can proceed with Phase 1 implementation. This work is blocking final integration of the macOS PMDA enhancements.

cc: @natescott @kmcdonell

[natoscott]

@tallpsmith as long as the end goal moves towards integration with all platforms QA below qa/, and normal CI that uses qa/, I don't mind what approach is used. And to clarify when I said earlier "tests below src/pmdas/darwin are fine" I meant just unit tests like some other PMDAs do (like src/pmdas/activemq) - there should also always be qa/ integrated tests as well.

[tallpsmith]

This should be addressed by 04b32bc in the #2442 branch.