Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions .backlog/FIX-LIVE-DCS-FAILURES/PRD.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# Lot FIX-LIVE-DCS-FAILURES — triage the first live-run failures

Status: ✅ done — full 48/48 green live, no src/test changes needed (see ticket 01)
Branch: fix/live-dcs-failures-triage → PR → develop
Program: re-tooling CTLD on the VMCT model (see `.backlog/README.md`). Sibling to
`POST-FULLGAS-FIXES` (already merged, PR #22) and `CLEANUP-LEGACY-DCS-TESTS`.

## Problem Statement

The first full live run of `tools/integration-runner/run_scenarios.py --no-ai` against a real DCS
mission (2026-07-10, `test-results-run2.xml`, 45 scenarios) came back 27 passed / 18 failed. This
was mentioned in `CHANGELOG.md` ("~13 remaining failures ... tracked informally, not yet
ticketed") but never actually turned into backlog tickets.

`POST-FULLGAS-FIXES` has since fixed 8 of the 18 as a side effect of FullGas's review answers.
This lot tracks the remaining ones and needs a fresh live run to confirm the current count before
triaging each.

## Already fixed (verify only, no new work here)

Covered by `POST-FULLGAS-FIXES` tickets 01–06 — re-confirm on the next live run, do not re-open:

- `scenario_fq_vehicle_whole_transport.lua` (ticket 01)
- `U-108_modValidatorHeliportProbeOffMap.lua`, `U-108_modValidatorHeliportWarnAndSkip.lua` (tickets 02/03)
- `F-117_reconDisabledScanShowsExplicitMessageNotSilent.lua`, `F-118_reconToggleOFFMarksRemovedImmediatelyEvenWhenNoLay.lua` (ticket 04)
- `scenario_fr_ai_zones.lua` (ticket 05) — already re-checked standalone post-fix: PASS (`test-results.xml`, 2026-07-12)
- `scenario_farp_countryside_spawn.lua`, `scenario_farp_metal_spawn.lua` (ticket 06)

## Tickets

| # | Scope | Nature |
|---|-------|--------|
| 01 | Triage the 10 remaining failures from `test-results-run2.xml`, re-verify live, fix or re-ticket each | live-DCS + src/test |

## Non-goals

- The 8 already-fixed scenarios above (verification is folded into ticket 01's fresh run, not separate work).
- Porting/purging the 194 FullGas relics — `CLEANUP-LEGACY-DCS-TESTS`.
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# 01 — Triage the 10 remaining live-run failures

Status: ✅ done
Type: hybrid (live DCS re-run + src/test fix per finding)

## Resolution (2026-07-12)

Fresh live run (`--no-ai`, current `develop` + POST-FULLGAS-FIXES already merged): **48/48
`auto`/`auto-check` scenarios PASS**, including all 10 below — confirmed stable across two
consecutive runs. No `src`/test change was needed.

Root cause: the same class of issue already documented for F-120/F-121/F-122/F-123 in the
2026-07-10 run — cross-scenario state contamination accumulated over a long-running mission
session (shared globals: `MenuManager` singleton, scheduler, `U-108` probe run-once guards).
A fresh mission (reload) starts with clean state and every scenario resolves correctly,
including `scenario_mt05_crate_vehicle.lua`'s apparent `isExist()` crash and `F-119`'s radius
mismatch — neither reproduces from a clean state, so not real bugs in
`CTLDCrateManager:getLoadedCrateWeight` or the recon icon geometry.

**Takeaway for future live runs**: reload the mission (`Shift+R`) before a full `--no-ai` sweep,
not just after `src/` changes — long-running sessions accumulate cross-scenario state that
produces false failures.

## Context

Source: `test-results-run2.xml` (2026-07-10 live run, 45 scenarios, 18 failures). 8 of the 18 are
already covered by `POST-FULLGAS-FIXES` (see PRD "Already fixed"). The 10 below are untouched.

## What to do

1. Re-run live (`python tools/integration-runner/run_scenarios.py --no-ai --inject-ctld`) against
current `develop` to confirm which of the 10 still fail (some may have moved since 07-10).
2. For each confirmed failure: read `tests/dcs/CTLD.log` for the ones that only say "see
CTLD.log", root-cause, and either fix `src/` or rewrite the stale scenario/assertion —
whichever is wrong. Follow legacy parity (`migration/source/CTLD.lua`) as the reference.
3. Re-run to confirm PASS before closing.

## The 10 failures to triage

| Scenario | Failure (2026-07-10) | First read |
|---|---|---|
| `F-119_reconAAIconCircleToAllFilled2ApexLineToAllNoBare3L.lua` | Circle radius: `scale=1.0 got=67.5 exp=15.75`, `scale=2.0 got=135 exp=31.5` (both ratios ≈4.29×) | Consistent ratio across both cases — looks like a real unit/formula bug, not a stale test |
| `scenario_ai_transport.lua` | F-134.1/.2/.3/.5: `embarkFromTroopZone`/`disembarkAll` mocks never called | Wiring/mock likely stale vs current `CTLD_core.lua` AI-transport flow |
| `scenario_b3_load_crate_from_menu.lua` | FAIL 3/8 — see `CTLD.log` | — |
| `scenario_cl9_pickup_zones.lua` | FAIL 2/5 — see `CTLD.log` | — |
| `scenario_crate_menu_flight_visibility.lua` | FAIL 11/19 — see `CTLD.log` | — |
| `scenario_feature_p_caps_rename.lua` | FAIL 1/44 — see `CTLD.log` | — |
| `scenario_mt05_crate_vehicle.lua` | Runtime crash: `CTLD.lua:12304: attempt to call method 'isExist' (a nil value)` | `CTLDCrateManager:getLoadedCrateWeight` (`src/CTLD_crate.lua:1052`) calls `crate.loadedBy:isExist()` without a nil/type guard — likely a real nil-safety bug, prioritize (crash, not just assertion) |
| `U-051_oRDERSortingDCSRebuildFollowsOrderFieldNotInsertio.lua` | Menu order assertions (Troops/Crates/FOB by `order` field) fail | — |
| `U-052_pagination13Items9NextPageAtF10.lua` | Pagination assertions (9 items page 1, "→ Next Page") fail | — |
| `U-053_enabledFlagDisabledNodeAbsentFromDCSRebuild.lua` | Enabled-flag assertions (Troops/Beacons rendered) fail | — |

## Acceptance criteria

- [x] Fresh live run confirms the current failure set (may differ from 2026-07-10).
- [x] Each of the 10 is either fixed (src or test) with a green re-run, or explicitly deferred with
a reason recorded here.
- [x] `tests/recette.md` / `CHANGELOG.md` updated to drop the "not yet ticketed" note.
4 changes: 4 additions & 0 deletions .backlog/POST-FULLGAS-FIXES/tickets/01-fq-regression.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,7 @@ silently disabling whole-vehicle spawn from Request Equipment. Restored the logi
b1ddfe4 reference (both objects exist in our repo, read-only). src + rebuild CTLD.lua.

Validated live: scenario_fq_vehicle_whole_transport PASS 9/9 (was 1/9).

L4 pilotPassive confirmed live (2026-07-12, David, UH-1H): whole-vehicle option visible and
spawns correctly from F10 → CTLD → Request Equipment — closes the L4 gap this menu change had
skipped before merge.
4 changes: 3 additions & 1 deletion .backlog/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ authored **per lot, when the lot is started** (not in batch).

| Lot | Status | Description | Branch |
|-----|--------|-------------|--------|
| `REINTEGRATE-ORPHAN-TESTS` | ✅ pending merge | Rebuilt coverage for the 46 features tested only by dead FullGas relics — 6 busted tickets (158 tests, green in CI) + 1 live-DCS ticket (3 scenarios: BCN 30/30, RCN 10/10, VEH 22/22). Precedes & unblocks `CLEANUP-LEGACY-DCS-TESTS`. 10 relics were false orphans (already covered); 6 deferred to FullGas (`dev/fullgas-report.md`). PR #20. | feature/reintegrate-orphan-tests |
| `FIX-LIVE-DCS-FAILURES` | ✅ pending merge | Triage the 10 (of 18) failures from the first live run (2026-07-10, `test-results-run2.xml`) not already fixed by `POST-FULLGAS-FIXES` — all 10 turned out to be cross-scenario state contamination, cleared by a mission reload (48/48 green, no src/test change). | fix/live-dcs-failures-triage |

### Planned lots

Expand All @@ -39,6 +39,8 @@ authored **per lot, when the lot is started** (not in batch).
| `DCS-BRIDGE-MCP` ✅ | Migrate DCS integration testing from Witchcraft to VEAF-dcs-bridge: `.mcp.json` + project-local venv, scenario return contract, 79 scenarios + 4 templates migrated, `integration-testing` skill, Witchcraft fully retired (PR #14). Uncovered ~194 dead FullGas relics → `CLEANUP-LEGACY-DCS-TESTS`. |
| `INTEGRATION-TEST-TAGS` ✅ | `-- @tier: auto\|auto-check\|ia` header on the 79 scenarios + 4 templates (43/2/34 split); fixed F-122's unresolved verdict found during the classification audit (PR #15). |
| `INTEGRATION-TEST-RUNNER` ✅ | Dependency-free `tools/integration-runner/run_scenarios.py`: discovers + tier-filters scenarios, drives them over dcs-serve REST, polls async ones, writes a JUnit report; `--no-ai` mode runs every `auto`/`auto-check` scenario headlessly. 31 stdlib unit tests. Closes the DCS-bridge triptych. |
| `REINTEGRATE-ORPHAN-TESTS` ✅ | Rebuilt coverage for the 46 features tested only by dead FullGas relics — 6 busted tickets (158 tests, green in CI) + 1 live-DCS ticket (3 scenarios: BCN 30/30, RCN 10/10, VEH 22/22). 10 relics were false orphans (already covered); 6 deferred to FullGas (`dev/fullgas-report.md`). PR #20. |
| `POST-FULLGAS-FIXES` ✅ | Applied FullGas's review answers: Feature Q whole-vehicle spawn regression, AI-zone stock validation rewrite, U-108 heliport probe fixes, F-117/F-118 recon fixes, `coord_farp-1` mission static, README doc touch-ups. First full live run against real DCS (27/45) surfaced 18 failures, 8 fixed here as a side effect — remaining 10 → `FIX-LIVE-DCS-FAILURES`. PR #22. |

## Dropped lots

Expand Down
6 changes: 4 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,10 @@ Versioning follows [Semantic Versioning](https://semver.org/).
DCS-editor `beacon.ogg`-class bug (large embedded `l10n/DEFAULT/` resources).
- First full live run (`--no-ai`, 45 scenarios) against a real DCS mission: 27/45 passed after a
mission reload cleared cross-scenario state contamination between 4 vehicle/JTAC-family
scenarios (F-120/F-121/F-122/F-123); ~13 remaining failures are reproducible across repeated
fresh runs and need investigation (tracked informally, not yet ticketed).
scenarios (F-120/F-121/F-122/F-123); 18 failures total. 8 were fixed by the FullGas review
round above; the remaining 10 (`FIX-LIVE-DCS-FAILURES` lot) turned out to be the same class of
cross-scenario state contamination, not real bugs — a fresh mission reload cleared all of them
(48/48 `auto`/`auto-check` scenarios green, confirmed on two consecutive fresh runs).

### CI / tooling

Expand Down