QA Report: feat: update dev scripts in apps to use portless for port assignment (PR #40)

[sortakool]

QA Report

PR: #40 — feat: update dev scripts in apps to use portless for port assignment
Branch: feat/update-dev-scripts-portless
Date: 2025-07-18
Verdict: ⚠️ PARTIAL

All 15 Playwright tests pass, but a critical runtime bug was discovered outside the test suite: the -- separator in both dev scripts causes bun run dev to fail with exit code 127.

---

Gherkin Scenarios

#	Scenario	Tests	Result
1	apps/web dev script runs through portless without a hardcoded port	3	✅ PASS
2	apps/api dev script runs through portless	2	✅ PASS
3	Running bun run dev from the root starts both apps on auto-assigned ports	2	✅ PASS
4	Web app is accessible at its portless subdomain	1	✅ PASS
5	API app is accessible at its portless subdomain	1	✅ PASS
6	No hardcoded port numbers remain in apps/web dev script	3	✅ PASS
7	No hardcoded port numbers remain in apps/api dev script	3	✅ PASS

Totals: 15 passed, 0 failed, 0 errors (1.1s)

---

Failures

No Playwright test failures. All 15 assertions passed.

However, a critical runtime bug was found — see below.

---

🐛 Bug Found: -- separator breaks bun run dev (Severity: HIGH)

Problem

Both dev scripts use a -- separator that portless does not understand:

File	Current (broken)	Correct
apps/web/package.json	portless virtual-agent -- next dev	portless virtual-agent next dev
apps/api/package.json	portless api.virtual-agent -- bun run --hot src/index.ts	portless api.virtual-agent bun run --hot src/index.ts

Evidence (from dev-server.log)

@virtual-agent/web dev: Running: PORT=4299 HOST=127.0.0.1 -- next dev
@virtual-agent/web dev: /bin/sh: --: command not found
@virtual-agent/web dev: Exited with code 127

@virtual-agent/api dev: Running: PORT=4219 HOST=127.0.0.1 -- bun run --hot src/index.ts
@virtual-agent/api dev: /bin/sh: --: command not found
@virtual-agent/api dev: Exited with code 127

Why Tests Still Passed

1. Scenarios 1, 2, 6, 7 (static checks) — Assert the scripts.dev string content matches what the Gherkin spec says. The Gherkin spec itself contains the incorrect -- syntax, so the tests match a broken value.
2. Scenarios 3, 4, 5 (runtime checks) — The apps were manually started using the corrected commands (without --) before running Playwright.

Recommended Fix

Remove -- from both dev scripts and update the Gherkin scenario expectations accordingly.

---

Screenshots

No screenshots captured. Playwright was configured with screenshot: "only-on-failure" and all tests passed.

---

Notes

1. Test validity gap: Scenario 3 claims to verify "Running bun run dev from the root starts both apps" but only checks static config and HTTP responses. It does not actually invoke bun run dev and verify the process exits cleanly.
2. No browser-level tests. All 15 tests are fs-based assertions or request.get() API calls. No page.goto() usage — reasonable for a dev-script wiring PR but means no visual regression coverage.
3. Portless proxy assumed running. The Playwright config has no webServer block to auto-start portless or the dev servers.

---

Raw Output

Full Playwright output

Running 15 tests using 1 worker

  ✓   1 .attractor/qa-report/gherkin.spec.ts:26:7 › Scenario 1: apps/web dev script runs through portless without a hardcoded port › Given the file "apps/web/package.json" exists (2ms)
  ✓   2 .attractor/qa-report/gherkin.spec.ts:31:7 › Scenario 1: apps/web dev script runs through portless without a hardcoded port › Then its "scripts.dev" field should be "portless virtual-agent -- next dev" (0ms)
  ✓   3 .attractor/qa-report/gherkin.spec.ts:36:7 › Scenario 1: apps/web dev script runs through portless without a hardcoded port › And the "scripts.dev" field should not contain "--port" (0ms)
  ✓   4 .attractor/qa-report/gherkin.spec.ts:46:7 › Scenario 2: apps/api dev script runs through portless › Given the file "apps/api/package.json" exists (1ms)
  ✓   5 .attractor/qa-report/gherkin.spec.ts:51:7 › Scenario 2: apps/api dev script runs through portless › Then its "scripts.dev" field should be "portless api.virtual-agent -- bun run --hot src/index.ts" (1ms)
  ✓   6 .attractor/qa-report/gherkin.spec.ts:63:7 › Scenario 3: Running bun run dev from the root starts both apps on auto-assigned ports › Given the root "package.json" has a "dev" script (1ms)
  ✓   7 .attractor/qa-report/gherkin.spec.ts:70:7 › Scenario 3: Running bun run dev from the root starts both apps on auto-assigned ports › Then both the web and api apps should start successfully (verified via portless proxy) (101ms)
  ✓   8 .attractor/qa-report/gherkin.spec.ts:98:7 › Scenario 4: Web app is accessible at its portless subdomain › When I send a GET request to "http://virtual-agent.localhost:1355" Then the response status should be 200 (28ms)
  ✓   9 .attractor/qa-report/gherkin.spec.ts:113:7 › Scenario 5: API app is accessible at its portless subdomain › When I send a GET request to "http://api.virtual-agent.localhost:1355/health" Then the response status should be 200 (4ms)
  ✓  10 .attractor/qa-report/gherkin.spec.ts:128:7 › Scenario 6: No hardcoded port numbers remain in apps/web dev script › Given the file "apps/web/package.json" exists (1ms)
  ✓  11 .attractor/qa-report/gherkin.spec.ts:133:7 › Scenario 6: No hardcoded port numbers remain in apps/web dev script › Then its "scripts.dev" field should not contain "3000" (0ms)
  ✓  12 .attractor/qa-report/gherkin.spec.ts:138:7 › Scenario 6: No hardcoded port numbers remain in apps/web dev script › And its "scripts.dev" field should not contain "4000" (0ms)
  ✓  13 .attractor/qa-report/gherkin.spec.ts:148:7 › Scenario 7: No hardcoded port numbers remain in apps/api dev script › Given the file "apps/api/package.json" exists (1ms)
  ✓  14 .attractor/qa-report/gherkin.spec.ts:153:7 › Scenario 7: No hardcoded port numbers remain in apps/api dev script › Then its "scripts.dev" field should not contain "3000" (1ms)
  ✓  15 .attractor/qa-report/gherkin.spec.ts:158:7 › Scenario 7: No hardcoded port numbers remain in apps/api dev script › And its "scripts.dev" field should not contain "4000" (0ms)

  15 passed (1.1s)