[TEST] Integration Test Suite - HA, VNC, Multi-Platform

[JoshuaAFerguson]

Problem

While unit tests validate individual components, integration tests are needed to validate complete end-to-end flows across the v2.0-beta architecture:

• Multi-pod API deployment (Redis-backed AgentHub)
• K8s Agent + Docker Agent together
• VNC streaming (Control Plane → Agent → Container)
• High Availability failover scenarios
• Cross-platform session management

Current Status

• Existing Integration Tests: 5 files (tests/integration/)
• Status: Unknown (not executed in recent analysis)
• Coverage: Likely outdated for v2.0-beta features

Required Integration Test Scenarios

1. Multi-Pod API Deployment

Test: Deploy 2-3 API replicas with Redis, verify distributed operation

Scenarios:

• Agent connections distributed across API pods
• Session creation via different API pods
• Cross-pod command routing (Redis pub/sub)
• Agent status visible across all pods
• API pod failure (session survival)
• Redis failure (graceful degradation)

Expected Output: .claude/reports/INTEGRATION_TEST_HA_MULTI_POD_API.md

---

2. K8s Agent High Availability

Test: Deploy 3+ K8s agent replicas with leader election enabled

Scenarios:

• Leader election process (< 30s)
• Only leader processes commands
• Followers reject commands gracefully
• Leader pod failure (automatic failover)
• New leader elected (< 30s)
• Session survival during failover
• Zero data loss confirmation

Expected Output: .claude/reports/INTEGRATION_TEST_HA_K8S_AGENT_LEADER_ELECTION.md

---

3. Docker Agent Session Lifecycle

Test: Complete Docker agent E2E flow

Scenarios:

• Session creation (container + network + volume)
• VNC configuration and port mapping
• Session hibernation (container stop, volume persist)
• Session wake (container restart from hibernated)
• Session termination (full cleanup)
• Resource limit enforcement (CPU, memory)
• Multiple concurrent sessions

Expected Output: .claude/reports/INTEGRATION_TEST_DOCKER_AGENT_LIFECYCLE.md

---

4. VNC Streaming End-to-End

Test: Complete VNC streaming flow (all components)

Scenarios:

K8s Agent VNC:

• Session creation with VNC template
• VNC tunnel creation (agent → pod port-forward)
• Control Plane VNC proxy connection
• Browser WebSocket → Proxy → Agent → Pod flow
• Bidirectional data streaming
• Connection cleanup on termination

Docker Agent VNC:

• Session creation with VNC template
• VNC port mapping (container 5900)
• Control Plane VNC proxy connection
• Browser WebSocket → Proxy → Agent → Container flow
• Bidirectional data streaming

Expected Output: .claude/reports/INTEGRATION_TEST_VNC_STREAMING_E2E.md

---

5. Combined HA Chaos Testing

Test: Multi-pod API + Multi-agent deployment under chaos

Scenarios:

• Deploy 3 API pods + 3 K8s agents + 2 Docker agents
• Create 10+ active sessions (mixed K8s + Docker)
• Kill random API pod (verify zero session loss)
• Kill K8s agent leader (verify automatic failover)
• Kill Docker agent (verify session survival if HA enabled)
• Redis connection loss (verify graceful degradation)
• Network partition simulation
• Full recovery verification

Expected Output: .claude/reports/INTEGRATION_TEST_HA_CHAOS_TESTING.md

---

6. Multi-User Concurrent Sessions

Test: Multiple users creating concurrent sessions

Scenarios:

• 3-5 different users
• 2-3 sessions per user (10-15 total)
• Mixed templates (K8s + Docker)
• Session isolation verification
• Resource quota enforcement
• VNC access for all sessions
• Concurrent termination
• Cleanup verification

Expected Output: .claude/reports/INTEGRATION_TEST_MULTI_USER_CONCURRENT_SESSIONS.md

---

7. Performance & Load Testing

Test: System behavior under load

Scenarios:

Session Creation Throughput:

• Create 5, 10, 15, 20 sessions concurrently
• Measure creation time (target: 10 sessions/min)
• Identify bottlenecks
• Database connection pool usage
• Agent command queue depth

Resource Profiling:

• API memory/CPU under load (20+ sessions)
• Agent memory/CPU under load
• Database connection count
• VNC streaming latency measurements
• WebSocket connection overhead

Load Testing:

• 50 concurrent sessions (stress test)
• Monitor system behavior at limits
• Document failure points
• Resource requirements

Expected Output: .claude/reports/INTEGRATION_TEST_PERFORMANCE_LOAD.md

---

8. Cross-Platform Session Management

Test: Managing sessions across K8s and Docker platforms

Scenarios:

• K8s agent online, Docker agent online
• Create sessions on both platforms
• List all sessions (mixed platforms)
• Filter by platform (K8s vs Docker)
• Terminate sessions on both platforms
• Agent platform failover (if supported)

Expected Output: .claude/reports/INTEGRATION_TEST_CROSS_PLATFORM.md

---

9. Database Migration Scenarios

Test: Database schema evolution and rollback

Scenarios:

• Fresh installation (all migrations)
• Incremental migration (v1.x → v2.0)
• Migration rollback (reverse migrations)
• Data integrity after migration
• Schema validation
• Constraint verification

Expected Output: .claude/reports/INTEGRATION_TEST_DATABASE_MIGRATIONS.md

---

Test Infrastructure Requirements

Environment Setup

Kubernetes Cluster:

# Local testing with Kind
kind create cluster --name streamspace-test
kubectl apply -f manifests/

# Deploy with HA
helm install streamspace ./chart \
  --set replicaCount=3 \
  --set redis.enabled=true \
  --set redis.agentHubEnabled=true

Docker Environment:

# Docker daemon with API access
docker info

# Docker Compose for test stack
docker-compose -f docker-compose.test.yml up -d

Dependencies

• Kind (Kubernetes in Docker)
• Redis (for multi-pod API)
• PostgreSQL (test database)
• Docker daemon
• Test container images

Test Data

• Sample Template CRDs
• Sample user accounts
• Test container images
• Mock VNC servers

---

Deliverables

1. Integration Test Scripts (10+ files):

   • tests/integration/ha_multi_pod_api_test.go
   • tests/integration/ha_k8s_agent_leader_election_test.go
   • tests/integration/docker_agent_lifecycle_test.go
   • tests/integration/vnc_streaming_e2e_test.go
   • tests/integration/ha_chaos_testing_test.go
   • tests/integration/multi_user_concurrent_test.go
   • tests/integration/performance_load_test.go
   • tests/integration/cross_platform_test.go
   • tests/integration/database_migrations_test.go

2. Test Reports (9+ files in .claude/reports/):

   • One detailed report per integration test scenario
   • Performance benchmarks
   • Failure analysis (if any)

3. Test Infrastructure:

   • Docker Compose for test stack
   • Kind cluster configuration
   • Test data fixtures
   • Helper scripts

4. Documentation:

   • Integration test execution guide
   • Environment setup instructions
   • Troubleshooting guide

---

Success Criteria

• ✅ 50+ integration test scenarios
• ✅ All HA features validated
• ✅ VNC streaming E2E confirmed
• ✅ Multi-platform operation verified
• ✅ Performance benchmarks documented
• ✅ Zero-downtime failover validated
• ✅ Load testing completed (50+ sessions)
• ✅ All tests passing in CI/CD

---

References

• Existing Integration Tests: tests/integration/
• Test Coverage Analysis: .claude/reports/TEST_COVERAGE_ANALYSIS_2025-11-23.md
• Integration Test Plan: .claude/reports/INTEGRATION_TESTING_PLAN.md
• Wave 15-17 Deliveries: MULTI_AGENT_PLAN.md

---

Estimated Effort

3-4 days (24-32 hours)

• Environment setup: 4-8 hours
• Test implementation: 12-16 hours
• Test execution & reports: 8-12 hours

---

Acceptance Criteria

1. All 9 integration test scenarios implemented
2. 50+ test cases total
3. All HA features validated
4. VNC streaming E2E confirmed
5. Performance benchmarks documented
6. All tests passing
7. CI/CD pipeline integration complete

[JoshuaAFerguson]

Closing — the VNC leg of this integration suite is obsolete (#261 removed VNC). The integration testing need is real; recommend opening a fresh [TEST] Selkies end-to-end integration suite issue framed around the current pipeline (/api/v1/http/<session>/ proxy → Selkies pod → WebRTC). The HA + multi-platform legs are still wanted.