test: Add comprehensive integration testing for end-to-end workflows

[doughayden]

Testing Enhancement

Add comprehensive integration testing to cover complete user workflows and improve confidence in system reliability.

Level of Effort: 🔥 Large (4-6 days)

• Test framework setup: 1-2 days for integration test infrastructure
• Test implementation: 2-3 days for comprehensive workflow coverage
• CI/CD integration: 1 day for automated testing pipeline

Current Testing State

Strengths:

• Excellent unit test coverage: 94% overall
• Comprehensive mocking strategy in tests/conftest.py
• Individual component testing well-established

Testing gaps:

• No end-to-end integration tests
• Missing complete user workflow validation
• No testing of service-to-service communication
• Limited error scenario coverage across services

Missing Integration Test Scenarios

1. Complete User Workflows

# Example: Full query workflow
async def test_complete_query_workflow():
    # OAuth authentication → Query submission → 
    # Discovery Engine processing → Response formatting → 
    # BigQuery logging → User feedback → Feedback logging

2. Service Communication

• Frontend → Backend authentication flow
• Backend → Discovery Engine API interaction
• Backend → BigQuery logging pipeline
• Error propagation between services

3. Error Scenarios

• Network failures between services
• API timeouts and retries
• Authentication failures
• Invalid responses from external services

4. Performance Under Load

• Concurrent user scenarios
• Rate limiting behavior
• Resource exhaustion handling

Recommended Implementation

1. Integration Test Framework

# tests/integration/conftest.py
@pytest.fixture
async def integration_client():
    """Real HTTP client for integration tests."""
    async with httpx.AsyncClient() as client:
        yield client

@pytest.fixture
def test_environment():
    """Test environment configuration."""
    return {
        "backend_url": "http://localhost:8888",
        "test_user": "integration-test@example.com"
    }

2. Test Categories

Category A: Authentication Flow

• OAuth token acquisition
• Service-to-service authentication
• Token refresh scenarios
• Authentication failure handling

Category B: Query Processing

• End-to-end query submission
• Discovery Engine integration
• Response formatting and citations
• BigQuery logging verification

Category C: User Management

• Session creation and management
• User preference handling
• Session cleanup and deletion

Category D: Error Handling

• Network failure scenarios
• API timeout handling
• Invalid input processing
• Service unavailability responses

3. Test Environment Setup

# tests/integration/docker-compose.test.yml
services:
  answer-app:
    build: ../../src/answer_app
    environment:
      - GOOGLE_CLOUD_PROJECT=test-project
  
  client:
    build: ../../src/client
    depends_on:
      - answer-app

Implementation Structure

New Test Files:

• tests/integration/test_auth_workflow.py: Authentication integration tests
• tests/integration/test_query_workflow.py: Complete query processing tests
• tests/integration/test_error_scenarios.py: Error handling integration tests
• tests/integration/test_performance.py: Basic performance and load tests

Test Infrastructure:

• tests/integration/conftest.py: Integration test fixtures
• tests/integration/helpers.py: Common integration test utilities
• tests/integration/docker-compose.test.yml: Test environment setup

CI/CD Integration

GitHub Actions Workflow:

integration-tests:
  runs-on: ubuntu-latest
  steps:
    - name: Start test environment
      run: docker-compose -f tests/integration/docker-compose.test.yml up -d
    
    - name: Run integration tests
      run: poetry run pytest tests/integration/
    
    - name: Cleanup test environment
      run: docker-compose -f tests/integration/docker-compose.test.yml down

Testing Strategy

Mock vs Real Services:

• Real services: HTTP communication, authentication flows
• Mocked services: BigQuery, Discovery Engine API (for cost control)
• Hybrid approach: Real local services, mocked external APIs

Test Data Management:

• Isolated test datasets
• Cleanup procedures for test artifacts
• Reproducible test scenarios

Acceptance Criteria

• Authentication workflow integration tests
• Complete query processing integration tests
• Error scenario integration tests
• Service communication validation
• Test environment automation (Docker Compose)
• CI/CD pipeline integration
• Test data management strategy
• Documentation for running integration tests

Priority

High - Integration tests significantly improve confidence in system reliability and catch issues that unit tests miss.

Benefits

1. Earlier bug detection: Catch integration issues before deployment
2. Deployment confidence: Verify complete workflows work end-to-end
3. Regression prevention: Ensure changes don't break existing functionality
4. Documentation: Integration tests serve as executable documentation
5. Onboarding: Help new developers understand system interactions