feat(agentic): implement MAF-based agentic PDF processing pipeline with auth, structured logging, and comprehensive tests

[cherninkiy]

Summary

This PR introduces a major evolution of the PDF processing system — migration from a simple message-driven pipeline to a Microsoft Agent Framework (MAF) based agentic architecture with checkpoint/resume capabilities, JWT authentication, structured logging, and significantly expanded test coverage.

What's Changed

Agentic Architecture (MAF)

• Shared abstractions: introduced IAgent, IAgentOrchestrator, ICheckpointStore, AgentContext, AgentDefinition, AgentResult, WorkflowCheckpoint models in Shared project
• DocumentProcessingAgent: new MAF-based agent in Worker with full checkpoint support for resilient multi-step PDF processing
• PostgreSQL checkpoint store: PostgreSqlCheckpointStore for durable agent state persistence
• Consumer refactoring: PdfProcessingConsumer now delegates to DocumentProcessingAgent via MAF pipeline
• DI registration: MAF services registered in Worker DI container
• .NET 10 migration: updated Worker to .NET 10 and installed Microsoft.Agents.AI packages

Authentication

• JWT authentication added to ApiGateway with AuthenticationExtensions and TokenModels
• AuthController with dev token endpoint for development/testing
• Configuration updated in appsettings.json and appsettings.Development.json

Structured Logging

• Replaced default ILogger with Serilog structured logging across ApiGateway and Worker

Testing (846+ new test lines)

• DocumentProcessingAgentTests (465 lines): unit tests covering checkpoint scenarios (create, resume, complete)
• ConcurrencyTests (171 lines): concurrency and idempotency tests
• PdfProcessingConsumerRetryTests (182 lines): retry and dead-letter queue behavior
• OutboxPublisherTests (208 lines): outbox pattern reliability tests
• DocumentProcessingServiceTests: minor fixes and improvements

Infrastructure & Reliability

• Typed exceptions: DocumentProcessingException for better error handling
• CancellationToken checks throughout processing pipeline
• Docker Compose healthcheck for Worker service
• Prometheus MetricsHostedService: wrapped MetricServer in `IHostedService

[gemini-code-assist]

Code Review

This pull request migrates the PDF processing system to an 'Agentic' architecture using the Microsoft Agent Framework (MAF) and upgrades the entire solution to .NET 10. Key changes include the implementation of a checkpoint-based workflow in the Worker service to enable durable processing and resume capabilities, the addition of JWT authentication in the ApiGateway, and the integration of Serilog for structured logging. The test suite has been significantly expanded to cover the new MAF agent, checkpoint persistence, and concurrency scenarios. Feedback identifies several security and reliability improvements, including the need for secret length validation, environment-based restrictions for the development authentication endpoint, and more flexible configuration for the metrics server port. Additionally, a potential issue with exception message lengths exceeding database column limits in the checkpoint store was noted.

[gemini-code-assist]

The jwtSecret is used to create a SymmetricSecurityKey. For the HMAC SHA256 algorithm, the key must be at least 128 bits (16 bytes), but 256 bits (32 bytes) is recommended for security. If the configured secret is too short, the application will throw an exception at runtime during token validation. Consider adding a validation check for the secret length in the configuration logic.

[gemini-code-assist]

The AuthController allows generating JWT tokens using a symmetric key. This endpoint should be restricted to the Development environment to prevent security risks in production. You can inject IHostEnvironment and check env.IsDevelopment() in the constructor or use a conditional attribute.

[gemini-code-assist]

The Prometheus metric server port is hardcoded to 5091. It is recommended to inject IConfiguration and retrieve the port from settings to allow flexibility across different environments.

    public MetricsHostedService(IConfiguration configuration, ILogger<MetricsHostedService> logger)
    {
        _logger = logger;
        var port = configuration.GetValue<int>("Metrics:Port", 5091);
        _metricServer = new MetricServer(port: port);
    }

[gemini-code-assist]

The ErrorMessage column in the workflow_checkpoints table is limited to 4096 characters. If an exception message is longer, saving the checkpoint will fail. It's safer to truncate the message before assignment.

            existing.ErrorMessage = result.ErrorMessage?.Length > 4096 ? result.ErrorMessage[..4096] : result.ErrorMessage;