Add support for Codex and Gemini#2
Open
mgorunuch wants to merge 3 commits into
Open
Conversation
This commit enables OpenAI Codex and Google Gemini as runner options for task execution, expanding the available AI providers beyond Claude. Changes: - Enable codex and gemini runners in pipeline-executor.ts - Remove from disabled runners list - Add to runnerKeyMap with appropriate API key requirements - OPENAI_API_KEY for codex - GOOGLE_API_KEY for gemini - Implement codex-pipeline.ts with OpenAI SDK integration - Add workspace cloning and git branch management - Integrate OpenAI SDK for code generation - Support MOCK_MODE for testing - Track usage metrics and costs - Push changes to file spaces with merge requests - Implement gemini-pipeline.ts with Google Generative AI SDK - Add workspace cloning and git branch management - Integrate Google Generative AI SDK - Support MOCK_MODE for testing - Track usage metrics and costs - Push changes to file spaces with merge requests Both pipelines follow the same pattern as claude-pipeline.ts: - Validate environment variables - Fetch session and task from API - Clone and prepare workspaces - Execute AI agent with task prompt - Run completion and clarification checks - Push results and create merge requests
This commit adds two new worker microservices for executing AI tasks using OpenAI Codex and Google Gemini, completing the multi-provider support for the ADI system. New Microservices: - micros-codex-worker: OpenAI Codex worker for task evaluation and implementation - micros-gemini-worker: Google Gemini worker for task evaluation and implementation Features: - RabbitMQ-based task queue integration via worker-sdk - Support for both evaluation and implementation phases - Workspace cloning and git branch management - Cost tracking and usage metrics - Graceful shutdown and error handling - Configurable concurrency Architecture: Both workers follow the same pattern as micros-claude-worker: - src/index.ts: Main worker entry point with RabbitMQ integration - src/utils/*-executor.ts: AI model execution logic - src/utils/workspace-cloner.ts: Shared workspace management Configuration: Codex Worker: - RABBITMQ_URL: RabbitMQ connection URL - WORKER_NAME: Worker identifier (default: codex-worker) - MAX_CONCURRENCY: Max parallel tasks (default: 3) - OPENAI_API_KEY: OpenAI API key (can be overridden per task) Gemini Worker: - RABBITMQ_URL: RabbitMQ connection URL - WORKER_NAME: Worker identifier (default: gemini-worker) - MAX_CONCURRENCY: Max parallel tasks (default: 3) - GOOGLE_API_KEY: Google API key (can be overridden per task) These microservices work alongside the existing pipeline-based runners to provide flexible execution options for different use cases.
This commit adds Docker support for the Codex and Gemini worker microservices, enabling containerized deployment of the multi-provider AI system.
Changes:
- docker-compose.prod.yaml: Add micros-codex-worker and micros-gemini-worker services
- packages/micros-codex-worker/Dockerfile: Multi-stage Docker build for Codex worker
- packages/micros-gemini-worker/Dockerfile: Multi-stage Docker build for Gemini worker
Docker Services:
Both services follow the same architecture as micros-claude-worker:
- Depend on RabbitMQ and database migrations
- Support configurable concurrency
- Include health checks for monitoring
- Auto-restart on failure
- Run with tsx for TypeScript support
Environment Variables:
Codex Worker:
- RABBITMQ_URL: RabbitMQ connection
- WORKER_NAME: codex-worker
- MAX_CONCURRENCY: ${CODEX_WORKER_CONCURRENCY:-3}
- OPENAI_API_KEY: OpenAI API key (optional fallback)
- OPENAI_API_BASE: Custom endpoint (optional)
- OPENAI_ORGANIZATION: Organization ID (optional)
Gemini Worker:
- RABBITMQ_URL: RabbitMQ connection
- WORKER_NAME: gemini-worker
- MAX_CONCURRENCY: ${GEMINI_WORKER_CONCURRENCY:-3}
- GOOGLE_API_KEY: Google API key (optional fallback)
Deployment:
# Start all services including new workers
docker-compose -f docker-compose.prod.yaml up -d
# Scale workers independently
docker-compose -f docker-compose.prod.yaml up -d --scale micros-codex-worker=2
docker-compose -f docker-compose.prod.yaml up -d --scale micros-gemini-worker=2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This commit enables OpenAI Codex and Google Gemini as runner options for task execution, expanding the available AI providers beyond Claude.
Changes:
Enable codex and gemini runners in pipeline-executor.ts
Implement codex-pipeline.ts with OpenAI SDK integration
Implement gemini-pipeline.ts with Google Generative AI SDK
Both pipelines follow the same pattern as claude-pipeline.ts: