Full Multi-Tenancy Implementation: Run Multiple Tenants Simultaneously

[Christiantyemele]

Problem Statement

Currently, OpenFlows can only run one tenant at a time. When starting the controller with ./scripts/prod.sh run, it uses the environment variables GITHUB_REPOSITORY and OPENFLOWS_TENANT to configure a single tenant. Multiple tenants can be added via ./scripts/prod.sh tenant, but each nexus workspace is isolated and the controller only serves one at a time.

This limits OpenFlows to single-team deployments and prevents true multi-tenant SaaS use cases.

Current Limitation

# Can add multiple tenants
./scripts/prod.sh tenant The-AgenticFlow/openflows --name my-team
./scripts/prod.sh tenant other-org/repo --name another-team

# But controller only runs one
./scripts/prod.sh run  # Uses OPENFLOWS_TENANT=default + GITHUB_REPOSITORY from .env

The Nexus workspace for each tenant runs independently. There's no single control plane that orchestrates work across all registered tenants.

Required Implementation

1. Controller Architecture Change

• Discovery: Controller queries Redis or Coder API to discover all registered tenants at startup
• Per-Tenant Orchestration Loop: Each tenant gets its own independent orchestration loop (NEXUS→FORGE→SENTINEL→VESSEL cycle)
• Shared Redis Namespace: Use tenant prefixes (ns:{tenant}:*) to isolate state per tenant (already implemented per docs/tenancy.md)
• Concurrent Polling: Run all tenant loops concurrently with configurable poll intervals

2. Tenant Configuration Management

• Tenant Registry: Store tenant metadata in Redis or a config file
• Environment Variable or Config File: Option to load tenants from file or environment
• Add/Remove Tenants at Runtime: New command to enable/disable tenants without restarting

3. CLI Changes

# Start controller with all enabled tenants
./scripts/prod.sh run

# Or filter to specific tenant(s)
./scripts/prod.sh run --tenants my-team,another-team

# Or list registered tenants
./scripts/prod.sh tenants list

# Enable/disable tenants at runtime
./scripts/prod.sh tenants enable my-team
./scripts/prod.sh tenants disable my-team

4. Logging & Observability

• Per-Tenant Logging: Include tenant ID in all log messages
• Metrics: Track per-tenant stats (issues processed, PRs merged, worker utilization)
• Health Checks: ./scripts/prod.sh doctor should report per-tenant health

5. Error Isolation

• Tenant Failure Isolation: If tenant A's loop crashes, tenant B continues running
• Recovery Per Tenant: Use existing recovery logic, but scoped to each tenant

Implementation Steps

1. Design Phase

   • Decide on tenant discovery mechanism (Redis vs config file vs Coder API)
   • Define tenant registry schema
   • Plan concurrent loop architecture

2. Core Changes

   • Modify openflows::orchestration::Controller to discover and iterate over tenants
   • Create TenantRegistry struct to manage tenant state
   • Implement per-tenant PocketFlow execution with Redis namespace isolation
   • Update openflows run command to handle multi-tenant startup

3. CLI & Configuration

   • Add tenants list, tenants enable, tenants disable commands
   • Support --tenants flag for selective startup
   • Load tenant list from environment or config file

4. Testing

   • Unit tests for TenantRegistry
   • Integration tests: Add 3+ tenants, verify issues are dispatched to correct nexus
   • Verify Redis namespace isolation per tenant
   • Chaos test: Crash one tenant loop, verify others continue

5. Documentation

   • Update docs/tenancy.md with multi-tenant running instructions
   • Update README.md Quick Start to show adding 2+ tenants
   • Document tenant registry format

Acceptance Criteria

• Controller starts with all enabled tenants registered in tenant registry
• Each tenant gets its own independent orchestration loop
• Issues created in tenant A are routed to tenant A's workers; tenant B issues go to tenant B workers
• Redis state is isolated per tenant (no key collisions)
• One tenant's failure does not affect others
• Logs clearly indicate which tenant each action belongs to
• ./scripts/prod.sh doctor reports per-tenant health
• Tenants can be enabled/disabled without restarting controller

Related Issues

• See docs/tenancy.md for existing multi-tenant state isolation design
• See crates/coder-client/src/bootstrap.rs for tenant creation flow