feat: Add prepare endpoint for two-step build workflow #2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add optional two-step build process that allows users to preview and approve package changes before building. ## New Features ### API Endpoints - Add POST /api/v1/build/prepare endpoint for package resolution - Add skip_package_resolution query parameter to POST /api/v1/build ### Package Resolution - New asu/package_resolution.py module with PackageResolver class - Tracks package changes (migrations, additions, removals) - Supports version-specific package migrations - Provides detailed reasons for package changes ### Build Request Model - Add from_version field to BuildRequest for upgrade tracking - Helps apply correct package migrations when upgrading ### Build Process - Update build() to accept skip_package_resolution parameter - Skip package changes when building from prepared request - Maintain backward compatibility with existing clients ## Implementation Details The prepare endpoint: 1. Validates the build request 2. Applies package changes based on version/target/profile 3. Returns resolved packages and detailed change list 4. Checks cache availability 5. Provides prepared request for /build endpoint The two-step workflow: 1. Client calls /build/prepare to see what will change 2. User reviews and approves package changes 3. Client calls /build with skip_package_resolution=true 4. Build uses exact packages from prepare step ## Testing - Add comprehensive tests for PackageResolver - Add tests for prepare endpoint - Add integration tests for prepare → build workflow - Test package migrations (auc → owut) - Test hardware-specific additions - Test language pack renames ## Documentation - Update README with two-step build process examples - Document prepare endpoint usage - Document backward compatibility ## Backward Compatibility - Existing clients work unchanged - /build endpoint still applies changes automatically - Two-step process is entirely optional - No breaking changes to existing API This incremental improvement adds maximum value with minimal code changes (~400 lines total) and maintains full backward compatibility.
Refactor ASU to support independent microservices deployment where prepare and build endpoints can run in separate containers. ## Architecture Changes ### New Microservices **1. Prepare Service (Lightweight)** - Independent service in `asu/services/prepare_service.py` - Handles `/api/v1/build/prepare` endpoint - NO dependencies on Redis, RQ, Podman, or ImageBuilder - Stateless, fast (<1s response), minimal resources (512MB RAM, 0.5 CPU) - Can scale horizontally independently - Only performs validation and package resolution **2. Build Service (Heavy)** - Independent service in `asu/services/build_service.py` - Handles `/api/v1/build` endpoint - Requires Redis, RQ, Podman, and ImageBuilder - Resource-intensive (4GB+ RAM, 4+ CPU) - Manages build queue and cache - Can scale workers independently ### Shared Libraries Both services share common code but not runtime state: - `asu/build_request.py` - Request models - `asu/package_resolution.py` - Resolution logic - `asu/package_changes.py` - Migration rules - `asu/config.py` - Configuration - `asu/util.py` - Utilities ### API Router Updates - `asu/routers/api.py` refactored to use services - Prepare endpoint uses `PrepareService` - Build endpoint uses `BuildService` - Services instantiated via singleton pattern ## Deployment Configuration ### Containerization **Prepare Service:** - New `Containerfile.prepare` - Minimal image with no heavy dependencies - Port 8001 **Build Service:** - New `Containerfile.build` - Full image with all build tools - Port 8000 ### Orchestration **New `podman-compose.microservices.yml`:** - Separate containers for prepare, build, workers, redis - Nginx load balancer for routing - Independent scaling for each service - Resource limits per service **Nginx Configuration:** - New `misc/nginx-microservices.conf` - Routes `/api/v1/build/prepare` → prepare service - Routes `/api/v1/build` → build service - Different timeouts per service ## Testing **New `tests/test_services.py`:** - Test service independence - Verify prepare has no Redis/Podman dependencies - Test both services use same models - Validate singleton pattern ## Documentation **Updated `README.md`:** - Added microservices architecture section - Deployment options (monolithic vs microservices) - Benefits and trade-offs - Resource requirements **New `ARCHITECTURE.md`:** - Comprehensive architecture documentation - Service responsibilities and dependencies - Deployment models and scaling strategies - Communication patterns - Security considerations - Migration path from monolithic - Troubleshooting guide ## Benefits ✅ **Independent Scaling**: Scale prepare and build separately ✅ **Cost Efficiency**: Run many cheap prepare instances, fewer expensive build instances ✅ **Better Performance**: Prepare requests don't wait for build resources ✅ **Resource Isolation**: Failures isolated to specific services ✅ **Deployment Flexibility**: Deploy services on different infrastructure ✅ **Backward Compatible**: Monolithic deployment still works ## Deployment Options ```bash # Option 1: Monolithic (default, backward compatible) podman-compose up -d # Option 2: Microservices (new, recommended for production) podman-compose -f podman-compose.microservices.yml up -d ``` ## Design Principles 1. **Shared Code, Separate Runtime**: Services share Python modules but not state 2. **Single Responsibility**: Each service has one clear purpose 3. **Independence**: Services can be deployed and scaled separately 4. **Backward Compatibility**: Existing deployment method still works 5. **Production Ready**: Includes monitoring, health checks, resource limits This refactoring enables ASU to scale efficiently in production while maintaining simplicity for development and small deployments.
aparcar
force-pushed
the
claude/analyze-code-PCj61
branch
from
e3f04d4 to
2ac84ac
Compare
Convert to true microservices architecture where prepare and build
services have COMPLETELY SEPARATE CODEBASES with NO shared code.
Services communicate ONLY via HTTP.
```
/
├── asu/ # Build service (heavy, existing codebase)
│ └── Calls prepare service via HTTP
└── asu-prepare/ # Prepare service (NEW, separate codebase)
├── main.py
├── build_request.py
├── package_changes.py
├── package_resolution.py
├── config.py
└── Containerfile
```
NEW completely independent microservice for package resolution:
**Files Created:**
- `asu-prepare/main.py` - Standalone FastAPI app (170 lines)
- `asu-prepare/build_request.py` - Data models (175 lines)
- `asu-prepare/package_changes.py` - Migration logic (copied, 198 lines)
- `asu-prepare/package_resolution.py` - Resolution (copied, 195 lines)
- `asu-prepare/config.py` - Minimal config (35 lines)
- `asu-prepare/pyproject.toml` - Minimal dependencies
- `asu-prepare/Containerfile` - Lightweight container
- `asu-prepare/README.md` - Documentation
**Dependencies:** ONLY FastAPI, Pydantic, Uvicorn
**NO Dependencies on:** Redis, RQ, Podman, ImageBuilder, asu/*
**API:**
- `POST /api/v1/prepare` - Package resolution
- `GET /health` - Health check
**Resources:**
- 512MB RAM, 0.5 CPU
- <1s response time
- Completely stateless
Modified to call prepare service via HTTP:
**Modified Files:**
- `asu/routers/api.py`:
- `/build/prepare` now proxies to prepare service via HTTP
- Removed services imports
- Restored original build logic
- Added httpx for HTTP calls
- `asu/config.py`:
- Added `prepare_service_url` setting
- Default: `http://asu-prepare:8001`
**Deleted:**
- `asu/services/` directory (no longer sharing code)
- `Containerfile.prepare` (moved to asu-prepare/)
- `Containerfile.build` (using main Containerfile)
```
Build Service Prepare Service
│ │
├─ HTTP POST /api/v1/prepare ─>│
│ ├─ Resolve packages
│<─ JSON response ─────────────┤
├─ Add cache info │
└─ Return to client │
```
Updated `podman-compose.microservices.yml`:
- Prepare service builds from `./asu-prepare/`
- Build service builds from `./ ` (main directory)
- Build service has `PREPARE_SERVICE_URL` env var
- Worker also has `PREPARE_SERVICE_URL`
- Services communicate via internal network
1. **No Shared Code**
- Each service has its OWN copy of files
- NO `from asu.X import Y` between services
- Can version/deploy independently
2. **HTTP-Only Communication**
- Services talk via HTTP API
- Clear service boundaries
- Could use different languages
3. **Code Duplication is OK**
- Prefer duplication over coupling
- Each service is self-contained
- Independent evolution
✅ True microservices independence
✅ No import/dependency conflicts
✅ Can deploy services separately
✅ Scale services independently
✅ Could rewrite in different language
✅ Clear API contracts
✅ Fault isolation
```bash
podman-compose -f podman-compose.microservices.yml up -d
podman-compose up -d
```
- Updated `ARCHITECTURE.md` with new design
- Added `asu-prepare/README.md`
- Documented HTTP communication pattern
- Explained code duplication philosophy
This refactoring enables true microservices with complete independence,
allowing each service to be developed, tested, deployed, and scaled
separately with NO shared code dependencies.
aparcar
force-pushed
the
claude/analyze-code-PCj61
branch
from
2ac84ac to
f56301a
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add optional two-step build process that allows users to preview and
approve package changes before building.
New Features
API Endpoints
Package Resolution
Build Request Model
Build Process
Implementation Details
The prepare endpoint:
The two-step workflow:
Testing
Documentation
Backward Compatibility
This incremental improvement adds maximum value with minimal code
changes (~400 lines total) and maintains full backward compatibility.