Addon System Overhaul - Database-Backed Addons with Visual DAG Builder

[tsanders-rh]

Addon System Overhaul - Database-Backed Addons with Visual DAG Builder

Overview

Redesign the addon system to support:

1. Dual Storage Model: System addons (YAML, read-only) + User addons (database, editable)
2. Profile Migration: Convert all inline postDeployment to defaultAddons references
3. Visual DAG Builder: Advanced drag-and-drop interface for creating custom addons
4. Immutable Versioning: Published addon versions cannot be edited, only cloned

Current Pain Points:

• 9 profiles with inline postDeployment configurations (duplicated logic)
• No UI for creating/editing addons (YAML editing only)
• Profile-level and addon-level configs can conflict (e.g., oadp160win528 cluster)
• Addons can't be shared across profiles without duplication
• No version control for addons

Expected Benefits:

• Reusable addon library accessible to all users
• Visual DAG builder reduces barrier to creating complex post-deployment workflows
• Profiles reference addons by ID (DRY principle)
• Clear separation: profiles define infrastructure, addons define software
• Version control enables safe iteration and rollback

---

Architecture Design

Dual Storage Model

System Addons (Curated, Read-Only):

• Storage: YAML files in internal/addon/definitions/
• Synced to DB: On API startup (existing behavior)
• Marked with: addon_source = 'system', is_immutable = true
• Examples: cnv-nightly, oadp, mtc, mta, rhwa
• Purpose: Officially supported, well-tested addon catalog

User Addons (Custom, Editable):

• Storage: PostgreSQL only (no YAML files)
• Marked with: addon_source = 'user', created_by_user_id
• Versioning: Draft → Published (immutable) → New Draft (clone)
• Examples: Custom operator bundles, internal tooling, experimental configs
• Purpose: User-created addons via visual DAG builder

Addon Type System

Each addon can contain four task types (existing structure):

1. Operators - Install OLM operators (source, channel, approval mode)
2. Scripts - Execute bash scripts (inline or S3 URIs)
3. Manifests - Apply Kubernetes YAML (inline or S3 URIs)
4. Helm Charts - Install Helm charts (repo URL, values)

Tasks support dependsOn for DAG-based execution order.

Profile → Addon References

Before (inline postDeployment):

# aws-virt-windows-minimal-ga.yaml
postDeployment:
  operators:
    - name: kubevirt-hyperconverged
      source: redhat-operators
      channel: stable-4.18

After (addon references):

# aws-virt-windows-minimal-ga.yaml
defaultAddons:
  - cnv-nightly  # Addon ID
  - cnv-windows-vm  # Addon ID (new, extracted from inline configs)

When a cluster is created:

1. Profile's defaultAddons are pre-selected in UI
2. User can add/remove addons before cluster creation
3. Selected addons are stored in cluster.selected_addon_ids (new field)
4. Post-deployment executes addons in dependency order

---

Database Schema Changes

Migration 00058: Addon System Enhancements

-- Add addon source tracking and versioning
ALTER TABLE post_config_addons
  ADD COLUMN addon_source VARCHAR(20) DEFAULT 'system' CHECK (addon_source IN ('system', 'user')),
  ADD COLUMN created_by_user_id VARCHAR(255),
  ADD COLUMN is_published BOOLEAN DEFAULT FALSE,
  ADD COLUMN published_at TIMESTAMP WITH TIME ZONE,
  ADD COLUMN parent_version_id VARCHAR(64),  -- For version lineage (draft cloned from this)
  ADD COLUMN version_number INTEGER DEFAULT 1,  -- Incremental version counter
  ADD COLUMN is_immutable BOOLEAN DEFAULT FALSE;  -- Published versions can't be edited

-- Create index for user addon queries
CREATE INDEX idx_post_config_addons_user ON post_config_addons(created_by_user_id) WHERE addon_source = 'user';

-- Create index for version lineage queries
CREATE INDEX idx_post_config_addons_parent ON post_config_addons(parent_version_id);

-- Add check constraint: system addons must be immutable
ALTER TABLE post_config_addons
  ADD CONSTRAINT system_addons_immutable CHECK (
    (addon_source = 'system' AND is_immutable = true) OR addon_source = 'user'
  );

-- Add check constraint: published user addons must be immutable
ALTER TABLE post_config_addons
  ADD CONSTRAINT published_addons_immutable CHECK (
    (is_published = false) OR (is_published = true AND is_immutable = true)
  );

Migration 00059: Cluster Addon References

-- Add selected addon IDs to clusters table
ALTER TABLE clusters
  ADD COLUMN selected_addon_ids TEXT[];  -- Array of addon IDs

-- Create GIN index for array queries (e.g., "which clusters use this addon?")
CREATE INDEX idx_clusters_selected_addons ON clusters USING GIN(selected_addon_ids);

-- Backfill existing clusters with empty array
UPDATE clusters SET selected_addon_ids = '{}' WHERE selected_addon_ids IS NULL;

-- Add NOT NULL constraint after backfill
ALTER TABLE clusters ALTER COLUMN selected_addon_ids SET DEFAULT '{}';
ALTER TABLE clusters ALTER COLUMN selected_addon_ids SET NOT NULL;

---

Implementation Phases

Phase 1: Database & Backend (Week 1-2)

1.1 Database Migrations

• Files: internal/store/migrations/00058_addon_system_enhancements.sql
• Files: internal/store/migrations/00059_cluster_addon_references.sql
• Action: Create and test migrations locally with rollback scripts

1.2 Update Go Types

• File: pkg/types/cluster.go

  • Add SelectedAddonIDs []string to Cluster struct

• File: pkg/types/addon.go (may need to create)

  • Add fields: AddonSource, CreatedByUserID, IsPublished, PublishedAt, ParentVersionID, VersionNumber, IsImmutable

1.3 Update Database Store Layer

• File: internal/store/addons.go

  • Update GetAllAddons() to support filtering by source (system/user/all)
  • Add GetUserAddons(userID string) for user's addon library
  • Add CreateUserAddon(addon *types.Addon, userID string)
  • Add UpdateUserAddon(id string, addon *types.Addon) (only if not immutable)
  • Add PublishAddon(id string) (set is_published=true, is_immutable=true)
  • Add CloneAddon(parentID string, userID string) for versioning
  • Update GetAddonByID() to include all new fields

• File: internal/store/clusters.go

  • Update CreateCluster() to accept SelectedAddonIDs
  • Add GetClustersByAddonID(addonID string) for impact analysis

1.4 API Endpoints

• File: internal/api/handler_addons.go (new or update existing)

  • GET /api/v1/addons - List all addons (filter by source, category, platform)
  • GET /api/v1/addons/user - Get current user's custom addons
  • GET /api/v1/addons/:id - Get addon details (including version history)
  • POST /api/v1/addons - Create new user addon (draft mode)
  • PUT /api/v1/addons/:id - Update user addon (only if not immutable)
  • POST /api/v1/addons/:id/publish - Publish addon (makes immutable)
  • POST /api/v1/addons/:id/clone - Clone published addon to new draft
  • DELETE /api/v1/addons/:id - Delete user addon (only if not published)
  • GET /api/v1/addons/:id/usage - Get clusters using this addon

• File: internal/api/handler_clusters.go

  • Update cluster creation endpoint to accept selected_addon_ids
  • Return selected_addon_ids in cluster details response

1.5 Addon Sync Logic Update

• File: internal/addon/loader.go (or wherever sync happens)
  • Mark all YAML-sourced addons with addon_source='system', is_immutable=true
  • Skip deleting user addons during sync (only system addons)
  • Log addon sync statistics (system addons synced, user addons preserved)

---

Phase 2: Profile Migration (Week 2)

2.1 Create New System Addons

New Addon: internal/addon/definitions/cnv-windows-vm.yaml

id: cnv-windows-vm
name: "OpenShift Virtualization - Windows VM Support"
category: virtualization
enabled: true
supportedPlatforms: [openshift]
versions:
  - channel: stable
    displayName: "Windows VM Infrastructure"
    isDefault: true
    config:
      scripts:
        - name: setup-windows-vm-infrastructure
          uri: s3://ocpctl-binaries/manifests/windows-vm/auto-setup-irsa.sh
          timeout: 7200  # 2 hours (includes potential 30-50 min S3 download)
          env:
            WINDOWS_VM: "true"
          dependsOn: [kubevirt-hyperconverged]
metadata:
  description: "Automated Windows VM infrastructure setup with IRSA, S3 image import, and regional EBS snapshot support"
  notes:
    - "Requires CNV addon to be installed first (kubevirt-hyperconverged operator)"
    - "Fast path: 2-3 minutes in snapshot-enabled regions"
    - "Fallback path: 30-50 minutes for first deployment in new region"

New Addon: internal/addon/definitions/k8s-dashboard.yaml

id: k8s-dashboard
name: "Kubernetes Dashboard"
category: monitoring
enabled: true
supportedPlatforms: [eks, gke, iks]
versions:
  - channel: stable
    displayName: "Dashboard v2.7"
    isDefault: true
    config:
      manifests:
        - name: kubernetes-dashboard
          uri: https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml
metadata:
  description: "Official Kubernetes web UI for managing cluster resources"

2.2 Migrate Profiles

Update 9 profiles to use defaultAddons instead of inline postDeployment:

1. aws-virt-windows-minimal-ga.yaml

   • Remove postDeployment section (operators for CNV)
   • Add defaultAddons: [cnv-nightly, cnv-windows-vm]

2. aws-virt-windows-minimal-prerelease.yaml

   • Remove postDeployment section
   • Add defaultAddons: [cnv-nightly, cnv-windows-vm]

3. aws-virt-windows-minimal.yaml

   • Remove postDeployment section
   • Add defaultAddons: [cnv-nightly, cnv-windows-vm]

4. eks-minimal.yaml

   • Remove postDeployment section (K8s dashboard manifests)
   • Add defaultAddons: [k8s-dashboard]

5. eks-standard.yaml

   • Remove postDeployment section
   • Add defaultAddons: [k8s-dashboard]

6. iks-minimal.yaml

   • Remove postDeployment section
   • Add defaultAddons: [k8s-dashboard]

7. gcp-gke-standard.yaml

   • Remove postDeployment section
   • Add defaultAddons: [k8s-dashboard]

8. gcp-sno-ga.yaml

   • Remove empty postDeployment: {} section

9. gcp-standard.yaml

   • Remove empty postDeployment: {} section

2.3 Update Profile Types

• File: internal/profile/types.go
  • Mark PostDeployment *PostDeploymentConfig with // DEPRECATED: Use defaultAddons instead
  • Add DefaultAddons []string field

2.4 Profile Loader Update

• File: internal/profile/loader.go
  • Update validation to warn if both postDeployment and defaultAddons are present
  • Support both fields during transition period (backward compatibility)

---

Phase 3: Visual DAG Builder UI (Week 3-5)

3.1 Backend API for DAG Validation

• File: internal/api/handler_addons.go
  • POST /api/v1/addons/validate-dag - Validate DAG structure (cycle detection, dependency existence)
  • Returns validation errors or success

3.2 Frontend Components

Component: web/components/addons/AddonBuilder.tsx (new)

• Main component with tabbed interface:
  • Metadata tab (name, description, category, supported platforms)
  • DAG Editor tab (React Flow canvas)
  • Version History tab (timeline of published versions)
• State management: useState/useReducer for addon draft
• Auto-save to localStorage every 30s (prevent loss)
• Publish button (validates, calls API, marks immutable)

Component: web/components/addons/DAGCanvas.tsx (new)

• React Flow integration for visual DAG editing
• Node types: OperatorNode, ScriptNode, ManifestNode, HelmNode
• Edge rendering with dependency arrows
• Minimap for large DAGs
• Controls: zoom, fit view, auto-layout (dagre)
• Drag-and-drop from task palette to canvas

Component: web/components/addons/TaskNodeEditor.tsx (new)

• Property editor panel (appears when node selected)
• Task-specific forms:
  • Operator: name, source, channel, approvalMode, namespace
  • Script: name, uri (S3 or inline), timeout, env vars
  • Manifest: name, uri (URL or inline YAML)
  • Helm Chart: name, repo, chart, version, values (YAML editor)
• Dependency selector: multi-select dropdown of other tasks
• Validation: required fields, YAML syntax (for manifests/helm)

Component: web/components/addons/AddonVersionManager.tsx (new)

• Version timeline visualization (published versions with timestamps)
• "Clone to Draft" button for editing published versions
• Impact analysis: "X clusters using this version"
• Version diff viewer (show changes between versions)

Component: web/components/addons/AddonLibrary.tsx (new)

• Tabbed view: System Addons | My Addons | Published Community Addons
• Grid/list view toggle
• Search by name/description
• Filter by category, platform, source
• "Create New Addon" button → AddonBuilder
• "Edit Draft" / "Clone Published" actions

3.3 Update Cluster Creation Flow

• File: web/app/(dashboard)/clusters/create/page.tsx
  • After profile selection, show addon selection step
  • Pre-select profile's defaultAddons (checkboxes enabled)
  • Allow user to add/remove addons
  • Display addon descriptions in expandable cards
  • Show dependency warnings ("Addon X requires Addon Y")
  • Send selected_addon_ids in cluster creation request

3.4 React Flow Integration

• Dependencies: Add to web/package.json:
  • reactflow@^11.11.0 - Node-based graph editor
  • dagre@^0.8.5 - Auto-layout algorithm for DAGs
  • @monaco-editor/react@^4.6.0 - Inline YAML/script editing

---

Phase 4: Backward Compatibility & Cleanup (Week 5-6)

4.1 Hybrid Support in POST_CONFIGURE Handler

• File: internal/worker/handler_post_configure.go

  • Current behavior: Execute profile's postDeployment, then cluster's CustomPostConfig
  • New behavior:
    1. If cluster has selected_addon_ids:
       • Load addons from database by IDs
       • Merge all addon configs into single DAG
       • Execute combined DAG with dependency resolution
    2. Else if profile has legacy postDeployment:
       • Execute inline config (backward compatibility)
       • Log deprecation warning
    3. Then execute cluster's CustomPostConfig (user overrides)

• Priority: selected_addon_ids > profile.postDeployment (for migration period)

4.2 Deprecation Warnings

• API: Return deprecation header when profile has postDeployment
  • X-Deprecated-Feature: profile.postDeployment (use defaultAddons)
• UI: Show banner when editing profile with inline postDeployment
  • "This profile uses deprecated inline configuration. Migrate to addons."
• Worker Logs: Log warning when executing inline postDeployment

4.3 Profile Validation Update

• File: internal/profile/validation.go
  • Add validation rule: "If defaultAddons is specified, postDeployment should be empty"
  • Warning (not error) for backward compatibility

---

Phase 5: Testing & Validation (Week 6-7)

5.1 Unit Tests

• File: internal/store/addons_test.go

  • Test addon creation (system vs user)
  • Test immutability enforcement (published addons can't be updated)
  • Test version cloning (parent_version_id linkage)
  • Test filtering (GetUserAddons, GetAddonsBySource)

• File: internal/api/handler_addons_test.go

  • Test CRUD endpoints (create, read, update, delete)
  • Test publish workflow (draft → published)
  • Test authorization (user can only edit their own addons)
  • Test validation (DAG cycle detection, missing dependencies)

5.2 Integration Tests

• Test Case 1: Create cluster with defaultAddons

  • Create cluster from aws-virt-windows-minimal-ga profile
  • Verify selected_addon_ids = [cnv-nightly, cnv-windows-vm]
  • Verify post-deployment executes both addons in correct order
  • Verify Windows VM infrastructure is created

• Test Case 2: Create custom addon via UI

  • Use DAG builder to create addon with operator + script
  • Add dependency: script depends on operator
  • Publish addon
  • Verify immutability (update should fail)
  • Clone to new draft, verify version linkage

• Test Case 3: Backward compatibility

  • Create cluster from old profile with inline postDeployment
  • Verify post-deployment executes inline config
  • Verify deprecation warning in logs

• Test Case 4: Conflict resolution

  • Create profile with defaultAddons: [cnv-nightly]
  • User also selects cnv-nightly in UI (duplicate)
  • Verify deduplication (addon only executed once)

5.3 Manual Testing Checklist

• System addons sync correctly on API startup
• User can create addon via visual DAG builder
• Drag-and-drop task nodes works smoothly
• Dependency arrows render correctly
• Task property editor validates inputs
• Publish makes addon immutable
• Clone creates new draft with parent linkage
• Profile migration: all 9 profiles work with defaultAddons
• Cluster creation: defaultAddons pre-selected in UI
• Post-deployment: selected addons execute in dependency order
• Version history displays timeline correctly
• Impact analysis shows clusters using addon

---

Critical Files

Files to Create

1. internal/store/migrations/00058_addon_system_enhancements.sql - Database schema updates
2. internal/store/migrations/00059_cluster_addon_references.sql - Cluster addon references
3. internal/addon/definitions/cnv-windows-vm.yaml - Windows VM addon (extracted from profiles)
4. internal/addon/definitions/k8s-dashboard.yaml - K8s dashboard addon
5. web/components/addons/AddonBuilder.tsx - Main addon builder UI
6. web/components/addons/DAGCanvas.tsx - Visual DAG editor with React Flow
7. web/components/addons/TaskNodeEditor.tsx - Task property editor
8. web/components/addons/AddonVersionManager.tsx - Version history and cloning
9. web/components/addons/AddonLibrary.tsx - Addon browsing and management
10. internal/api/handler_addons.go - Addon CRUD API endpoints (may already exist, update)

Files to Modify

11. pkg/types/cluster.go - Add SelectedAddonIDs []string field
12. pkg/types/addon.go - Add versioning/source fields (may need to create)
13. internal/store/addons.go - Update addon queries with new fields
14. internal/store/clusters.go - Update cluster queries for selected_addon_ids
15. internal/profile/types.go - Add DefaultAddons []string, deprecate PostDeployment
16. internal/profile/loader.go - Support defaultAddons in profile loading
17. internal/worker/handler_post_configure.go - Execute selected_addon_ids
18. internal/addon/loader.go - Mark system addons as immutable during sync
19. web/app/(dashboard)/clusters/create/page.tsx - Addon selection UI in cluster creation
20. 9 Profile Files:
    • internal/profile/definitions/aws-virt-windows-minimal-ga.yaml
    • internal/profile/definitions/aws-virt-windows-minimal-prerelease.yaml
    • internal/profile/definitions/aws-virt-windows-minimal.yaml
    • internal/profile/definitions/eks-minimal.yaml
    • internal/profile/definitions/eks-standard.yaml
    • internal/profile/definitions/iks-minimal.yaml
    • internal/profile/definitions/gcp-gke-standard.yaml
    • internal/profile/definitions/gcp-sno-ga.yaml
    • internal/profile/definitions/gcp-standard.yaml

---

Deployment Strategy

Phase 1: Backend + Migrations (Deploy 1)

• Deploy migrations 00058, 00059
• Deploy updated API with addon CRUD endpoints
• Deploy updated worker with hybrid post-config execution
• Deploy new system addon YAMLs (cnv-windows-vm, k8s-dashboard)
• Verification: System addons sync correctly, new endpoints accessible

Phase 2: Profile Migration (Deploy 2)

• Deploy 9 updated profiles with defaultAddons
• Deploy updated profile loader supporting both patterns
• Verification: Profiles load correctly, no validation errors

Phase 3: UI (Deploy 3)

• Deploy visual DAG builder components
• Deploy updated cluster creation flow with addon selection
• Verification: Addon builder UI functional, cluster creation works

Phase 4: Monitoring & Iteration

• Monitor addon usage analytics
• Collect user feedback on DAG builder UX
• Iterate on UI/UX based on feedback

---

Rollback Plan

If Issues Arise:

1. Database rollback: Apply reverse migrations (drop new columns)
2. Profile rollback: Revert profiles to inline postDeployment (keep YAML backups)
3. API rollback: Deploy previous version (backward compatibility maintained)
4. UI rollback: Hide addon builder feature flag, use old AddonBrowser

Graceful Degradation:

• Profiles support BOTH defaultAddons AND postDeployment during transition
• Worker executes whichever is present (prioritize selected_addon_ids)
• Old clusters without selected_addon_ids continue using inline configs

---

Success Criteria

Quantitative Goals:

• All 9 profiles migrated to defaultAddons (zero inline configs)
• Users can create custom addon in <5 minutes via DAG builder
• Zero post-deployment failures due to addon system changes
• 10+ user-created addons within first month (indicates adoption)

Qualitative Goals:

• Users report DAG builder is intuitive and reduces complexity
• Profile maintainers report reduced duplication and easier updates
• Support tickets related to post-deployment conflicts eliminated
• Addon versioning enables safe experimentation and rollback

---

Verification Steps

After full implementation, verify end-to-end:

1. System Addon Sync

# Check API logs on startup
sudo journalctl -u ocpctl-api -n 100 | grep "addon"
# Expected: "Synced 6 system addons, preserved N user addons"

# Verify database
psql $DATABASE_URL -c "SELECT addon_id, addon_source, is_immutable FROM post_config_addons WHERE addon_source='system';"
# Expected: All system addons marked immutable

2. Profile Migration

# Check profile loading
curl http://localhost:8080/api/v1/profiles/aws-virt-windows-minimal-ga
# Expected: "defaultAddons": ["cnv-nightly", "cnv-windows-vm"]

# Verify no postDeployment in response
jq '.postDeployment' response.json
# Expected: null or absent

3. Create Custom Addon via UI

• Navigate to Addon Library → "Create New Addon"
• Drag operator node to canvas (name: test-operator, source: redhat-operators)
• Drag script node to canvas (name: post-install-script)
• Connect dependency: script depends on operator
• Fill metadata: name="Test Addon", category="testing"
• Click "Publish"
• Verify: Addon appears in library with version 1, is_published=true
• Verify: Edit button disabled (immutable)
• Click "Clone to Draft"
• Verify: New draft created with version 2, parent_version_id set

4. Cluster Creation with Addons

• Create cluster from aws-virt-windows-minimal-ga profile
• Verify: cnv-nightly and cnv-windows-vm pre-selected
• Add custom addon from library
• Submit cluster creation
• Verify: Database shows selected_addon_ids = [cnv-nightly, cnv-windows-vm, custom-addon-id]
• Monitor post-deployment logs
• Verify: All three addons execute in dependency order
• Verify: Windows VM infrastructure created successfully

5. Backward Compatibility

• Create cluster from old profile with inline postDeployment (if any remain for testing)
• Verify: Inline config executes correctly
• Verify: Deprecation warning in worker logs

6. Performance

• Create 10 clusters concurrently with different addon combinations
• Verify: All post-deployments complete without race conditions
• Verify: No duplicate addon executions (deduplication works)

---

Future Enhancements (Out of Scope)

Community Addon Sharing:

• Allow users to publish addons to community catalog
• Upvote/rating system for addons
• Addon marketplace with search and discovery

Addon Templates:

• Pre-built templates for common patterns (e.g., "Monitoring Stack", "CI/CD Pipeline")
• One-click template instantiation

Rollback Support:

• POST_CONFIGURE job type: ROLLBACK_ADDON
• Undo addon installation (operator uninstall, resource cleanup)

Advanced DAG Features:

• Conditional execution (if cluster.platform == 'openshift')
• Parallel task execution (tasks without dependencies run concurrently)
• Retry policies (maxAttempts, backoff strategy)

Addon Testing:

• Dry-run mode (validate addon without executing)
• Sandbox environment for testing addons

---

Timeline Summary

Week	Phase	Deliverables
1	Backend Foundation	Migrations 00058/00059, updated store layer
2	API & Profile Migration	Addon CRUD endpoints, 9 profiles migrated, 2 new addons
3	UI - DAG Builder Core	AddonBuilder, DAGCanvas, TaskNodeEditor components
4	UI - Versioning & Library	AddonVersionManager, AddonLibrary, cluster creation flow
5	Integration & Testing	Unit tests, integration tests, manual testing
6	Backward Compatibility	Hybrid post-config handler, deprecation warnings
7	Testing & Polish	End-to-end testing, bug fixes, UX improvements
8	Documentation	User guide, API docs, migration guide for profile authors
9	Production Deployment	Phased rollout (backend → profiles → UI), monitoring

Total Timeline: 9 weeks (2 months)

Critical Path: Database migrations → Profile migration → UI development

Parallel Work Opportunities:

• UI development can start while backend is being tested
• Profile migration can happen alongside API endpoint development
• Documentation can be written throughout all phases