Refactor to local db by SecretLaboratory · Pull Request #3 · SecretLaboratory/cursor-s3-cms

SecretLaboratory · 2025-10-20T01:06:53Z

lets see

- Fixed databaseService.ts parseEntry() to handle both S3 and new data structures - S3 structure: metadata at root level alongside data fields - New structure: data wrapped with nested metadata and tags - Extracts title, slug, description to root level for UI compatibility - Fixed migrationService.ts to preserve metadata during S3 import - Now wraps entry data properly: { data, metadata, tags } - Previously was losing metadata by only importing data field - Updated ContentManager.tsx for consistent entry structure - Entry creation now includes empty metadata and tags - Entry updates properly wrap data, metadata, and tags - Removed unused imports (slugify, useLocalCollections, useLocalEntries) Result: Entry metadata (title, slug, description) now displays correctly in the entry editor after S3 import or database re-creation.

- Created new PublishingService that reads from local SQLite database - Publishing now reads collection/entry data from databaseService - Generates API JSON endpoints and uploads to S3 - Maintains separation of concerns: - DatabaseService: local data operations - ContentService: S3 write operations (for generated JSON) - PublishingService: orchestrates publishing workflow - Updated main.ts to initialize and configure PublishingService - Updated IPC handlers to use publishingService for: - generateAllApiEndpoints - generateSelectiveApiEndpoints - getLastGenerationDate Result: Publishing workflow no longer reads from S3, only local DB

- Comprehensive audit of S3 operations - Publishing workflow now fully local-first (reads from SQLite) - Identified legacy IPC handlers that are no longer used - Performance improvements documented (5-10x faster) - Includes architecture diagrams and testing checklist

- Import ContentEntry instead of non-existent Entry type - Add loadCollectionsWithEntries() helper method - Properly convert database Collection type (snake_case) to ContentCollection (camelCase) - Load entries and entry order for each collection - All publishing methods now use type-safe ContentCollection[]

- parseEntry() returns camelCase properties at runtime - Added type assertion to handle snake_case vs camelCase mismatch - Fixed debug logging to use metadata.title consistently - All TypeScript compilation errors resolved

…ction - Added schemaId, schemaName, bucketId, publishedAt, etc. to parseCollection() - Collections now properly display schema name instead of 'Unknown' - Fixes UI display issue where schema information wasn't available

- Add bucket_collection_order table to track collection order per bucket - Add updateCollectionOrder/getCollectionOrder methods to DatabaseService - Update listCollections to JOIN and ORDER BY collection order from bucket_collection_order - Add collection.updateOrder IPC handler and preload method - Wire up collection drag-and-drop to save order to database - Add MigrationService step to import collection order from S3 collections index - Fix DraggableCollectionItem TypeScript errors (add FC import, Schema interface) - Fix entry count not showing (add entries/entryOrder to Collection interface) - Fix collection display to use collection.description consistently - Update PublishingService schema lookup path to match old format - Changed from api/schemas/{schemaId}.json to api/collections/{SchemaName}.json - Added missing fields: description, metadata, published, publishedAt, etc. - Add getEntryOrderSync and getCollectionOrderSync helper methods - Update listCollections to include entryOrder for each collection - Update getCollection to include entryOrder Entry and collection order now persists across: - UI reloads ✓ - Publishing to S3 ✓ - S3 imports/migration ✓

- Add build:copy-database script to copy SQL file to dist directory - Include src/main/database in electron-builder files array - Add additional path resolution checks for packaged app locations - Ensures S3 import migration runs correctly in production builds The migration was failing in production because schema.sql wasn't being included in the packaged app. Now the build process copies it to dist/ and electron-builder includes it in the package.

- Add assets table to database schema to track images, audio, and embeds - Add asset management methods to DatabaseService (create, get, list, delete) - Add asset migration to MigrationService to import from S3 on first run - Refactor MediaService to read from local DB instead of scanning S3 - Add setDatabaseService() method to inject database dependency - Add trackAssetUsageFromDB() to track asset usage from local entries - Keep S3 scanning methods as fallback for compatibility - Wire up MediaService in main.ts to use DatabaseService Benefits: - Media library loads instantly from local DB (no S3 API calls) - Eliminates ListObjectsV2 and GetObject calls on every media library open - Preserves asset metadata (size, dimensions, upload date, etc.) - Maintains backward compatibility with S3 fallback The media library previously scanned ALL S3 objects (images, audio, entries) on every load. Now it reads from a local SQLite table, improving performance dramatically.

- Add clearDatabase() method to DatabaseService - Closes database connection - Deletes SQLite file and WAL/SHM files - Reinitializes empty database - Automatically triggers re-import from S3 - Add getDatabasePath() method to show file location - Add IPC handlers for db:clear and db:getPath - Wire up methods in preload.ts and electron.d.ts - Add "Danger Zone" section to DatabaseTestPanel - Shows database file path - Provides clear button with warning confirmation - Displays progress during clear & re-import - Auto-reloads app after completion Users can now: 1. Click button in Settings > Database POC tab 2. Manually delete file at displayed path 3. Use for troubleshooting sync issues or bucket switches

- Fixed assets table schema by removing invalid foreign key to non-existent buckets table - Fixed Vite config to use IPv4 (127.0.0.1) instead of IPv6 to avoid macOS firewall issues - Enhanced MediaService.trackAssetUsageFromDB with: - Duplicate prevention for usage tracking - Audio asset tracking support - Better filename matching with image variant support (thumbnail-, small-, etc.) - Fixed critical bug where listEntries was called with bucket ID instead of collection IDs - Added comprehensive logging for asset import and usage tracking - Media library now fully functional with local database, with S3 fallback

- Added SQLite triggers to auto-queue source file syncs on collection/entry changes - Triggers fire on INSERT, UPDATE, and DELETE operations - SyncService now processes source_file_sync operations - Migration v2 applies triggers to existing databases - Removed redundant manual sync queue calls (triggers handle automatically) - Publishing already filters for published content only (verified) How it works: 1. User creates/updates/deletes content 2. Database trigger fires automatically 3. Sync queue populated with source file operations 4. SyncService background process handles sync (POC logs for now) 5. Publish button only generates APIs for published content

Use the same fallback pattern (s3BucketName || bucketName || name) that's used elsewhere in the codebase to ensure bucket name is properly resolved.

- Added PublishingService.syncSourceFile() to upload individual collection/entry source files - Added PublishingService.deleteSourceFile() to delete source files from S3 - Updated SyncService to actually call these methods instead of just logging - Background sync now automatically backs up all content changes to S3 - Source files saved to: collections/{id}.json and entries/{id}.json Now when you create/edit content: 1. Database triggers queue the sync operation 2. SyncService picks it up every 10 seconds 3. File is actually uploaded to S3 ✅ 4. You'll see: [PublishingService] ✓ Synced collection source: collections/{id}.json

Includes: - Complete architecture overview with diagrams - Current state and what's working - Key files and services documentation - Known issues and workarounds - Next steps and TODO list - Debugging tips and development commands - Quick start guide for new sessions - Important technical context This document enables seamless continuation of development in new chat sessions.

When user switches active bucket, all services (especially MediaService) need to be reconfigured to use the new bucket's ID and configuration. Key fix: MediaService.setDatabaseService(databaseService, newBucketId) This ensures the media library loads assets from the correct bucket without requiring app restart. Also reconfigures: - S3Service with new credentials - PublishingService with new bucket config - SyncService with new bucket name - All directory paths Fixes issue where media library showed stale assets from previous bucket.

Migration UX Improvements: - Created MigrationModal component with loading/success/error states - Added migration event listeners in App.tsx - Migration now automatically triggers when switching to empty bucket - No manual app restart needed - migration happens seamlessly Technical Changes: - Extracted runMigrationIfNeeded() helper function - Added migration:started and migration:complete IPC events - Updated bucket switching handler to check for migration needs - Added migration event types to electron.d.ts - Exposed migration event listeners in preload.ts User Experience: 1. Switch to new bucket → Migration starts automatically 2. Modal shows "Importing Data" with spinner 3. On success: Shows checkmark, auto-closes after 2s 4. On error: Shows error message with close button 5. App remains responsive during migration Fixes issue where users had to restart app to trigger migration for new buckets.

Critical Bug Fix: - ContentManager was depending on 'bucket' prop (S3 bucket NAME) but loading with 'bucketId' (internal ID) - This caused race conditions where old bucket's data could show when switching buckets - Changed useEffect dependency from 'bucket' to 'bucketId' for correct behavior Additional Improvements: - Clear all state (collections, entries, selected collection) when bucket changes - Added comprehensive logging to track bucket_id throughout the stack - DatabaseService now logs total collections and warns if bucket_id mismatch detected - ContentManager logs when bucket changes and which bucketId it's loading This ensures complete bucket isolation - each bucket's data stays separate. Testing: 1. Switch to Bucket A → See only Bucket A collections 2. Switch to Bucket B → See only Bucket B collections 3. No mixing of data between buckets Fixes issue where collections from multiple buckets were appearing together.

Critical Fix for Bucket Isolation: - Added clearBucketData() method that runs BEFORE every migration - Deletes all collections, entries, and assets for the bucket before importing - Prevents old data from previous imports mixing with new data Migration Flow (NEW): 1. Check if migration needed 2. Clear ALL existing data for this bucket 3. Import fresh data from S3 with correct bucket_id 4. Verify each collection is tagged with correct bucket_id Enhanced Logging: - Migration start banner with bucket_id and S3 bucket name - Logs each collection being cleared and migrated - Logs successful migration per collection - Verifies bucket_id throughout the process Why This Was Needed: - Before: Migration would append/skip, leaving old data mixed in - After: Clean slate for each migration, perfect isolation Testing Steps: 1. Import Bucket A → Should see ONLY Bucket A data 2. Settings → Clear Database → Reimport → Should see same data, no duplication 3. Switch to Bucket B → Should see ONLY Bucket B data, no Bucket A data 4. Switch back to Bucket A → Should trigger reimport, see clean Bucket A data Watch terminal for migration logs to verify correct bucket_id.

Critical Fixes: 1. Pause/Resume Sync During Migration - Added pause() and resume() methods to SyncService - Migration now pauses sync at start, resumes on completion - Prevents database triggers from queuing operations during import - Sync resumes automatically after migration (success or error) 2. Enhanced Migration Logging - Logs bucket_id for every collection being migrated - Logs bucket_id for every entry being migrated - Clear visual separators (======) for migration start/end - Entry logging includes title and ID for debugging Why This Was Critical: - Before: Sync service would process database trigger events during migration - During migration: Collections/entries being created would trigger sync queue - Race condition: Sync could write partial data to S3 - Mixed data: Old data + partial new data = confusion - After: Migration completes cleanly, then sync processes final state Migration Flow (FINAL): 1. Pause sync service ⏸️ 2. Clear all bucket data 3. Import collections with bucket_id verification 4. Import entries with bucket_id verification 5. Import assets 6. Resume sync service ▶️ 7. Sync processes final clean state Testing: Watch terminal for: [SyncService] ⏸️ Pausing sync (migration in progress)... [MigrationService] Migrating collection: "Gallery" with bucket_id: abc-123 [MigrationService] - Entry "Photo 1" with bucket_id: abc-123 [SyncService] ▶️ Resuming sync (migration complete)... This ensures zero interference between migration and sync processes.

CRITICAL FIX for bucket mixing issue. Root Cause Analysis: Previous migrations were setting INCORRECT bucket_ids on collections/entries. When trying to reimport: 1. clearBucketData filtered by bucket_id 2. Found 0 collections (because they had wrong bucket_id) 3. Tried to import same collection IDs again 4. UNIQUE constraint failures 5. Result: Mixed data from multiple buckets Evidence from logs: - Total collections in database: 18 - Found 0 collections for bucket 36a251d1-d767-4879-96e5-d21aeb147d05 - UNIQUE constraint failed: entries.id (entries already exist!) Temporary Solution (NUCLEAR): clearBucketData() now calls clearDatabase() to wipe EVERYTHING. This ensures: - No orphaned data with wrong bucket_ids - Clean import every time - Correct bucket isolation going forward Trade-off: - Loses multi-bucket support temporarily - Each bucket switch triggers full DB clear + reimport - This is necessary to fix the data corruption Next Steps: 1. Test that imports work cleanly now 2. Verify bucket_ids are correct after import 3. Once stable, can optimize to per-bucket clear Testing: Delete local DB, restart app, verify clean import with correct bucket_id.

- Fix bucketId undefined bug in ContentManager * Add guard in handleCollectionDragEnd to prevent undefined bucketId * Remove unsafe non-null assertion operator - Fix bucket content mixing in sync service (CRITICAL) * Add bucketId filtering to DatabaseService.getSyncQueue() * Update SyncService to track and filter by bucketId * Prevents sync service from processing all buckets' queue items - Fix migration clearing all buckets' data * Add DatabaseService.clearBucketData() for bucket-specific clearing * Update MigrationService to use bucket-specific clearing * Preserves other buckets' data during migration - Fix database table name bug * Correct sync_queue -> data_sync_queue - Add debug tools * Add db.debug.getBucketIds() IPC handler for troubleshooting * Add corresponding preload and TypeScript definitions

- Fix transformUrlsToCloudFront to use config.cloudFrontUrl - Add support for S3 URLs with region in pattern matching - Published API JSON now uses CloudFront URLs for assets when configured - Transforms: bucket.s3.region.amazonaws.com -> CloudFront domain

- Add transformation to individual collection endpoints (slug.json) - Add transformation to schema lookup files (SchemaName.json) - Ensures primaryImage and all asset URLs use CloudFront in published API - Previously only collections/index.json was being transformed

- Add global polyfill to vitest.config.ts - Add Node.js globals polyfills to test setup - Mock URL, Buffer, and process for test environment compatibility - Fixes webidl-conversions and whatwg-url unhandled errors in CI

- Updated onClick handler for the Close button to safely call onComplete using optional chaining. - This change prevents potential runtime errors if onComplete is not provided.

SecretLaboratory added 24 commits October 19, 2025 02:45

yerrrp

e80ac52

fix: use correct bucket name property for SyncService configuration

b2c97b4

Use the same fallback pattern (s3BucketName || bucketName || name) that's used elsewhere in the codebase to ensure bucket name is properly resolved.

docs: update continuation prompt with bucket switching fix

e76154d

docs: update continuation prompt with migration system details

3939276