diff --git a/packages/migration-bundle/CHANGELOG.md b/packages/migration-bundle/CHANGELOG.md new file mode 100644 index 00000000..6c6cd817 --- /dev/null +++ b/packages/migration-bundle/CHANGELOG.md @@ -0,0 +1,123 @@ +# Changelog + +## 1.3.0 - 2026-03-02 + +> **Development Note:** This release was developed with assistance from Claude Code (Anthropic's AI coding assistant). + +### Features + +- **Tab-Based UI Redesign:** Complete redesign of the Migration Options interface with organized tabs for each category (Schema, Content, Files, Users, Flows, Extensions, Settings, Translations, Bookmarks, Insights). Addresses [Issue #294](https://github.com/directus-labs/extensions/issues/294). + +- **Users Granular Filtering:** Fine-grained control over user-related migrations: + - Roles filtering with `selectedRoles` + - Policies filtering with `selectedPolicies` + - Permissions filtering with `selectedPermissions` + - User accounts filtering with `selectedUsers` + - Access rules filtering with `selectedAccess` + - Dependency auto-selection (selecting a user auto-selects their role) + +- **Flows Granular Filtering:** Select specific flows to migrate using `selectedFlows`. Supports both flow ID (UUID) and flow name for flexible filtering. + +- **Extensions Granular Filtering:** Select specific extensions to migrate using `selectedExtensions`. Supports ID, schema name, and bundle name matching. + +- **Files/Folders Filtering:** New Files tab with folder-based filtering using `selectedFolders`. Migrate only files from selected folders. + +- **Settings Granular Filtering:** Select specific settings fields to migrate instead of all settings. + +- **Translations Granular Filtering:** Filter translations by: + - Selected languages with `selectedLanguages` + - Key pattern matching with `translationKeyPattern` + +- **Bookmarks Tab:** New dedicated tab for bookmark (preset) management with `selectedPresets` filtering. + +- **Insights Tab:** New dedicated tab for dashboard management with `selectedDashboards` filtering. + +- **Comments Integration:** Comments can now be migrated independently or tied to content collections using `includeCommentsForContent`. + +- **GUI Collection Context Filtering:** Bookmarks and Flows lists automatically filter based on selected schema collections, showing only relevant items. + +- **Local Extensions Warning:** UI notice when local extensions are detected, informing users they must be manually installed on the target. + +### Bug Fixes + +- **Tab Visibility Bug:** Fixed tabs permanently disappearing after deselecting all items. Root cause was Directus v-select returning `null` instead of `[]`. +- **selectedExtensions Filter:** Now accepts both extension ID and schema.name for matching. +- **Empty usersSelection Skip:** Migration now correctly skips empty user selection objects. +- **Null Collection Handling:** Fixed include/exclude modes to properly handle items with null collection references. +- **Comments Migration:** Moved comments migration outside content block to allow independent migration. + +### Environment Variables + +New environment variables for granular control: + +```bash +# Flows filtering (ID or name, comma-separated) +MIGRATION_BUNDLE_SELECTED_FLOWS=flow-uuid,My Flow Name + +# Extensions filtering +MIGRATION_BUNDLE_SELECTED_EXTENSIONS=@directus-labs/field-comments + +# Folders filtering +MIGRATION_BUNDLE_SELECTED_FOLDERS=folder-uuid-1,folder-uuid-2 + +# Users granular options +MIGRATION_BUNDLE_USERS_ROLES=true +MIGRATION_BUNDLE_USERS_POLICIES=true +MIGRATION_BUNDLE_USERS_PERMISSIONS=true +MIGRATION_BUNDLE_USERS_ACCOUNTS=true +MIGRATION_BUNDLE_USERS_ACCESS=true +``` + +--- + +## 1.2.0 - 2026-02-24 + +> **Development Note:** This release was developed with assistance from Claude Code (Anthropic's AI coding assistant). + +### Features + +- **Collection-Level Selection:** Users can now select specific collections to migrate instead of migrating all content at once. This feature addresses [Issue #266](https://github.com/directus-labs/extensions/issues/266) and [Issue #294](https://github.com/directus-labs/extensions/issues/294). + - Schema filtering: Only selected collections' schemas are migrated + - Content filtering: Only selected collections' data is migrated + - Parent group collections are automatically included when child collections are selected + +- **File Filtering by Collection:** Files are now filtered based on selected collections. Only files referenced by the selected collections are migrated, reducing migration payload size and improving performance. + +- **Content Collection Selector UI:** New multi-select interface in the Migration Options to choose which collections to migrate. + +- **Environment Variable Support:** + - `MIGRATION_BUNDLE_SELECTED_COLLECTIONS`: Comma-separated list of collections to include + - `MIGRATION_BUNDLE_EXCLUDED_COLLECTIONS`: Comma-separated list of collections to exclude + +### Improvements + +- **Schema Option Checkbox:** Added "Schema" as a selectable migration option (previously schema was always migrated) +- **Filtered Panels/Flows/Presets:** Dashboard panels, flows, and presets are now filtered based on their collection references +- **Debug Logging:** Added debug output during schema filtering for troubleshooting + +### Important Notes + +#### PostgreSQL Requirement for Target Instance + +Due to a known Directus bug ([GitHub Issue #20428](https://github.com/directus/directus/issues/20428)), schema migrations may fail on MySQL target instances. The issue occurs because Directus automatically converts `char(36)` to `varchar(36)` in schema diffs for PostgreSQL compatibility, which breaks MySQL foreign key constraints. + +**Recommendation:** Use PostgreSQL for the target (destination) Directus instance to ensure reliable schema migrations. + +| Target Database | Schema Apply | Status | +|-----------------|--------------|--------| +| PostgreSQL | Works correctly | ✅ Recommended | +| MySQL | May fail with FK constraint errors | ⚠️ Known issue | + +### Internal Changes + +- Refactored `extract-data.ts` to filter collections list with parent group resolution +- Added `filter-schema.ts` utility for schema snapshot filtering +- Added `filter-schema-diff.ts` utility for diff response filtering +- Modified `extract-system.ts` to filter panels, flows, presets, and comments by collection + +--- + +## 1.1.1 - Previous Release + +- Initial release with category-level migration options +- Support for migrating content, users, flows, dashboards, extensions, presets, and comments diff --git a/packages/migration-bundle/README.md b/packages/migration-bundle/README.md index 8baa7f95..33f6f3d7 100644 --- a/packages/migration-bundle/README.md +++ b/packages/migration-bundle/README.md @@ -108,6 +108,106 @@ This approach provides: ![Customise the module](https://raw.githubusercontent.com/directus-labs/extensions/main/packages/migration-bundle/docs/migration-module-customize.jpg) +## Granular Migration Options + +> **New in v1.3.0** - Developed with assistance from Claude Code (Anthropic's AI coding assistant). + +The migration module now features a **tab-based UI** with granular control over every aspect of your migration. Each category has its own dedicated tab with filtering options. + +### Available Tabs + +| Tab | Description | Filtering Options | +|-----|-------------|-------------------| +| **Schema** | Database schema | Include/exclude specific collections | +| **Content** | Collection data | Select collections for content migration | +| **Files** | Files and folders | Filter by specific folders | +| **Users** | User management | Roles, policies, permissions, accounts, access rules | +| **Flows** | Automation flows | Select specific flows by ID or name | +| **Extensions** | Installed extensions | Select specific extensions | +| **Settings** | Project settings | Select specific setting fields | +| **Translations** | Custom translations | Filter by language and key pattern | +| **Bookmarks** | User bookmarks | Select specific bookmarks | +| **Insights** | Dashboards | Select specific dashboards | + +### Users Granular Options + +Control exactly which user-related data to migrate: + +- **Roles** - User roles and their configurations +- **Policies** - Access policies +- **Permissions** - Collection permissions +- **User Accounts** - Actual user records +- **Access Rules** - Role-policy-user mappings + +Dependencies are automatically handled - selecting a user will auto-select their assigned role. + +### Flows and Extensions Filtering + +You can filter flows and extensions by: +- **ID (UUID)** - Exact match +- **Name** - Human-readable name matching + +```bash +# Environment variable examples +MIGRATION_BUNDLE_SELECTED_FLOWS=abc-123-uuid,My Automation Flow +MIGRATION_BUNDLE_SELECTED_EXTENSIONS=@directus-labs/field-comments,my-custom-extension +``` + +### Local Extensions Notice + +When local extensions are detected on the source instance, a warning notice appears informing you that these must be manually installed on the target instance before migration. + +--- + +## Collection-Level Selection + +> **New in v1.2.0** - Developed with assistance from Claude Code (Anthropic's AI coding assistant). + +You can now select specific collections to migrate instead of migrating all content at once. This feature is useful for: +- Large schemas where only specific collections need synchronization +- Reducing migration payload size +- Partial content migrations between instances + +### How to Use Collection Filtering + +1. Configure your migration as usual (destination URL and token) +2. Click **Check** to validate the connection +3. In Migration Options, check **Schema** and/or **Content** +4. A collection selector will appear - choose which collections to migrate +5. Start the migration + +**Note:** When you select child collections, their parent group collections are automatically included to maintain schema integrity. + +### File Filtering + +Files are automatically filtered based on your collection selection. Only files that are referenced by the selected collections will be migrated, significantly reducing migration time for partial migrations. + +### Environment Variables for Collection Filtering + +```bash +# Include only specific collections (comma-separated) +MIGRATION_BUNDLE_SELECTED_COLLECTIONS=articles,authors,categories + +# Exclude specific collections (used if SELECTED is not set) +MIGRATION_BUNDLE_EXCLUDED_COLLECTIONS=logs,temp_data,analytics +``` + +## Important: PostgreSQL Requirement for Target Instance + +Due to a known Directus bug ([GitHub Issue #20428](https://github.com/directus/directus/issues/20428)), schema migrations may fail on MySQL target instances. + +**Technical Details:** +- Directus automatically converts `char(36)` to `varchar(36)` in schema diffs for PostgreSQL compatibility +- This conversion triggers MySQL foreign key constraint errors: `Cannot change column 'X': used in a foreign key constraint` +- The issue affects any migration that includes schema changes + +**Recommendation:** Use **PostgreSQL** for the target (destination) Directus instance to ensure reliable schema migrations. + +| Target Database | Schema Migration | Recommendation | +|-----------------|------------------|----------------| +| PostgreSQL | ✅ Works correctly | **Recommended** | +| MySQL | ⚠️ May fail with FK errors | Use with caution | + ## Permissions -This extension requires admin privilages on both instances to ensure all data is accessible and writable. +This extension requires admin privileges on both instances to ensure all data is accessible and writable. diff --git a/packages/migration-bundle/package.json b/packages/migration-bundle/package.json index 2dd8e7a8..6ab31915 100644 --- a/packages/migration-bundle/package.json +++ b/packages/migration-bundle/package.json @@ -1,7 +1,7 @@ { "name": "@directus-labs/migration-bundle", "type": "module", - "version": "1.1.1", + "version": "1.3.0", "description": "Migrate from the current source Directus instance to another target Directus instance.", "license": "MIT", "repository": { diff --git a/packages/migration-bundle/src/migration-endpoint/composables/extract-data.ts b/packages/migration-bundle/src/migration-endpoint/composables/extract-data.ts index aaeeb868..5de9b124 100644 --- a/packages/migration-bundle/src/migration-endpoint/composables/extract-data.ts +++ b/packages/migration-bundle/src/migration-endpoint/composables/extract-data.ts @@ -34,33 +34,124 @@ async function extractContent({ try { res.write('* Fetching collections'); - const collections: Collection[] = await collectionService.readByQuery({ + // Load all collections first + const allCollections: Collection[] = await collectionService.readByQuery({ limit: -1, }); - res.write(' ...done\r\n\r\n'); + // Fix 8: Filter collections based on scope (same logic as fullData/singletons) + let collections = allCollections.filter(c => !c.collection.startsWith('directus_')); + + const hasContentFilter = scope.contentCollections && scope.contentCollections.length > 0; + const hasSelectedFilter = scope.selectedCollections && scope.selectedCollections.length > 0; + const hasExcludedFilter = scope.excludedCollections && scope.excludedCollections.length > 0; + + if (hasContentFilter || hasSelectedFilter) { + // Build initial set of included collection names + const selectedNames = hasContentFilter + ? scope.contentCollections! + : scope.selectedCollections!; + + const includedNames = new Set(selectedNames); + + // Recursively add parent groups (same logic as filter-schema.ts) + let changed = true; + let iterations = 0; + const maxIterations = 100; + + while (changed && iterations < maxIterations) { + changed = false; + iterations++; + for (const c of allCollections) { + if (includedNames.has(c.collection) && c.meta?.group) { + if (!includedNames.has(c.meta.group)) { + includedNames.add(c.meta.group); + changed = true; + } + } + } + } + + // Filter collections to only include selected + parent groups + collections = collections.filter(c => includedNames.has(c.collection)); + } + else if (hasExcludedFilter) { + collections = collections.filter(c => + !scope.excludedCollections!.includes(c.collection) + ); + } + + res.write(` (${collections.length} collections) ...done\r\n\r\n`); res.write('* Fetching fields'); const primaryKeyMap = await getCollectionPrimaryKeys(fieldService); res.write(' ...done\r\n\r\n'); res.write('* Fetching full data'); - const fullData: UserCollectionItems[] = scope.content ? await loadFullData(collections, ItemsService, primaryKeyMap, accountability, schema) : []; + const fullData: UserCollectionItems[] = scope.content ? await loadFullData(collections, ItemsService, primaryKeyMap, accountability, schema, scope) : []; res.write(' ...'); await saveToFile(fullData, 'items_full_data', fileService, folder, storage); res.write('done\r\n\r\n'); res.write('* Fetching singletons'); - const singletons: UserCollectionItem[] = scope.content ? await loadSingletons(collections, ItemsService, accountability, schema) : []; + const singletons: UserCollectionItem[] = scope.content ? await loadSingletons(collections, ItemsService, accountability, schema, scope) : []; res.write(' ...'); await saveToFile(singletons, 'items_singleton', fileService, folder, storage); res.write('done\r\n\r\n'); - res.write('* Fetching files'); - const files: File[] = scope.content ? await fileService.readByQuery({ fields: directusFileFields, filter: { _or: [{ folder: { _neq: folder } }, { folder: { _null: true } }] }, limit: -1 }) : []; - res.write(' ...'); - await saveToFile(files, 'files', fileService, folder, storage); - res.write('done\r\n\r\n'); + // Files: fetch if files is selected, or content is selected (backward compatibility) + // Issue #009: Skip if selectedFolders is an empty array (explicit skip) + const hasEmptyFolderSelection = Array.isArray(scope.selectedFolders) && scope.selectedFolders.length === 0; + const shouldFetchFiles = !hasEmptyFolderSelection && (scope.files === true || (scope.content && scope.files !== false)); + res.write(shouldFetchFiles ? '* Fetching files' : '* Skipping files\r\n\r\n'); + + let files: File[] = []; + if (shouldFetchFiles) { + // Fix 6.3: Filter files based on selectedCollections only (not contentCollections) + // contentCollections is for content-specific filtering, files should be independent + const contentCols = scope.selectedCollections?.length ? scope.selectedCollections : null; + + // Issue #009: Build folder filter if selectedFolders is specified + const folderFilter = scope.selectedFolders && scope.selectedFolders.length > 0 + ? { folder: { _in: scope.selectedFolders } } + : { _or: [{ folder: { _neq: folder } }, { folder: { _null: true } }] }; + + if (contentCols && contentCols.length > 0) { + // Get file IDs from selected collections + const fileIds = await getFileIdsFromCollections(contentCols, ItemsService, fieldService, accountability, schema); + res.write(` (${fileIds.size} files from ${contentCols.length} collections)`); + + if (fileIds.size > 0) { + files = await fileService.readByQuery({ + fields: directusFileFields, + filter: { + _and: [ + { id: { _in: Array.from(fileIds) } }, + folderFilter, + ], + }, + limit: -1, + }); + } + } + else { + // No collection filtering - get all files (with optional folder filter) + files = await fileService.readByQuery({ + fields: directusFileFields, + filter: folderFilter, + limit: -1, + }); + } + + // Issue #009: Log folder filtering if active + if (scope.selectedFolders && scope.selectedFolders.length > 0) { + res.write(` (folder filter: ${scope.selectedFolders.length} folders)`); + } + + res.write(' ...'); + await saveToFile(files, 'files', fileService, folder, storage); + res.write('done\r\n\r\n'); + } return { collections, @@ -120,11 +211,25 @@ async function extractSingleton(collection: string, ItemsService: any, accountab return await itemService.readSingleton({}); } -async function loadFullData(collections: Collection[], itemService: any, primaryKeyMap: any, accountability: Accountability, schema: SchemaOverview): Promise { +async function loadFullData(collections: Collection[], itemService: any, primaryKeyMap: any, accountability: Accountability, schema: SchemaOverview, scope: Scope): Promise { const userCollections = collections .filter((item) => !item.collection.startsWith('directus_', 0)) .filter((item) => item.schema !== null) - .filter((item) => !item.meta?.singleton); + .filter((item) => !item.meta?.singleton) + // Collection-level filtering for content + // If contentCollections is specified, use that; otherwise fall back to selectedCollections/excludedCollections + .filter((item) => { + if (scope.contentCollections && scope.contentCollections.length > 0) { + return scope.contentCollections.includes(item.collection); + } + if (scope.selectedCollections && scope.selectedCollections.length > 0) { + return scope.selectedCollections.includes(item.collection); + } + if (scope.excludedCollections && scope.excludedCollections.length > 0) { + return !scope.excludedCollections.includes(item.collection); + } + return true; + }); return await Promise.all(userCollections.map(async (collection) => { const name = collection.collection; @@ -137,10 +242,24 @@ async function loadFullData(collections: Collection[], itemService: any, primary })); } -async function loadSingletons(collections: Collection[], itemService: any, accountability: Accountability, schema: SchemaOverview): Promise { +async function loadSingletons(collections: Collection[], itemService: any, accountability: Accountability, schema: SchemaOverview, scope: Scope): Promise { const singletonCollections = collections .filter((item) => !item.collection.startsWith('directus_', 0)) - .filter((item) => item.meta?.singleton); + .filter((item) => item.meta?.singleton) + // Collection-level filtering for content + // If contentCollections is specified, use that; otherwise fall back to selectedCollections/excludedCollections + .filter((item) => { + if (scope.contentCollections && scope.contentCollections.length > 0) { + return scope.contentCollections.includes(item.collection); + } + if (scope.selectedCollections && scope.selectedCollections.length > 0) { + return scope.selectedCollections.includes(item.collection); + } + if (scope.excludedCollections && scope.excludedCollections.length > 0) { + return !scope.excludedCollections.includes(item.collection); + } + return true; + }); return await Promise.all(singletonCollections.map(async (collection) => { const name = collection.collection; @@ -174,4 +293,58 @@ function getPrimaryKey(collectionsMap: any, collection: string) { return collectionsMap[collection]; } +// Fix 6.3: Get file IDs from selected collections +async function getFileIdsFromCollections( + collections: string[], + ItemsService: any, + fieldService: any, + accountability: Accountability, + schema: SchemaOverview, +): Promise> { + const fileIds = new Set(); + + // Get all fields to find file-type fields + const allFields = await fieldService.readAll(); + if (!allFields) return fileIds; + + // Find file-type fields in selected collections + const fileFields: Array<{ collection: string; field: string }> = []; + for (const field of allFields) { + if (!collections.includes(field.collection)) continue; + + // Check for file field (uuid with file interface or special) + const isFileField = field.type === 'uuid' && + (field.meta?.interface === 'file' || + field.meta?.interface === 'file-image' || + field.meta?.special?.includes('file')); + + if (isFileField) { + fileFields.push({ collection: field.collection, field: field.field }); + } + } + + // Query each collection for file IDs + for (const { collection, field } of fileFields) { + try { + const itemService = new ItemsService(collection, { accountability, schema }); + const items = await itemService.readByQuery({ + fields: [field], + filter: { [field]: { _nnull: true } }, + limit: -1, + }); + + for (const item of items) { + if (item[field]) { + fileIds.add(item[field]); + } + } + } + catch (error) { + console.error(`Failed to get files from ${collection}.${field}:`, error); + } + } + + return fileIds; +} + export default extractContent; diff --git a/packages/migration-bundle/src/migration-endpoint/composables/extract-system.ts b/packages/migration-bundle/src/migration-endpoint/composables/extract-system.ts index 3b93974c..dd149f90 100644 --- a/packages/migration-bundle/src/migration-endpoint/composables/extract-system.ts +++ b/packages/migration-bundle/src/migration-endpoint/composables/extract-system.ts @@ -1,6 +1,25 @@ import type { Accountability, Permission, Preset, SchemaOverview, Settings, Share } from '@directus/types'; import type { Access, CommentRaw, DashboardRaw, DirectusError, Extension, Folder, ModifiedFlowRaw, OperationRaw, PanelRaw, PolicyRaw, RoleRaw, Scope, SystemExtract, Translation, UserRaw } from '../../types/extension'; import saveToFile from '../../utils/save-file'; + +function shouldIncludeCollection(collectionName: string, scope: Scope): boolean { + const { selectedCollections, excludedCollections } = scope; + + // If no filtering specified, include all + if ((!selectedCollections || selectedCollections.length === 0) && + (!excludedCollections || excludedCollections.length === 0)) { + return true; + } + + if (selectedCollections && selectedCollections.length > 0) { + return selectedCollections.includes(collectionName); + } + if (excludedCollections && excludedCollections.length > 0) { + return !excludedCollections.includes(collectionName); + } + return true; +} + import { directusDashboardFields, directusFlowFields, @@ -54,55 +73,141 @@ async function extractSystemData({ res, services, accountability, schema, scope, const shareService = new SharesService({ accountability, schema }); try { - res.write(scope.users ? '* Fetching roles' : '* Skipping roles\r\n\r\n'); - const roles: RoleRaw[] = scope.users ? await roleService.readByQuery({ fields: directusRoleFields, limit: -1 }) : []; + // Get users granular options with defaults + const usersGranular = scope.usersGranular || { + roles: true, + policies: true, + permissions: true, + userAccounts: true, + access: true, + }; - if (scope.users) { + // Phase 6: Get users selection options + const usersSelection = scope.usersSelection || {}; + + // Empty selection = skip (same pattern as selectedFolders, selectedPresets, etc.) + const hasEmptyRolesSelection = Array.isArray(usersSelection.selectedRoles) && usersSelection.selectedRoles.length === 0; + const hasEmptyPoliciesSelection = Array.isArray(usersSelection.selectedPolicies) && usersSelection.selectedPolicies.length === 0; + const hasEmptyPermissionsSelection = Array.isArray(usersSelection.selectedPermissions) && usersSelection.selectedPermissions.length === 0; + const hasEmptyUsersSelection = Array.isArray(usersSelection.selectedUsers) && usersSelection.selectedUsers.length === 0; + const hasEmptyAccessSelection = Array.isArray(usersSelection.selectedAccess) && usersSelection.selectedAccess.length === 0; + + const shouldFetchRoles = scope.users && usersGranular.roles && !hasEmptyRolesSelection; + const shouldFetchPolicies = scope.users && usersGranular.policies && !hasEmptyPoliciesSelection; + const shouldFetchPermissions = scope.users && usersGranular.permissions && !hasEmptyPermissionsSelection; + const shouldFetchUsers = scope.users && usersGranular.userAccounts && !hasEmptyUsersSelection; + const shouldFetchAccess = scope.users && usersGranular.access && !hasEmptyAccessSelection; + + res.write(shouldFetchRoles ? '* Fetching roles' : '* Skipping roles\r\n\r\n'); + let roles: RoleRaw[] = shouldFetchRoles ? await roleService.readByQuery({ fields: directusRoleFields, limit: -1 }) : []; + + // Phase 6: Filter roles by selection + if (shouldFetchRoles && usersSelection.selectedRoles && usersSelection.selectedRoles.length > 0) { + roles = roles.filter(r => usersSelection.selectedRoles!.includes(r.id)); + res.write(` (filtered to ${roles.length} selected roles)\r\n`); + } + + if (shouldFetchRoles) { res.write(' ...'); await saveToFile(roles, 'roles', fileService, folder, storage); res.write('done\r\n\r\n'); } - res.write(scope.users ? '* Fetching policies' : '* Skipping policies\r\n\r\n'); - const policies: PolicyRaw[] = scope.users ? await policyService.readByQuery({ fields: directusPolicyFields, limit: -1 }) : []; + res.write(shouldFetchPolicies ? '* Fetching policies' : '* Skipping policies\r\n\r\n'); + let policies: PolicyRaw[] = shouldFetchPolicies ? await policyService.readByQuery({ fields: directusPolicyFields, limit: -1 }) : []; - if (scope.users) { + // Phase 6: Filter policies by selection + if (shouldFetchPolicies && usersSelection.selectedPolicies && usersSelection.selectedPolicies.length > 0) { + policies = policies.filter(p => usersSelection.selectedPolicies!.includes(p.id)); + res.write(` (filtered to ${policies.length} selected policies)\r\n`); + } + + if (shouldFetchPolicies) { res.write(' ...'); await saveToFile(policies, 'policies', fileService, folder, storage); res.write('done\r\n\r\n'); } - res.write(scope.users ? '* Fetching permissions' : '* Skipping permissions\r\n\r\n'); - const permissions: Permission[] = scope.users ? await permissionService.readByQuery({ limit: -1 }) : []; + res.write(shouldFetchPermissions ? '* Fetching permissions' : '* Skipping permissions\r\n\r\n'); + let permissions: Permission[] = shouldFetchPermissions ? await permissionService.readByQuery({ limit: -1 }) : []; - if (scope.users) { + // Phase 6: Filter permissions by selection + if (shouldFetchPermissions && usersSelection.selectedPermissions && usersSelection.selectedPermissions.length > 0) { + permissions = permissions.filter(p => usersSelection.selectedPermissions!.includes(p.id)); + res.write(` (filtered to ${permissions.length} selected permissions)\r\n`); + } + + if (shouldFetchPermissions) { res.write(' ...'); await saveToFile(permissions, 'permissions', fileService, folder, storage); res.write('done\r\n\r\n'); } - res.write(scope.users ? '* Fetching users' : '* Skipping users\r\n\r\n'); - const users: UserRaw[] = scope.users ? await userService.readByQuery({ fields: directusUserFields, limit: -1 }) : []; + res.write(shouldFetchUsers ? '* Fetching users' : '* Skipping users\r\n\r\n'); + let users: UserRaw[] = shouldFetchUsers ? await userService.readByQuery({ fields: directusUserFields, limit: -1 }) : []; - if (scope.users) { + // Phase 6: Filter users by selection + if (shouldFetchUsers && usersSelection.selectedUsers && usersSelection.selectedUsers.length > 0) { + users = users.filter(u => usersSelection.selectedUsers!.includes(u.id)); + res.write(` (filtered to ${users.length} selected users)\r\n`); + } + + if (shouldFetchUsers) { res.write(' ...'); await saveToFile(users, 'users', fileService, folder, storage); res.write('done\r\n\r\n'); } - res.write(scope.users ? '* Fetching access' : '* Skipping access\r\n\r\n'); - const access: Access[] = scope.users ? await accessService.readByQuery({ limit: -1 }) : []; + res.write(shouldFetchAccess ? '* Fetching access' : '* Skipping access\r\n\r\n'); + let access: Access[] = shouldFetchAccess ? await accessService.readByQuery({ limit: -1 }) : []; - if (scope.users) { + // Phase 6: Filter access by selection + if (shouldFetchAccess && usersSelection.selectedAccess && usersSelection.selectedAccess.length > 0) { + access = access.filter(a => usersSelection.selectedAccess!.includes(Number(a.id))); + res.write(` (filtered to ${access.length} selected access rules)\r\n`); + } + + if (shouldFetchAccess) { res.write(' ...'); await saveToFile(access, 'access', fileService, folder, storage); res.write('done\r\n\r\n'); } - res.write(scope.content ? '* Fetching folders' : '* Skipping folders\r\n\r\n'); - const folders: Folder[] = scope.content ? await folderService.readByQuery({ fields: directusFolderFields, filter: { id: { _neq: folder } }, limit: -1 }) : []; + // Folders: fetch only if explicitly enabled OR if files is enabled (folders needed for file organization) + // Removed scope.content check to prevent unwanted folder migration when only content is selected + // Issue #009: Skip if selectedFolders is an empty array (explicit skip) + const hasEmptyFolderSelection = Array.isArray(scope.selectedFolders) && scope.selectedFolders.length === 0; + const shouldFetchFolders = !hasEmptyFolderSelection && (scope.folders === true || scope.files === true); + res.write(shouldFetchFolders ? '* Fetching folders' : '* Skipping folders\r\n\r\n'); + let folders: Folder[] = shouldFetchFolders ? await folderService.readByQuery({ fields: directusFolderFields, filter: { id: { _neq: folder } }, limit: -1 }) : []; + + // Issue #009: Filter folders by selectedFolders if specified + if (shouldFetchFolders && scope.selectedFolders && scope.selectedFolders.length > 0 && folders.length > 0) { + const beforeCount = folders.length; + // Include selected folders and their parent folders (to maintain structure) + const selectedIds = new Set(scope.selectedFolders); + const includedIds = new Set(); + + // Helper to include folder and all its parents + const includeWithParents = (folderId: string) => { + includedIds.add(folderId); + const folder = folders.find(f => f.id === folderId); + if (folder?.parent) { + includeWithParents(folder.parent); + } + }; + + // Include all selected folders with their parents + scope.selectedFolders.forEach(id => { + const folder = folders.find(f => f.id === id); + if (folder) includeWithParents(id); + }); + + folders = folders.filter(f => includedIds.has(f.id)); + res.write(` (filtered: ${folders.length}/${beforeCount})`); + } - if (scope.content) { + if (shouldFetchFolders) { res.write(' ...'); await saveToFile(folders, 'folders', fileService, folder, storage); res.write('done\r\n\r\n'); @@ -118,7 +223,24 @@ async function extractSystemData({ res, services, accountability, schema, scope, } res.write(scope.dashboards ? '* Fetching panels' : '* Skipping panels\r\n\r\n'); - const panels: PanelRaw[] = scope.dashboards ? await panelService.readByQuery({ fields: directusPanelFields, limit: -1 }) : []; + let panels: PanelRaw[] = scope.dashboards ? await panelService.readByQuery({ fields: directusPanelFields, limit: -1 }) : []; + + // Filter panels by options.collection if collection filtering is active + if (scope.dashboards && panels.length > 0) { + panels = panels.filter(panel => { + const panelCollection = panel.options?.collection; + // Null collection handling based on filter mode + if (!panelCollection) { + // Include mode: null does not match any selected collection + if (scope.selectedCollections && scope.selectedCollections.length > 0) { + return false; + } + // Exclude mode or no filter: null is not excluded + return true; + } + return shouldIncludeCollection(panelCollection, scope); + }); + } if (scope.dashboards) { res.write(' ...'); @@ -127,7 +249,34 @@ async function extractSystemData({ res, services, accountability, schema, scope, } res.write(scope.flows ? '* Fetching flows' : '* Skipping flows\r\n\r\n'); - const flows: ModifiedFlowRaw[] = scope.flows ? await flowService.readByQuery({ fields: directusFlowFields, limit: -1 }) : []; + let flows: ModifiedFlowRaw[] = scope.flows ? await flowService.readByQuery({ fields: directusFlowFields, limit: -1 }) : []; + + // Filter flows by selectedFlows if specified + // Accepts both flow ID and flow name for flexible filtering (similar to selectedExtensions) + if (scope.flows && scope.selectedFlows && scope.selectedFlows.length > 0) { + flows = flows.filter(f => + scope.selectedFlows!.includes(f.id) || + scope.selectedFlows!.includes(f.name), + ); + res.write(` (filtered to ${flows.length} selected flows)\r\n`); + } + + // Also filter flows by options.collection if collection filtering is active + if (scope.flows && flows.length > 0) { + flows = flows.filter(flow => { + const flowCollection = flow.options?.collection; + // Null collection handling based on filter mode + if (!flowCollection) { + // Include mode: null does not match any selected collection + if (scope.selectedCollections && scope.selectedCollections.length > 0) { + return false; + } + // Exclude mode or no filter: null is not excluded + return true; + } + return shouldIncludeCollection(flowCollection, scope); + }); + } if (scope.flows) { res.write(' ...'); @@ -136,7 +285,13 @@ async function extractSystemData({ res, services, accountability, schema, scope, } res.write(scope.flows ? '* Fetching operations' : '* Skipping operations\r\n\r\n'); - const operations: OperationRaw[] = scope.flows ? await operationService.readByQuery({ fields: directusOperationFields, limit: -1 }) : []; + let operations: OperationRaw[] = scope.flows ? await operationService.readByQuery({ fields: directusOperationFields, limit: -1 }) : []; + + // Filter operations: only include operations belonging to included flows + if (scope.flows && flows.length > 0) { + const includedFlowIds = new Set(flows.map(f => f.id)); + operations = operations.filter(op => includedFlowIds.has(op.flow)); + } if (scope.flows) { res.write(' ...'); @@ -144,20 +299,50 @@ async function extractSystemData({ res, services, accountability, schema, scope, res.write('done\r\n\r\n'); } - res.write('* Fetching settings'); - const settings: Settings = await settingsService.readSingleton({ fields: directusSettingsFields }); - res.write(' ...'); - await saveToFile(settings, 'settings', fileService, folder, storage); - res.write('done\r\n\r\n'); + // Settings: fetch only if scope.settings is not explicitly false (default true for backward compatibility) + const shouldFetchSettings = scope.settings !== false; + res.write(shouldFetchSettings ? '* Fetching settings' : '* Skipping settings\r\n\r\n'); + const settings: Settings | null = shouldFetchSettings + ? await settingsService.readSingleton({ fields: directusSettingsFields }) + : null; - res.write('* Fetching translations'); - const translations: Translation[] = await translationService.readByQuery({ limit: -1 }); - res.write(' ...'); - await saveToFile(translations, 'translations', fileService, folder, storage); - res.write('done\r\n\r\n'); + if (shouldFetchSettings && settings) { + res.write(' ...'); + await saveToFile(settings, 'settings', fileService, folder, storage); + res.write('done\r\n\r\n'); + } + + // Translations: fetch only if scope.translations is not explicitly false (default true for backward compatibility) + const shouldFetchTranslations = scope.translations !== false; + res.write(shouldFetchTranslations ? '* Fetching translations' : '* Skipping translations\r\n\r\n'); + const translations: Translation[] = shouldFetchTranslations + ? await translationService.readByQuery({ limit: -1 }) + : []; + + if (shouldFetchTranslations) { + res.write(' ...'); + await saveToFile(translations, 'translations', fileService, folder, storage); + res.write('done\r\n\r\n'); + } res.write(scope.presets ? '* Fetching presets' : '* Skipping presets'); - const presets: Preset[] = scope.presets ? await presetService.readByQuery({ fields: directusPresetFields, limit: -1 }) : []; + let presets: Preset[] = scope.presets ? await presetService.readByQuery({ fields: directusPresetFields, limit: -1 }) : []; + + // Filter presets by collection if collection filtering is active + if (scope.presets && presets.length > 0) { + presets = presets.filter(preset => { + // Null collection handling based on filter mode + if (!preset.collection) { + // Include mode: null does not match any selected collection + if (scope.selectedCollections && scope.selectedCollections.length > 0) { + return false; + } + // Exclude mode or no filter: null is not excluded + return true; + } + return shouldIncludeCollection(preset.collection, scope); + }); + } if (scope.presets) { res.write(' ...'); @@ -166,7 +351,18 @@ async function extractSystemData({ res, services, accountability, schema, scope, } res.write(scope.extensions ? '* Fetching extensions' : '* Skipping extensions\r\n\r\n'); - const extensions: Extension[] = scope.extensions ? await extensionService.readAll() : []; + let extensions: Extension[] = scope.extensions ? await extensionService.readAll() : []; + + // Filter extensions by selectedExtensions if specified + // Accepts extension ID, schema.name, or bundle name for flexible filtering + if (scope.extensions && scope.selectedExtensions && scope.selectedExtensions.length > 0) { + extensions = extensions.filter(e => + scope.selectedExtensions!.includes(e.id) || + (e.schema?.name && scope.selectedExtensions!.includes(e.schema.name)) || + (e.bundle && scope.selectedExtensions!.includes(e.bundle)), + ); + res.write(` (filtered to ${extensions.length} selected extensions)\r\n`); + } if (scope.extensions) { res.write(' ...'); @@ -175,7 +371,16 @@ async function extractSystemData({ res, services, accountability, schema, scope, } res.write(scope.comments ? '* Fetching comments' : '* Skipping comments'); - const comments: CommentRaw[] = scope.comments ? await commentService.readByQuery({ limit: -1 }) : []; + let comments: CommentRaw[] = scope.comments ? await commentService.readByQuery({ limit: -1 }) : []; + + // Filter comments by collection if includeCommentsForContent is enabled (Issue #009) + // When true: only comments for selected collections are migrated + // When false or undefined: all comments are migrated (backward compatibility) + if (scope.comments && scope.includeCommentsForContent && comments.length > 0) { + const beforeCount = comments.length; + comments = comments.filter(comment => shouldIncludeCollection(comment.collection, scope)); + res.write(` (filtered: ${comments.length}/${beforeCount})`); + } if (scope.comments) { res.write(' ...'); diff --git a/packages/migration-bundle/src/migration-endpoint/composables/migrate-dashboards.ts b/packages/migration-bundle/src/migration-endpoint/composables/migrate-dashboards.ts index 6bfafe67..a673eccc 100644 --- a/packages/migration-bundle/src/migration-endpoint/composables/migrate-dashboards.ts +++ b/packages/migration-bundle/src/migration-endpoint/composables/migrate-dashboards.ts @@ -3,17 +3,44 @@ import type { DashboardRaw, DirectusError } from '../../types/extension'; import type { Schema } from '../api'; import { createDashboard, readDashboards } from '@directus/sdk'; -async function migrateDashboards({ res, client, dashboards, dry_run = false }: { res: any; client: RestClient; dashboards: DashboardRaw[] | null; dry_run: boolean }): Promise<{ response: string; name: string } | DirectusError> { +async function migrateDashboards({ + res, + client, + dashboards, + selectedDashboards, + dry_run = false, +}: { + res: any; + client: RestClient; + dashboards: DashboardRaw[] | null; + selectedDashboards?: string[]; + dry_run: boolean; +}): Promise<{ response: string; name: string } | DirectusError> { if (!dashboards) { res.write('* Couldn\'t read data from extract\r\n\r\n'); return { name: 'Directus Error', status: 404, errors: [{ message: 'No dashboards found' }] }; } - else if (dashboards.length === 0) { + + // Issue #014: Handle empty selection = skip + if (selectedDashboards && selectedDashboards.length === 0) { + res.write('* Dashboards skipped (empty selection)\r\n\r\n'); + return { response: 'Skipped', name: 'Insights' }; + } + + // Issue #014: Filter by selected dashboard IDs + let filteredBySelection = dashboards; + if (selectedDashboards && selectedDashboards.length > 0) { + const selectedIds = new Set(selectedDashboards); + filteredBySelection = dashboards.filter((dashboard) => selectedIds.has(dashboard.id)); + res.write(`* Filtering to ${filteredBySelection.length} selected dashboards (from ${dashboards.length})\r\n\r\n`); + } + + if (filteredBySelection.length === 0) { res.write('* No dashboards to migrate\r\n\r\n'); return { response: 'Empty', name: 'Insights' }; } - res.write(`* [Local] Found ${dashboards.length} dashboards\r\n\r\n`); + res.write(`* [Local] Found ${filteredBySelection.length} dashboards\r\n\r\n`); try { // Fetch existing dashboards @@ -29,7 +56,7 @@ async function migrateDashboards({ res, client, dashboards, dry_run = false }: { const existingDashboardIds = new Set(existingDashboards.map((dashboard) => dashboard.id)); - const filteredDashboards = dashboards.filter((dashboard) => { + const dashboardsToCreate = filteredBySelection.filter((dashboard) => { if (existingDashboardIds.has(dashboard.id)) { return false; } @@ -41,13 +68,15 @@ async function migrateDashboards({ res, client, dashboards, dry_run = false }: { return newDash; }); - res.write(filteredDashboards.length > 0 ? `* [Remote] Uploading ${filteredDashboards.length} ${filteredDashboards.length > 1 ? 'Dashboards' : 'Dashboard'} ` : '* No Dashboards to migrate\r\n\r\n'); + res.write(dashboardsToCreate.length > 0 ? `* [Remote] Uploading ${dashboardsToCreate.length} ${dashboardsToCreate.length > 1 ? 'Dashboards' : 'Dashboard'} ` : '* No Dashboards to migrate\r\n\r\n'); - if (filteredDashboards.length > 0) { - await Promise.all(filteredDashboards.map(async (dashboard) => { + if (dashboardsToCreate.length > 0) { + for (const dashboard of dashboardsToCreate) { res.write('.'); - await client.request(createDashboard(dashboard)); - })); + if (!dry_run) { + await client.request(createDashboard(dashboard)); + } + } res.write(dry_run ? 'skipped\r\n\r\n' : 'done\r\n\r\n'); res.write('* Dashboard Migration Complete\r\n\r\n'); diff --git a/packages/migration-bundle/src/migration-endpoint/composables/migrate-folders.ts b/packages/migration-bundle/src/migration-endpoint/composables/migrate-folders.ts index f2077ce4..eb040f9c 100644 --- a/packages/migration-bundle/src/migration-endpoint/composables/migrate-folders.ts +++ b/packages/migration-bundle/src/migration-endpoint/composables/migrate-folders.ts @@ -29,13 +29,23 @@ async function migrateFolders({ res, client, folders, dry_run = false }: { res: const existingFolderIds = new Set(existingFolders.map((folder) => folder.id)); - const foldersToAdd = folders.filter((folder) => { + // Fix 6.5: Track skipped folders for better logging + const foldersToAdd: Folder[] = []; + const skippedFolders: Folder[] = []; + + for (const folder of folders) { if (existingFolderIds.has(folder.id)) { - return false; + skippedFolders.push(folder); + } + else { + foldersToAdd.push(folder); } + } - return true; - }); + // Log skipped folders + if (skippedFolders.length > 0) { + res.write(`* [Remote] Skipping ${skippedFolders.length} existing folder(s): ${skippedFolders.map(f => f.name).join(', ')}\r\n\r\n`); + } res.write(foldersToAdd.length > 0 ? `* [Remote] Uploading ${foldersToAdd.length} ${foldersToAdd.length > 1 ? 'Folders' : 'Folder'} ` : '* No Folders to migrate\r\n\r\n'); diff --git a/packages/migration-bundle/src/migration-endpoint/composables/migrate-panels.ts b/packages/migration-bundle/src/migration-endpoint/composables/migrate-panels.ts index a899e690..adb232ff 100644 --- a/packages/migration-bundle/src/migration-endpoint/composables/migrate-panels.ts +++ b/packages/migration-bundle/src/migration-endpoint/composables/migrate-panels.ts @@ -3,17 +3,44 @@ import type { DirectusError, PanelRaw } from '../../types/extension'; import type { Schema } from '../api'; import { createPanel, readPanels } from '@directus/sdk'; -async function migratePanels({ res, client, panels, dry_run = false }: { res: any; client: RestClient; panels: PanelRaw[] | null; dry_run: boolean }): Promise<{ response: string; name: string } | DirectusError> { +async function migratePanels({ + res, + client, + panels, + selectedDashboards, + dry_run = false, +}: { + res: any; + client: RestClient; + panels: PanelRaw[] | null; + selectedDashboards?: string[]; + dry_run: boolean; +}): Promise<{ response: string; name: string } | DirectusError> { if (!panels) { res.write('* Couldn\'t read data from extract\r\n\r\n'); return { name: 'Directus Error', status: 404, errors: [{ message: 'No panels found' }] }; } - else if (panels.length === 0) { + + // Issue #014: Skip panels if dashboards were skipped (empty selection) + if (selectedDashboards && selectedDashboards.length === 0) { + res.write('* Panels skipped (dashboards empty selection)\r\n\r\n'); + return { response: 'Skipped', name: 'Panels' }; + } + + // Issue #014: Filter panels by selected dashboards + let filteredByDashboard = panels; + if (selectedDashboards && selectedDashboards.length > 0) { + const selectedDashboardIds = new Set(selectedDashboards); + filteredByDashboard = panels.filter((panel) => selectedDashboardIds.has(panel.dashboard)); + res.write(`* Filtering to ${filteredByDashboard.length} panels for selected dashboards (from ${panels.length})\r\n\r\n`); + } + + if (filteredByDashboard.length === 0) { res.write('* No panels to migrate\r\n\r\n'); return { response: 'Empty', name: 'Panels' }; } - res.write(`* [Local] Found ${panels.length} panels\r\n\r\n`); + res.write(`* [Local] Found ${filteredByDashboard.length} panels\r\n\r\n`); try { // Fetch existing panels @@ -29,7 +56,7 @@ async function migratePanels({ res, client, panels, dry_run = false }: { res: an const existingPanelIds = new Set(existingPanels.map((panel) => panel.id)); - const filteredPanels = panels.filter((panel) => { + const panelsToCreate = filteredByDashboard.filter((panel) => { if (existingPanelIds.has(panel.id)) { return false; } @@ -37,13 +64,15 @@ async function migratePanels({ res, client, panels, dry_run = false }: { res: an return true; }); - res.write(filteredPanels.length > 0 ? `* [Remote] Uploading ${filteredPanels.length} ${filteredPanels.length > 1 ? 'Panels' : 'Panel'} ` : '* No Panels to migrate\r\n\r\n'); + res.write(panelsToCreate.length > 0 ? `* [Remote] Uploading ${panelsToCreate.length} ${panelsToCreate.length > 1 ? 'Panels' : 'Panel'} ` : '* No Panels to migrate\r\n\r\n'); - if (filteredPanels.length > 0) { - await Promise.all(filteredPanels.map(async (panel) => { + if (panelsToCreate.length > 0) { + for (const panel of panelsToCreate) { res.write('.'); - await client.request(createPanel(panel)); - })); + if (!dry_run) { + await client.request(createPanel(panel)); + } + } res.write(dry_run ? 'skipped\r\n\r\n' : 'done\r\n\r\n'); res.write('* Panel Migration Complete\r\n\r\n'); diff --git a/packages/migration-bundle/src/migration-endpoint/composables/migrate-presets.ts b/packages/migration-bundle/src/migration-endpoint/composables/migrate-presets.ts index 436c6ce3..071b75e4 100644 --- a/packages/migration-bundle/src/migration-endpoint/composables/migrate-presets.ts +++ b/packages/migration-bundle/src/migration-endpoint/composables/migrate-presets.ts @@ -4,26 +4,55 @@ import type { DirectusError } from '../../types/extension'; import type { Schema } from '../api'; import { createPresets } from '@directus/sdk'; -async function migratePresets({ res, client, presets, dry_run = false }: { res: any; client: RestClient; presets: Preset[] | null; dry_run: boolean }): Promise<{ response: string; name: string } | DirectusError> { +async function migratePresets({ + res, + client, + presets, + selectedPresets, + dry_run = false, +}: { + res: any; + client: RestClient; + presets: Preset[] | null; + selectedPresets?: string[]; + dry_run: boolean; +}): Promise<{ response: string; name: string } | DirectusError> { if (!presets) { res.write('* Couldn\'t read data from extract\r\n\r\n'); return { name: 'Directus Error', status: 404, errors: [{ message: 'No presets found' }] }; } - else if (presets.length === 0) { + + // Issue #014: Handle empty selection = skip + if (selectedPresets && selectedPresets.length === 0) { + res.write('* Presets skipped (empty selection)\r\n\r\n'); + return { response: 'Skipped', name: 'Presets' }; + } + + // Issue #014: Filter by selected preset IDs + let filteredPresets = presets; + if (selectedPresets && selectedPresets.length > 0) { + // Convert IDs to strings for comparison (presets use numeric IDs) + const selectedIds = new Set(selectedPresets.map((id) => String(id))); + filteredPresets = presets.filter((preset) => selectedIds.has(String(preset.id))); + res.write(`* Filtering to ${filteredPresets.length} selected presets (from ${presets.length})\r\n\r\n`); + } + + if (filteredPresets.length === 0) { res.write('* No presets to migrate\r\n\r\n'); return { response: 'Empty', name: 'Presets' }; } - res.write(`* [Local] Found ${presets.length} presets\r\n\r\n`); + res.write(`* [Local] Found ${filteredPresets.length} presets\r\n\r\n`); try { - res.write(presets.length > 0 ? `* [Remote] Uploading ${presets.length} ${presets.length > 1 ? 'Presets' : 'Preset'} ` : '* No Presets to migrate\r\n\r\n'); + res.write(filteredPresets.length > 0 ? `* [Remote] Uploading ${filteredPresets.length} ${filteredPresets.length > 1 ? 'Presets' : 'Preset'} ` : '* No Presets to migrate\r\n\r\n'); - const presetsToAdd = presets.map((preset) => { + const presetsToAdd = filteredPresets.map((preset) => { // Remove the id field from the presets so we don't have to reset the autoincrement on the db delete preset.id; const cleanPreset = { ...preset }; cleanPreset.user = null; + cleanPreset.role = null; // Null out role to prevent FK error when migrating without users return cleanPreset; }); diff --git a/packages/migration-bundle/src/migration-endpoint/composables/migrate-schema.ts b/packages/migration-bundle/src/migration-endpoint/composables/migrate-schema.ts index 7c6c96fc..f7986339 100644 --- a/packages/migration-bundle/src/migration-endpoint/composables/migrate-schema.ts +++ b/packages/migration-bundle/src/migration-endpoint/composables/migrate-schema.ts @@ -2,12 +2,21 @@ import type { RestClient, SchemaDiffOutput, SchemaSnapshotOutput } from '@direct import type { DirectusError } from '../../types/extension'; import type { Schema } from '../api'; import { schemaApply, schemaDiff } from '@directus/sdk'; +import { filterSchemaDiff, getCollectionsFromSchema } from '../../utils/filter-schema-diff'; export async function migrateSchema({ res, client, schema, dry_run = false, force = false }: { res: any; client: RestClient; schema: SchemaSnapshotOutput; dry_run: boolean; force: boolean }): Promise { try { + // Get collections from the filtered schema for diff filtering + const schemaCollections = getCollectionsFromSchema(schema); + + // Debug: Log schema being sent (using res.write so it appears in output) + res.write(`[DEBUG] Sending schema with ${schemaCollections.size} collections: ${Array.from(schemaCollections).join(', ')}\r\n`); + res.write(`[DEBUG] Schema fields count: ${(schema as any).fields?.length || 0}\r\n`); + res.write(`[DEBUG] Schema relations count: ${(schema as any).relations?.length || 0}\r\n\r\n`); + res.write('1. Comparing Schemas ...'); - const diff: SchemaDiffOutput = force + const rawDiff: SchemaDiffOutput = force ? await client.request(() => ({ body: JSON.stringify(schema), method: 'POST', @@ -15,27 +24,55 @@ export async function migrateSchema({ res, client, schema, dry_run = false, forc })) : await client.request(schemaDiff(schema)); + // Filter the diff to exclude directus_* collections + const diff = filterSchemaDiff(rawDiff, schemaCollections); + + // Debug: Log diff info BEFORE filtering + if ('diff' in rawDiff && rawDiff.diff) { + const rawCollections = rawDiff.diff.collections?.map((c: any) => c.collection) || []; + res.write(`[DEBUG] RAW diff collections (${rawCollections.length}): ${rawCollections.join(', ')}\r\n`); + } + + // Debug: Log diff info AFTER filtering + if ('diff' in diff && diff.diff) { + const filteredCollections = diff.diff.collections?.map((c: any) => c.collection) || []; + const diffFields = diff.diff.fields?.length || 0; + const diffRelations = diff.diff.relations?.length || 0; + res.write(`[DEBUG] FILTERED diff collections (${filteredCollections.length}): ${filteredCollections.join(', ')}\r\n`); + res.write(`[DEBUG] FILTERED diff: ${filteredCollections.length} collections, ${diffFields} fields, ${diffRelations} relations\r\n`); + } + if (!('hash' in diff)) { res.write('match\r\n\r\n'); } else { - res.write('done\r\n\r\n'); + // Check if filtered diff has any changes + const hasChanges = (diff.diff?.collections?.length || 0) > 0 || + (diff.diff?.fields?.length || 0) > 0 || + (diff.diff?.relations?.length || 0) > 0; - res.write('2. Applying Schemas ...'); - - if (!dry_run) { - await (!force - ? client.request(schemaApply(diff)) - : client.request(() => ({ - body: JSON.stringify(diff), - method: 'POST', - path: '/schema/apply?force=true', - }))); - - res.write('done\r\n\r\n'); + if (!hasChanges) { + res.write('match (after filtering)\r\n\r\n'); } else { - res.write('skipped\r\n\r\n'); + res.write('done\r\n\r\n'); + + res.write('2. Applying Schemas ...'); + + if (!dry_run) { + await (!force + ? client.request(schemaApply(diff)) + : client.request(() => ({ + body: JSON.stringify(diff), + method: 'POST', + path: '/schema/apply?force=true', + }))); + + res.write('done\r\n\r\n'); + } + else { + res.write('skipped\r\n\r\n'); + } } } diff --git a/packages/migration-bundle/src/migration-endpoint/composables/migrate-settings.ts b/packages/migration-bundle/src/migration-endpoint/composables/migrate-settings.ts index 91516854..4a3fc184 100644 --- a/packages/migration-bundle/src/migration-endpoint/composables/migrate-settings.ts +++ b/packages/migration-bundle/src/migration-endpoint/composables/migrate-settings.ts @@ -49,12 +49,44 @@ function mergeJsonStrings(current: string, incoming: string): string { } } -async function migrateSettings({ res, client, settings, dry_run = false }: { res: any; client: RestClient; settings: Settings | null; dry_run: boolean }): Promise<{ response: string; name: string } | DirectusError> { +async function migrateSettings({ + res, + client, + settings, + selectedSettings, + dry_run = false, +}: { + res: any; + client: RestClient; + settings: Settings | null; + selectedSettings?: string[]; + dry_run: boolean; +}): Promise<{ response: string; name: string } | DirectusError> { res.write('* [Local] Loading Settings\r\n\r\n'); + // Issue #013: Handle empty selection = skip migration + if (selectedSettings && selectedSettings.length === 0) { + res.write('* Settings skipped (empty selection)\r\n\r\n'); + return { response: 'Skipped', name: 'Settings' }; + } + try { const currentSettings = await client.request(readSettings()); - const mergedSettings = customDefu(currentSettings, settings) as DirectusSettings; + + // Issue #013: Filter settings if selectedSettings provided + let settingsToMerge = settings; + if (selectedSettings && selectedSettings.length > 0 && settings) { + const filteredSettings: Partial = {}; + for (const field of selectedSettings) { + if (field in settings) { + (filteredSettings as any)[field] = (settings as any)[field]; + } + } + settingsToMerge = filteredSettings as Settings; + res.write(`* Filtering to ${selectedSettings.length} selected field(s): ${selectedSettings.join(', ')}\r\n\r\n`); + } + + const mergedSettings = customDefu(settingsToMerge, currentSettings) as DirectusSettings; if (!dry_run) { await client.request(updateSettings(mergedSettings)); diff --git a/packages/migration-bundle/src/migration-endpoint/composables/migrate-translations.ts b/packages/migration-bundle/src/migration-endpoint/composables/migrate-translations.ts index d2f7c214..86759bfe 100644 --- a/packages/migration-bundle/src/migration-endpoint/composables/migrate-translations.ts +++ b/packages/migration-bundle/src/migration-endpoint/composables/migrate-translations.ts @@ -3,17 +3,57 @@ import type { DirectusError, Translation } from '../../types/extension'; import type { Schema } from '../api'; import { createTranslations, readTranslations } from '@directus/sdk'; -async function migrateTranslations({ res, client, translations, dry_run = false }: { res: any; client: RestClient; translations: Translation[] | null; dry_run: boolean }): Promise<{ response: string; name: string } | DirectusError> { +async function migrateTranslations({ + res, + client, + translations, + selectedLanguages, + translationKeyPattern, + dry_run = false, +}: { + res: any; + client: RestClient; + translations: Translation[] | null; + selectedLanguages?: string[]; + translationKeyPattern?: string; + dry_run: boolean; +}): Promise<{ response: string; name: string } | DirectusError> { if (!translations) { res.write('* Couldn\'t read data from extract\r\n\r\n'); return { name: 'Directus Error', status: 404, errors: [{ message: 'No translations found' }] }; } - else if (translations.length === 0) { - res.write('* No Translations to migrate\r\n\r\n'); + + // Issue #013: Handle empty language selection = skip migration + if (selectedLanguages && selectedLanguages.length === 0) { + res.write('* Translations skipped (empty language selection)\r\n\r\n'); + return { response: 'Skipped', name: 'Translations' }; + } + + // Issue #013: Filter by selected languages + let filteredTranslations = translations; + if (selectedLanguages && selectedLanguages.length > 0) { + filteredTranslations = translations.filter((t) => selectedLanguages.includes(t.language)); + res.write(`* Filtering to languages: ${selectedLanguages.join(', ')}\r\n\r\n`); + } + + // Issue #013: Filter by key pattern + if (translationKeyPattern) { + try { + const regex = new RegExp(translationKeyPattern); + filteredTranslations = filteredTranslations.filter((t) => regex.test(t.key)); + res.write(`* Filtering by key pattern: ${translationKeyPattern}\r\n\r\n`); + } + catch (e) { + res.write(`* Invalid key pattern regex: ${translationKeyPattern}\r\n\r\n`); + } + } + + if (filteredTranslations.length === 0) { + res.write('* No Translations match filter criteria\r\n\r\n'); return { response: 'Empty', name: 'Translations' }; } - res.write(`* [Local] Found ${translations.length} translations\r\n\r\n`); + res.write(`* [Local] Found ${filteredTranslations.length} translations (filtered from ${translations.length})\r\n\r\n`); try { // Fetch existing translations @@ -23,7 +63,7 @@ async function migrateTranslations({ res, client, translations, dry_run = false const existingTranslationKeys = new Set(existingTranslations.map((t) => `${t.language}_${t.key}`)); - const newTranslations = translations.filter((t) => { + const newTranslations = filteredTranslations.filter((t) => { const key = `${t.language}_${t.key}`; if (existingTranslationKeys.has(key)) { diff --git a/packages/migration-bundle/src/migration-endpoint/index.ts b/packages/migration-bundle/src/migration-endpoint/index.ts index e53ad550..ffc2f458 100644 --- a/packages/migration-bundle/src/migration-endpoint/index.ts +++ b/packages/migration-bundle/src/migration-endpoint/index.ts @@ -4,6 +4,7 @@ import { ForbiddenError } from '@directus/errors'; import { defineEndpoint } from '@directus/extensions-sdk'; import { createDirectus, rest, staticToken } from '@directus/sdk'; import { toArray } from '@directus/utils'; +import { filterSchema } from '../utils/filter-schema'; import notifyUser from '../utils/notify-user'; import saveToFile from '../utils/save-file'; import { validate_data, validate_migration, validate_system } from '../utils/validate'; @@ -62,6 +63,13 @@ export default defineEndpoint({ options: env.MIGRATION_BUNDLE_DEFAULT_OPTIONS ? String(env.MIGRATION_BUNDLE_DEFAULT_OPTIONS).split(',').map((s: string) => s.trim()) : [], + // Collection-level filtering from environment variables + selectedCollections: env.MIGRATION_BUNDLE_SELECTED_COLLECTIONS + ? String(env.MIGRATION_BUNDLE_SELECTED_COLLECTIONS).split(',').map((s: string) => s.trim()) + : [], + excludedCollections: env.MIGRATION_BUNDLE_EXCLUDED_COLLECTIONS + ? String(env.MIGRATION_BUNDLE_EXCLUDED_COLLECTIONS).split(',').map((s: string) => s.trim()) + : [], }; res.json(defaults); } @@ -248,7 +256,9 @@ export default defineEndpoint({ try { // Step 1.1: Schema res.write(`

${spinner} Creating Schema Snapshot

\r\n\r\n`); - const currentSchema = await schemaService.snapshot(); + const fullSchema = await schemaService.snapshot(); + // Apply collection filtering to schema if specified + const currentSchema = scope.schema ? filterSchema(fullSchema, scope) : fullSchema; await saveToFile(currentSchema, 'schema', fileService, folder, storage); res.write(`

${Icon} Schema Snapshot Created

\r\n\r\n`); @@ -289,35 +299,86 @@ export default defineEndpoint({ // Step 2: Migration res.write(isDryRun ? '## Checking Destination\r\n\r\n' : '## Starting Migration\r\n\r\n'); - // Step 2.1 Schema - res.write(`

${spinner} Applying Schema

\r\n\r\n`); - const response = await migrateSchema({ res, client, schema: currentSchema, dry_run: isDryRun, force: scope.force }); // Can be { status: 204 } if there is no change - - if ('errors' in response) { - const message = Array.isArray(response.errors) && response.errors.length > 0 ? response.errors[0]?.message : 'Unknown'; - await notifyUser(notificationService, accountability, response); - res.write(`

${Icon} Schema Failed to Apply

\r\n\r\n`); - res.write(`Error Occurred: ${message}\r\n\r\n`); + + // Step 2.1 Schema (conditional) + let schemaMigrationOk = true; + if (scope.schema) { + res.write(`

${spinner} Applying Schema

\r\n\r\n`); + const response = await migrateSchema({ res, client, schema: currentSchema, dry_run: isDryRun, force: scope.force }); // Can be { status: 204 } if there is no change + + if ('errors' in response) { + const message = Array.isArray(response.errors) && response.errors.length > 0 ? response.errors[0]?.message : 'Unknown'; + await notifyUser(notificationService, accountability, response); + res.write(`

${Icon} Schema Failed to Apply

\r\n\r\n`); + res.write(`Error Occurred: ${message}\r\n\r\n`); + schemaMigrationOk = false; + } + else { + res.write(`

${Icon} Schema Applied

\r\n\r\n`); + } } else { - res.write(`

${Icon} Schema Applied

\r\n\r\n`); + res.write(`

${Icon} Schema Skipped

\r\n\r\n`); + } - // Step 2.2: Users + // Continue with other migrations if schema succeeded or was skipped + if (schemaMigrationOk) { + // Step 2.2: Users (with granular options) if (scope.users) { res.write(`

${spinner} Migrating Users

\r\n\r\n`); - const role_response = await migrateRoles({ res, client, roles: systemFetch.roles, dry_run: isDryRun }); - const policy_response = await migratePolicies({ res, client, policies: systemFetch.policies, dry_run: isDryRun }); - const permission_response = await migratePermissions({ res, client, permissions: systemFetch.permissions, dry_run: isDryRun }); - const user_response = await migrateUsers({ res, client, users: systemFetch.users, roles: systemFetch.roles, dry_run: isDryRun }); - const access_response = await migrateAccess({ res, client, access: systemFetch.access, roles: systemFetch.roles, dry_run: isDryRun }); - - const userMigrationValid = await validate_migration([ - role_response, - policy_response, - permission_response, - user_response, - access_response, - ]); + + // Get granular options with defaults + const granular = scope.usersGranular || { + roles: true, + policies: true, + permissions: true, + userAccounts: true, + access: true, + }; + + const migrationResponses = []; + + if (granular.roles) { + const role_response = await migrateRoles({ res, client, roles: systemFetch.roles, dry_run: isDryRun }); + migrationResponses.push(role_response); + } + else { + res.write('* Skipping roles\r\n'); + } + + if (granular.policies) { + const policy_response = await migratePolicies({ res, client, policies: systemFetch.policies, dry_run: isDryRun }); + migrationResponses.push(policy_response); + } + else { + res.write('* Skipping policies\r\n'); + } + + if (granular.permissions) { + const permission_response = await migratePermissions({ res, client, permissions: systemFetch.permissions, dry_run: isDryRun }); + migrationResponses.push(permission_response); + } + else { + res.write('* Skipping permissions\r\n'); + } + + if (granular.userAccounts) { + const user_response = await migrateUsers({ res, client, users: systemFetch.users, roles: systemFetch.roles, dry_run: isDryRun }); + migrationResponses.push(user_response); + } + else { + res.write('* Skipping user accounts\r\n'); + } + + if (granular.access) { + const access_response = await migrateAccess({ res, client, access: systemFetch.access, roles: systemFetch.roles, dry_run: isDryRun }); + migrationResponses.push(access_response); + } + else { + res.write('* Skipping access rules\r\n'); + } + + const userMigrationValid = await validate_migration(migrationResponses); res.write(userMigrationValid ? `

${Icon} Users Migrated

\r\n\r\n` : `

${Icon} Users Migration Failed

\r\n\r\n`); } @@ -325,13 +386,11 @@ export default defineEndpoint({ res.write(`

${Icon} Users Skipped

\r\n\r\n`); } - if (scope.content) { - res.write(`

${spinner} Removing Field Requirements

\r\n\r\n`); - const field_response = await updateRequiredFields({ res, client, service: fieldService, collections: dataFetch.collections, dry_run: isDryRun, task: 'remove' }); - const fieldUpdateValid = await validate_migration([field_response]); - res.write(fieldUpdateValid ? `

${Icon} Field Requirements Removed

\r\n\r\n` : `

${Icon} Failed to Remove Field Requirements

\r\n\r\n`); + // Step 2.3: Files (now separate from content) + // For backward compatibility: if content is selected but files is not explicitly set, include files + const shouldMigrateFiles = scope.files === true || (scope.content && scope.files !== false); - // Step 2.3: Files + if (shouldMigrateFiles) { res.write(`

${spinner} Migrating Files

\r\n\r\n`); const folder_response = await migrateFolders({ res, client, folders: systemFetch.folders, dry_run: isDryRun }); const file_response = await migrateFiles({ res, client, service: assetService, files: dataFetch.files, dry_run: isDryRun }); @@ -342,29 +401,35 @@ export default defineEndpoint({ ]); res.write(fileMigrationValid ? `

${Icon} Files Migrated

\r\n\r\n` : `

${Icon} Files Migration Partially Failed

\r\n\r\n`); + } + else { + res.write(`

${Icon} Files Skipped

\r\n\r\n`); + } + + // Step 2.3b: Folders Only (when folders enabled but files not migrating) + if (scope.folders && !shouldMigrateFiles) { + res.write(`

${spinner} Migrating Folders

\r\n\r\n`); + const folder_response = await migrateFolders({ res, client, folders: systemFetch.folders, dry_run: isDryRun }); + const folderMigrationValid = await validate_migration([folder_response]); + res.write(folderMigrationValid ? `

${Icon} Folders Migrated

\r\n\r\n` : `

${Icon} Folders Migration Failed

\r\n\r\n`); + } + + // Step 2.4: Content + if (scope.content) { + res.write(`

${spinner} Removing Field Requirements

\r\n\r\n`); + const field_response = await updateRequiredFields({ res, client, service: fieldService, collections: dataFetch.collections, dry_run: isDryRun, task: 'remove' }); + const fieldUpdateValid = await validate_migration([field_response]); + res.write(fieldUpdateValid ? `

${Icon} Field Requirements Removed

\r\n\r\n` : `

${Icon} Failed to Remove Field Requirements

\r\n\r\n`); if (fieldUpdateValid) { - // Step 2.4: Data + // Step 2.5: Data res.write(`

${spinner} Migrating Collections

\r\n\r\n`); const content_response = await migrateData({ res, client, fullData: dataFetch.fullData, singletons: dataFetch.singletons, dry_run: isDryRun }); const contentMigrationValid = await validate_migration([content_response]); res.write(contentMigrationValid ? `

${Icon} Collections Migrated

\r\n\r\n` : `

${Icon} Collections Migration Failed

\r\n\r\n`); - - if (contentMigrationValid) { - // Step 2.5: Comments - if (scope.comments) { - res.write(`

${spinner} Migrating Comments

\r\n\r\n`); - const comments_response = await migrateComments({ res, client, comments: systemFetch.comments, dry_run: isDryRun }); - const contentMigrationValid = await validate_migration([comments_response]); - res.write(contentMigrationValid ? `

${Icon} Comments Migrated

\r\n\r\n` : `

${Icon} Comments Migration Failed

\r\n\r\n`); - } - else { - res.write(`

${Icon} Comments Skipped

\r\n\r\n`); - } - } } - // Step 2.6: Required Fields + // Step 2.7: Required Fields res.write(`

${spinner} Updating Required Fields

\r\n\r\n`); const fields_response = await updateRequiredFields({ res, client, service: fieldService, collections: dataFetch.collections, dry_run: isDryRun, task: 'add' }); const fieldsUpdateValid = await validate_migration([fields_response]); @@ -374,11 +439,22 @@ export default defineEndpoint({ res.write(`

${Icon} Content Skipped

\r\n\r\n`); } + // Step 2.6: Comments (standalone - not dependent on content) + if (scope.comments) { + res.write(`

${spinner} Migrating Comments

\r\n\r\n`); + const comments_response = await migrateComments({ res, client, comments: systemFetch.comments, dry_run: isDryRun }); + const commentsMigrationValid = await validate_migration([comments_response]); + res.write(commentsMigrationValid ? `

${Icon} Comments Migrated

\r\n\r\n` : `

${Icon} Comments Migration Failed

\r\n\r\n`); + } + else { + res.write(`

${Icon} Comments Skipped

\r\n\r\n`); + } + // Step 2.7: Dashboards if (scope.dashboards) { res.write(`

${spinner} Migrating Insights

\r\n\r\n`); - const dashboard_response = await migrateDashboards({ res, client, dashboards: systemFetch.dashboards, dry_run: isDryRun }); - const panel_response = await migratePanels({ res, client, panels: systemFetch.panels, dry_run: isDryRun }); + const dashboard_response = await migrateDashboards({ res, client, dashboards: systemFetch.dashboards, selectedDashboards: scope.selectedDashboards, dry_run: isDryRun }); + const panel_response = await migratePanels({ res, client, panels: systemFetch.panels, selectedDashboards: scope.selectedDashboards, dry_run: isDryRun }); const insightsMigrationValid = await validate_migration([dashboard_response, panel_response]); res.write(insightsMigrationValid ? `

${Icon} Insights Migrated

\r\n\r\n` : `

${Icon} Insights Migration Failed

\r\n\r\n`); } @@ -398,21 +474,31 @@ export default defineEndpoint({ } // Step 2.9: Settings - res.write(`

${spinner} Migrating Settings

\r\n\r\n`); - const settings_response = await migrateSettings({ res, client, settings: systemFetch.settings, dry_run: isDryRun }); - const settingsMigrationValid = await validate_migration([settings_response]); - res.write(settingsMigrationValid ? `

${Icon} Settings Migrated

\r\n\r\n` : `

${Icon} Settings Migration Failed

\r\n\r\n`); + if (scope.settings !== false) { + res.write(`

${spinner} Migrating Settings

\r\n\r\n`); + const settings_response = await migrateSettings({ res, client, settings: systemFetch.settings, selectedSettings: scope.selectedSettings, dry_run: isDryRun }); + const settingsMigrationValid = await validate_migration([settings_response]); + res.write(settingsMigrationValid ? `

${Icon} Settings Migrated

\r\n\r\n` : `

${Icon} Settings Migration Failed

\r\n\r\n`); + } + else { + res.write(`

${Icon} Settings Skipped

\r\n\r\n`); + } // Step 2.10: Translations - res.write(`

${spinner} Migrating Translations

\r\n\r\n`); - const translations_response = await migrateTranslations({ res, client, translations: systemFetch.translations, dry_run: isDryRun }); - const translationsMigrationValid = await validate_migration([translations_response]); - res.write(translationsMigrationValid ? `

${Icon} Translations Migrated

\r\n\r\n` : `

${Icon} Translations Migration Failed

\r\n\r\n`); + if (scope.translations !== false) { + res.write(`

${spinner} Migrating Translations

\r\n\r\n`); + const translations_response = await migrateTranslations({ res, client, translations: systemFetch.translations, selectedLanguages: scope.selectedLanguages, translationKeyPattern: scope.translationKeyPattern, dry_run: isDryRun }); + const translationsMigrationValid = await validate_migration([translations_response]); + res.write(translationsMigrationValid ? `

${Icon} Translations Migrated

\r\n\r\n` : `

${Icon} Translations Migration Failed

\r\n\r\n`); + } + else { + res.write(`

${Icon} Translations Skipped

\r\n\r\n`); + } - // Step 2.11: Translations + // Step 2.11: Presets if (scope.presets) { res.write(`

${spinner} Migrating Presets

\r\n\r\n`); - const presets_response = await migratePresets({ res, client, presets: systemFetch.presets, dry_run: isDryRun }); + const presets_response = await migratePresets({ res, client, presets: systemFetch.presets, selectedPresets: scope.selectedPresets, dry_run: isDryRun }); const presetsMigrationValid = await validate_migration([presets_response]); res.write(presetsMigrationValid ? `

${Icon} Presets Migrated

\r\n\r\n` : `

${Icon} Presets Migration Failed

\r\n\r\n`); } @@ -430,7 +516,7 @@ export default defineEndpoint({ else { res.write(`

${Icon} Extensions Skipped

\r\n\r\n`); } - } + } // End of schemaMigrationOk block res.write(`## Migration ${isDryRun ? 'Dry Run' : ''} Complete\r\n\r\n`); res.write(`The files can be download from the [file library](/admin/files/folders/${folder}).\r\n\r\n`); diff --git a/packages/migration-bundle/src/migration-module/module.vue b/packages/migration-bundle/src/migration-module/module.vue index 2c16ad9d..3152d467 100644 --- a/packages/migration-bundle/src/migration-module/module.vue +++ b/packages/migration-bundle/src/migration-module/module.vue @@ -1,12 +1,12 @@