feat(migration-to-aws): Migration plugin#73
Open
icarthick wants to merge 64 commits intoawslabs:mainfrom
Open
Conversation
Add complete gcp-to-aws plugin for AWS Agent Plugins marketplace: - 5-phase migration workflow: Discover, Clarify, Design, Estimate, Execute - Terraform-based infrastructure discovery and resource classification - AWS service mapping with 2-pass design evaluation (fast-path + rubric) - Cost estimation with pricing MCP integration and fallback - Execution timeline and risk assessment Files: - plugin.json: Plugin manifest with metadata and version - .mcp.json: MCP server configuration (awspricing, awsknowledge) - SKILL.md: Main orchestrator (~280 lines) with phase routing - references/phases/: 5 phase implementations (discover, clarify, design, estimate, execute) - references/design-refs/: Service mapping rubrics (compute, database, storage, networking, messaging, ai) - references/design-refs/fast-path.md: Deterministic 1:1 mappings (8 services) - references/shared/: Clarification questions, output schemas, pricing fallback - README.md: Plugin overview and usage Scope (v1.0): - Terraform infrastructure only (no app code scanning yet) - Design and estimation (no IaC code generation) - Dev sizing by default, production overridable - State management via .migration/[MMDD-HHMM]/ directory Future (v1.1+): - App code scanning for AI workload detection - AI-only fast-track path in Clarify/Design - Billing data import from GCP - Flat Design path for non-Terraform codebases Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
…ar files Decomposed monolithic discover.md into: **Discover Phase** (references/phases/discover/): - discover.md: Lightweight orchestrator (~40 lines) - discover-iac.md: Terraform-specific discovery (~75 lines) - discover-billing.md: Billing stub v1.2+ (~35 lines) - discover-app-code.md: App code stub v1.1+ (~35 lines) - unify-resources.md: Combines domain outputs (~75 lines) **Clustering Logic** (references/clustering/terraform/): - classification-rules.md: PRIMARY/SECONDARY hardcoded lists (~85 lines) - typed-edges-strategy.md: Relationship type inference from HCL (~105 lines) - clustering-algorithm.md: Rules 1-6 deterministic clustering (~125 lines) - depth-calculation.md: Topological sort algorithm (~130 lines) **Changes**: - Updated SKILL.md Phase Summary Table to point to new discover/discover.md - Deleted old monolithic references/phases/discover.md Enables v1.1 (app code) and v1.2 (billing) support by adding new files only—zero changes to existing discoverers or orchestration logic. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
…ptive Updated discover phase files to enforce strict workflow compliance: **SKILL.md**: - Workflow Execution section now requires agent to: - Follow EVERY step in order - NOT skip or optimize - Validate outputs before proceeding - Stop immediately on any failure **discover.md**: - Numbered steps with explicit dependencies - WAIT statements before proceeding - Schema validation requirements - Error handling for each step - Prevents agent from deviating or optimizing **discover-iac.md**: - Step-by-step instructions for each phase (parse, classify, edge-build, depth, cluster) - Exact JSON schema for iac_resources.json with required fields - Validation confirmations after each step - MANDATORY file write requirement with schema specification - Prevents custom schema creation **unify-resources.md**: - PATH A (IaC) vs PATH B (non-IaC) decision logic - Intermediate file validation with required schemas - Exact JSON schemas for both output files (inventory + clusters) - Step-by-step merge logic - MANDATORY file write with validation - Prevents autonomous schema modifications **Key changes**: - Added "Execute ALL steps in order. Do not skip or deviate." to each file - Added "WAIT for completion" and "Validate file exists" checks - Embedded exact JSON schemas with required/optional fields - Added error handling with specific failure messages - Removed ambiguity that allowed agent optimization This ensures v1.1/v1.2 compatibility by enforcing deterministic, predictable workflow. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
… final outputs
Simplified architecture based on testing feedback:
**Key Changes:**
1. **discover.md** (simplified orchestrator):
- Step 0: Initialize .migration/[MMDD-HHMM]/
- Step 1: Scan for Terraform files
- Step 2: Call discover-iac.md (Terraform only, v1.0)
- Step 3: Update .phase-status.json
- No more intermediate file handling or unify-resources delegation
2. **discover-iac.md** (self-contained, produces FINAL outputs):
- Step 1: Parse Terraform files
- Step 2: Classify resources (PRIMARY/SECONDARY)
- Step 3: Build typed dependency edges + populate serves[]
- Step 4: Calculate topological depth (Kahn's algorithm)
- Step 5: Apply clustering algorithm (Rules 1-6)
- Step 6: Write FINAL outputs directly:
* gcp-resource-inventory.json (with metadata, all resources)
* gcp-resource-clusters.json (with all clusters)
- Based on proven logic from discover-full.md
- Exact JSON schemas provided for both outputs
3. **Deleted:**
- unify-resources.md (no longer needed)
- Intermediate iac_resources.json concept abandoned
4. **discover-billing.md & discover-app-code.md:**
- Remain as stubs for v1.1/v1.2
- Merging strategy deferred (TBD in future versions)
**Benefits:**
- No intermediate files or orchestration complexity
- discover-iac.md is self-contained and testable
- Clarify/Design phases work immediately after discover
- Cleaner error handling - outputs are final
- Scales easily for v1.1/v1.2 (each source produces own outputs)
**Critical Field Names:**
Inventory: address, type, classification, secondary_role, cluster_id, depth, serves
Clusters: cluster_id, primary_resources, secondary_resources, creation_order_depth, edges
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
…s ONLY 2 JSON files Added strict constraints to both discover.md and discover-iac.md: **Output files ONLY:** - gcp-resource-inventory.json (REQUIRED) - gcp-resource-clusters.json (REQUIRED) **Forbidden (waste tokens):** - README.md - discovery-summary.md - EXECUTION_REPORT.txt - discovery-log.md - Any documentation or report files All user communication via output messages, NOT written files. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
…per resource Critical fix to clustering algorithm Rule 2: **Problem:** Current implementation creates separate clusters for each resource: - 4× google_pubsub_topic → 4 clusters (WRONG) - 3× google_storage_bucket → 3 clusters (WRONG) **Solution:** Group ALL resources of same type into ONE cluster: - 4× google_pubsub_topic → 1 cluster (messaging_pubsubtopic_us-central1_001) - 3× google_storage_bucket → 1 cluster (storage_bucket_us-central1_001) - 2× google_sql_database_instance → 1 cluster (database_sql_us-central1_001) **Examples updated to show:** - ✅ CORRECT: 4 topics → 1 cluster - ❌ INCORRECT: 4 topics → 4 clusters **Expected result:** 5-7 clusters instead of 11+ Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Merged branch: refactor/discover-architecture **Changes:** - Simplified discover phase architecture - discover-iac.md now produces final JSON outputs directly - Removed intermediate file complexity (unify-resources.md) - Clarified Rule 2 in clustering algorithm (ONE cluster per type, not per resource) - Added constraints: Only 2 JSON files output, no extra documentation **Files modified:** - discover.md: Simplified orchestrator (3 steps) - discover-iac.md: Self-contained, produces both final outputs - clustering-algorithm.md: Clarified Rule 2 with explicit examples - Deleted: unify-resources.md **Status:** Ready for testing with discover-iac producing 5-7 clusters Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
icarthick
changed the title
icarthick
changed the title
- Add .bandit configuration file with exclude_dirs for node_modules, .tmp, .git - Update mise.toml bandit task to use .bandit config - Prevents false positives from dependency code Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
- Format markdown tables and code blocks per dprint standards - Improves consistency across plugin documentation Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
- Specifies npm dev dependencies (dprint, markdownlint-cli2, ajv-cli) - Locks dependency versions for consistent builds Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
krokoko
reviewed
Feb 26, 2026
krokoko
reviewed
Feb 26, 2026
krokoko
reviewed
Feb 26, 2026
krokoko
reviewed
Feb 26, 2026
These files are generated artifacts from npm and shouldn't be committed. Mise manages npm tool installation, not npm directly. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
These files are generated artifacts from npm package management and should not be tracked in version control. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Contributor
|
Thanks ! First pass:
|
Reverted: - mise.toml: Changed bandit command back to 'bandit -r .' - Removed .bandit configuration file Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
- Add gcp-to-aws to CODEOWNERS file with plugin team owners - Update root README.md with gcp-to-aws in plugins table - Add installation instructions for gcp-to-aws - Add detailed plugin section with workflow, triggers, and MCP servers Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Author
|
@krokoko Thanks for the feedback, all your comments have been addressed in the latest commits |
krokoko
reviewed
Feb 26, 2026
Contributor
|
@icarthick thanks ! Please see the last comment and make sure your build passes locally (fix linting/formatting issues), thanks ! |
Align markdown tables per dprint formatting standards. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
These files are not generated by mise, so no need to ignore them. Keep only node_modules/ which is the actual dependency artifact. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Author
|
@krokoko Apologies for all the back and forth. Let me know if the latest looks good |
…type 1. classification-rules.md: Move google_compute_network from SECONDARY network_path to PRIMARY list, matching output-schema.md and clustering Rule 1 (network anchors its own cluster as PRIMARY) 2. estimate.md: Align report example with JSON example — data transfer $10 (not $10,000), total $27,510 (not $32,500), payback 27.5 months (not 32.5), 5-year savings $32,490 (not $28,500) 3. output-schema.md: Change network_membership back to network_path — network_membership is not defined in typed-edges-strategy.md Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CRITICAL-2: discover.md Step 2 now STOPs when no .tf files are found instead of silently completing with zero output files. CRITICAL-3: clarify.md gains Step 0 input validation matching estimate.md rigor — checks for missing files, invalid JSON, and empty arrays. CRITICAL-4: execute.md Step 1 replaces vague "incomplete" check with structural validation — verifies non-empty clusters, required fields on each resource, non-zero cost totals, and all expected sections. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CRITICAL-1: DynamoDB example label said "100k writes, 1m reads" but costs matched 100M writes and 1B reads. Fixed label to match actual arithmetic and added calculation note. CRITICAL-2: Cluster schema in clustering-algorithm.md had 6 fields (name, type, description, network, must_migrate_together, dependencies) missing from canonical output-schema.md. Added all 6 to the schema example. CRITICAL-3: classification-rules.md listed google_firestore_database as PRIMARY but index.md/database.md expected google_firestore_document. Both are valid Terraform types — added both to classification-rules.md and index.md so neither falls through. CRITICAL-4: RDS pricing labeled "Multi-AZ" but values were Single-AZ rates. Corrected label to "Single-AZ" with note about Multi-AZ doubling. Also fixed: classification-rules.md Serves[] example incorrectly called google_compute_network SECONDARY when it is PRIMARY (Priority 1 list). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix Lambda pricing math: compute_cost 0.083→8.33, total 0.283→8.53 - Add google_redis_instance to fast-path.md and index.md (ElastiCache Redis) - Add pricing_source + timestamp to estimate.md inline example - Add clarified.json validation to estimate.md Step 0 - Add gcp-resource-inventory.json validation to design.md Step 0 - Add validation_status + timestamp to design.md Step 4 inline example - Add gcp_config to output-schema.md aws-design.json resource example - Fix execute.md gcp_teardown string → gcp_teardown_week integer + timestamp - Fix messaging.md: SNS FIFO supports exactly-once via deduplication Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CRITICAL fixes: - Fix DynamoDB eliminator: strong consistency IS supported, change to 100-item transaction limit - Add 3 missing PRIMARY resources to classification-rules.md: google_app_engine_application, google_cloud_tasks_queue, google_compute_forwarding_rule - Fix confidence value in design.md: google_compute_instance is not in fast-path.md, so confidence is "inferred" not "deterministic" - Fix Fargate common_sizes to match unit rate calculations - Add JSON validity + array content checks to design.md Step 0 HIGH fixes: - Fix execute.md hardcoded payback period, use placeholder from estimation - Add NAT Gateway line to estimate.md markdown report template - Fix messaging.md Signals: exactly-once → SNS FIFO + SQS FIFO - Fix Fargate eliminator: 10 GB is not a limit, use GPU/vCPU/memory limits - Add Redis section to database.md rubric coverage - Add db.t4g family to pricing-fallback.json (matches SKILL.md default) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CRITICAL fixes: - Fix simple_web_app total: 113.69 → 115.59 (sum of components) - Fix S3 Standard storage math: use 1024 GB consistently (23.55, not 23.04) - Fix ai_chatbot note: "1M reads" → "1B reads" (matches $250 cost) - Add config + dependencies to discover-iac.md Step 6a CRITICAL fields - Add 6 missing fields to discover-iac.md Step 6b CRITICAL fields - Fix BigQuery eliminator: Athena is OLAP not OLTP, recommend DynamoDB/Aurora - Add Bedrock, SQS, SNS, EventBridge, ElastiCache Redis pricing to fallback HIGH fixes: - Fix index.md usage: clarify fast-path.md lookup vs rubric routing - Fix SKILL.md default: Aurora Serverless v2 0.5 ACU (matches design refs) - Fix cloud_run equivalence: use compute.md rubric criteria, not memory - Move Redis signals from Examples section to Signals section in database.md Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- IMP-1: Standardize edge_type → relationship_type in typed-edges-strategy.md and discover-iac.md - IMP-3: Acknowledge LLM non-determinism in clustering-algorithm.md determinism guarantee - IMP-7: Add prominent user-facing warning when awsknowledge MCP unavailable in design.md - IMP-8: Add fallback pricing for EKS, ECS, VPC, Route 53, CloudFront, Redshift, Athena, SageMaker - IMP-9: Replace fabricated GCP cost (AWS*1.25) with cannot_calculate status in estimate.md ROI - IMP-10: Clarify pipe-separated convention note in output-schema.md - MED-1: Tiered staleness thresholds (≤30d, 30-60d, >60d) in estimate.md fallback path - MED-2/MED-4: Add version, empty file, and missing field validation to SKILL.md phase routing - MED-3: Bound cycle detection to max 3 attempts, only break inferred edges in depth-calculation.md - MED-5: Display phase-status.json contents when multiple migration sessions detected in SKILL.md - MED-6: Unknown resource types add to warnings instead of STOP in design.md - MED-8: Add low-confidence classification downstream flagging in classification-rules.md - Comment-13: Add non-null staleness_warning example to output-schema.md - XREF-1: Clarify unify-resources.md is planned v1.1+ in discover-app-code.md and discover-billing.md - XREF-2: Align regex pattern in discover-iac.md with typed-edges-strategy.md Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
krokoko
previously approved these changes
Mar 3, 2026
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
krokoko
previously approved these changes
Mar 3, 2026
Signed-off-by: Alain Krok <alkrok@amazon.com>
icarthick
added a commit
to icarthick/agent-plugins
that referenced
this pull request
Mar 4, 2026
Brings in all PR awslabs#73 review fixes from main: - Pricing fallback additions (EKS, ECS, VPC, Route 53, CloudFront, Redshift, Athena, SageMaker) - Depth calculation bounded cycle detection - Database/compute/messaging design-ref fixes - Fast-path and index.md updates Conflicts resolved by keeping parity-with-kiro versions for files that were significantly rewritten (SKILL.md, classification-rules.md, clustering-algorithm.md, typed-edges-strategy.md, clarify.md, design-infra.md, discover-*.md, output-schema.md). Old estimate.md and execute.md removed (replaced by subdirectory structure in parity-with-kiro). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Alain Krok <alkrok@amazon.com>
krokoko
previously approved these changes
Mar 6, 2026
| "author": { | ||
| "name": "Amazon Web Services" | ||
| }, | ||
| "description": "This no-cost tool assesses your current cloud provider's usage, geography, and billing data to estimate and compare AWS services and pricing, and recommends migration or continued use of your current provider. AWS pricing is based on current published pricing and may vary over time. The tool may generate a .migration folder containing comparison and migration execution data, which you may delete upon completion or use to migrate to AWS.", |
Author
There was a problem hiding this comment.
We had a long demo session with legal and this was their disclaimer. If you really feel strongly about this, am happy to go back to them.
Contributor
There was a problem hiding this comment.
cost is complicated but I believe the correct term is "no additional cost" or omit. The reality is there are normative API usage costs for things like querying logs, storage, etc. the plugins, however, are provided "at no additional charge".
Author
There was a problem hiding this comment.
Would it ok to get this merged as is? Am happy to go back and forth with Legal about this next week to settle on something that we all can agree?
theagenticguy
requested changes
Mar 6, 2026
| }, | ||
| { | ||
| "category": "migration", | ||
| "description": "This no-cost tool assesses your current cloud provider's usage, geography, and billing data to estimate and compare AWS services and pricing, and recommends migration or continued use of your current provider. AWS pricing is based on current published pricing and may vary over time. The tool may generate a .migration folder containing comparison and migration execution data, which you may delete upon completion or use to migrate to AWS.", |
Contributor
There was a problem hiding this comment.
Does this work? Does it need an LLM or are there real pattern-matching type tools to achieve the same outcome and then the agent can fill in the rest?
Author
There was a problem hiding this comment.
It does work, especially when there are complex relationships between gcp resources. The idea behind these prescriptive rules is to reduce the non-determinism from LLM.
In a subsequent, PR i have logic to skip these complex rules for simple GCP repositories, since it would be an overkill to load all these clustering rules
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
RFC: #50
This PR introduces the Migration-to-AWS migration plugin — the first plugin in the agent-plugins marketplace that helps developers systematically migrate GCP infrastructure to AWS.
The migration plugin follows a multi-phase workflow that analyzes existing GCP resources, recommends AWS equivalents, estimates costs, and generates working infrastructure-as-code.
High-Level Migration Flow
What This PR Delivers
This PR completes the DISCOVER phase with a modular, maintainable architecture that supports incremental feature additions for:
See RFC: GCP-to-AWS Migration Plugin Specification #50
Key Changes
File Structure Reorganization:
references/phases/discover/folder with modular files:discover.md- Lightweight router (detects Terraform, delegates to discover-iac.md)discover-iac.md- Terraform-specific discovery with full clustering pipelinediscover-billing.md- Stub for v1.2+ billing supportdiscover-app-code.md- Stub for v1.1+ app code scanningreferences/phases/discover.mdreferences/clustering/terraform/with 4 focused files:classification-rules.md- PRIMARY/SECONDARY resource classificationtyped-edges-strategy.md- Relationship inference from HCL contextclustering-algorithm.md- 6-rule deterministic clustering with fixesdepth-calculation.md- Topological sort for dependency depthClustering Algorithm Improvements:
Output Constraints:
gcp-resource-inventory.jsonandgcp-resource-clusters.jsonSpecification Precision:
Architectural Benefits
Merge Strategy
Please use 'Squash and Merge' when merging this PR. This will consolidate all commits into a single clean commit on main, keeping the repository history focused on architectural milestones.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the project license.