Skip to content

Commit 5416acf

Browse files
skyoo2003skyoo2003
committed
docs: add user and contributor documentation
Documentation suite under docs/, organized with MECE boundaries: - README.md: docs index / navigation map. - getting-started.md: Docker run, Docker Compose dev, boto3 / AWS CLI verification, source build pointer. - configuration.md: canonical config reference (YAML keys, env-var overrides including DEVCLOUD_SERVICES tier shortcuts and DEVCLOUD_DATA_DIR), zero-config behavior. - architecture.md: request flow, plugin system, protocol support, Smithy codegen pipeline, dashboard, startup flow, phased multi-CSP vision. - roadmap.md: four-phase plan from AWS depth to community-driven CSP breadth. - services-matrix.md: 96-service coverage matrix, operation counts, boto3 pass rate, cross-service integrations, supported protocols. - contributing.md: full dev setup, tests, codegen internals, adding a new service, license header and contribution terms. - faq.md: scope, compatibility, CI use, Windows, Terraform/CDK. - troubleshooting.md: common build and runtime errors with fixes. - services/{s3,sqs,dynamodb,lambda,iam-sts}.md: per-service API references with boto3 / AWS CLI examples. Positioning frames DevCloud as an on-ramp to the cloud, not a replacement; Phase-based narrative is centralized in roadmap.md.
1 parent 1dd4632 commit 5416acf

14 files changed

Lines changed: 1545 additions & 0 deletions

‎docs/README.md‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# DevCloud Documentation
2+
3+
This directory holds DevCloud's technical documentation. Use the map below to jump to what you need.
4+
5+
## Start here
6+
7+
- **[Getting Started](getting-started.md)** — install, first run, boto3 / AWS CLI / Terraform examples
8+
- **[Configuration](configuration.md)** — YAML options, env-var overrides, tier shortcuts
9+
- **[Architecture](architecture.md)** — system design, plugin model, codegen pipeline, multi-CSP vision
10+
- **[Roadmap](roadmap.md)** — phased plan toward multi-CSP support
11+
- **[Services Matrix](services-matrix.md)** — 96 services, coverage status, boto3 pass rate
12+
13+
## Per-service references
14+
15+
Located under [`services/`](services/):
16+
17+
- [S3](services/s3.md) — object storage (REST-XML)
18+
- [SQS](services/sqs.md) — message queue (Query + JSON)
19+
- [DynamoDB](services/dynamodb.md) — NoSQL KV + document (JSON 1.0)
20+
- [Lambda](services/lambda.md) — function runtime (REST-JSON)
21+
- [IAM / STS](services/iam-sts.md) — identity & tokens (Query)
22+
23+
## Problem solving
24+
25+
- **[FAQ](faq.md)** — common questions about scope, compatibility, CI use
26+
- **[Troubleshooting](troubleshooting.md)** — common errors and fixes
27+
28+
## Contributing
29+
30+
- **[Contributing Guide](contributing.md)** — dev setup, testing, codegen, adding new services
31+
- Root-level pointers: [CONTRIBUTING.md](../CONTRIBUTING.md), [CODE_OF_CONDUCT.md](../CODE_OF_CONDUCT.md), [SECURITY.md](../SECURITY.md), [SUPPORT.md](../SUPPORT.md)
32+
33+
## Meta
34+
35+
- [Changelog](../CHANGELOG.md)
36+
- [License (Apache 2.0)](../LICENSE)
37+
- [Trademarks](../TRADEMARKS.md)
38+
- [NOTICE](../NOTICE)
39+
40+
---
41+
42+
If something is missing from this index, that's a bug. Please file an [issue](https://github.com/skyoo2003/devcloud/issues) or open a PR to fix the link.

‎docs/architecture.md‎

Lines changed: 198 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,198 @@
1+
# Architecture
2+
3+
## Multi-CSP Vision
4+
5+
DevCloud's long-term direction is to support multiple Cloud Service Providers (AWS, Azure, GCP, and others) behind a single local runtime. Today's implementation targets **AWS only** — the sections below describe the current AWS-specific architecture.
6+
7+
A phased refactor is planned (see [roadmap.md](roadmap.md)):
8+
9+
- **Phase 1 (current)** — AWS services via Smithy codegen, single-port gateway, AWS SigV4 auth.
10+
- **Phase 2 (preparation)** — Introduce an Intermediate Representation (IR) between API models and codegen. Abstract `ModelSource` so OpenAPI (Azure) and Protocol Buffers / Discovery Documents (GCP) can feed the same pipeline. Extract a per-provider auth adapter interface.
11+
- **Phase 3 (pilot)** — First non-AWS service (candidate: Azure Blob Storage) validates the multi-CSP architecture.
12+
- **Phase 4 (breadth)** — Additional services across CSPs; community-owned providers.
13+
14+
The existing plugin system, protocol detector, and storage abstractions are intentionally CSP-agnostic where possible, to minimize rework when providers are added.
15+
16+
## Request Flow
17+
18+
```
19+
Client (boto3 / AWS CLI / Terraform / CDK)
20+
│
21+
▼
22+
API Gateway (port 4747)
23+
│
24+
├─ Middleware Chain
25+
│ ├─ ErrorRecovery (panic recovery)
26+
│ ├─ BodyLimit (request size limiting)
27+
│ ├─ CORS (cross-origin handling)
28+
│ ├─ RequestID (X-Amz-Request-Id)
29+
│ ├─ RequestLogger (structured logging)
30+
│ └─ LogCollector (dashboard live logs)
31+
│
32+
├─ Route: /devcloud/api/* → Dashboard API
33+
│
34+
▼
35+
Protocol Detector
36+
│
37+
├─ X-Amz-Target header present → JSON protocol (DynamoDB, SQS JSON)
38+
├─ Content-Type: x-www-form-urlencoded + Action= → Query protocol (IAM, STS, SQS Query)
39+
├─ SigV4 with Lambda path → REST-JSON (Lambda)
40+
└─ Default → REST-XML (S3)
41+
│
42+
▼
43+
Auth Validator (SigV4 format check, account ID extraction)
44+
│
45+
▼
46+
Service Router → Plugin Registry → ServicePlugin.HandleRequest()
47+
│
48+
▼
49+
Service Implementation (S3, SQS, DynamoDB, Lambda, IAM, STS)
50+
│
51+
▼
52+
Storage Backend → Response Serializer → HTTP Response
53+
```
54+
55+
## Plugin System
56+
57+
All services implement the `ServicePlugin` interface:
58+
59+
```go
60+
type ServicePlugin interface {
61+
ServiceID() string
62+
ServiceName() string
63+
Protocol() ProtocolType
64+
Init(config PluginConfig) error
65+
Shutdown(ctx context.Context) error
66+
HandleRequest(ctx context.Context, op string, req *http.Request) (*Response, error)
67+
ListResources(ctx context.Context) ([]Resource, error)
68+
GetMetrics(ctx context.Context) (*ServiceMetrics, error)
69+
}
70+
```
71+
72+
Plugins are registered in the `Registry` at startup. The gateway routes requests to the correct plugin based on protocol detection and service identification.
73+
74+
### Supported Protocols
75+
76+
| Protocol | Services | Request Format | Response Format |
77+
|----------|----------|----------------|-----------------|
78+
| REST-XML | S3 | HTTP path/headers | XML |
79+
| JSON 1.0 | DynamoDB, SQS | JSON body, `X-Amz-Target` header | JSON |
80+
| JSON 1.1 | Lambda | JSON body, REST path | JSON |
81+
| Query | IAM, STS, SQS | Form-encoded body with `Action=` | XML |
82+
83+
SQS supports both Query and JSON protocols. The protocol is auto-detected per request based on Content-Type and headers.
84+
85+
## Smithy Code Generation
86+
87+
DevCloud auto-generates Go code from AWS Smithy JSON models. This enables rapid tracking of AWS API changes with minimal manual work.
88+
89+
### Pipeline
90+
91+
```
92+
smithy-models/*.json (AWS Smithy model files)
93+
│
94+
▼
95+
Parser (internal/codegen/parser.go)
96+
Reads service definitions, operations, input/output shapes
97+
│
98+
▼
99+
Generator (internal/codegen/generator.go)
100+
Uses Go templates to produce:
101+
│
102+
├─ types.go — Request/response structs
103+
├─ interface.go — Service interface (all operations)
104+
├─ serializer.go — Request marshaling
105+
├─ deserializer.go — Response unmarshaling
106+
├─ router.go — Operation routing
107+
├─ errors.go — Service-specific error types
108+
└─ base_provider.go — Stub implementation (NotImplementedError)
109+
│
110+
▼
111+
internal/generated/{service}/ (DO NOT EDIT)
112+
```
113+
114+
### Running Codegen
115+
116+
```bash
117+
# Generate code for all services
118+
make codegen
119+
120+
# Generate for a specific service
121+
make codegen-s3
122+
```
123+
124+
### Weekly Auto-Sync
125+
126+
A GitHub Actions workflow runs weekly to:
127+
1. Fetch the latest Smithy models from AWS
128+
2. Run codegen
129+
3. Open a PR if any generated code changed
130+
131+
## Event Bus
132+
133+
The event bus (`internal/eventbus/`) provides in-memory pub/sub for internal communication:
134+
135+
- Services publish events (resource created, deleted, etc.)
136+
- Dashboard subscribes to stream real-time updates via WebSocket
137+
- Loose coupling between services and dashboard
138+
139+
## Dashboard
140+
141+
The dashboard (`internal/dashboard/`) provides:
142+
143+
- **REST API** at `/devcloud/api/` — service status, resource listing, request logs
144+
- **WebSocket** at `/devcloud/api/ws` — real-time event streaming
145+
- **Web UI** — Next.js static export served by the Go server
146+
147+
Log collector maintains a circular buffer of the last 1000 API requests for the dashboard log viewer.
148+
149+
## Directory Structure
150+
151+
```
152+
devcloud/
153+
├── cmd/
154+
│ ├── devcloud/ # Server entry point (main.go)
155+
│ └── codegen/ # Smithy code generator CLI (main.go)
156+
├── internal/
157+
│ ├── gateway/ # HTTP server, middleware, protocol detection, routing
158+
│ ├── plugin/ # ServicePlugin interface, Registry
159+
│ ├── codegen/ # Smithy parser, code generators, templates
160+
│ ├── config/ # YAML config loading, env overrides
161+
│ ├── generated/ # Auto-generated code (DO NOT EDIT)
162+
│ │ ├── s3/
163+
│ │ ├── sqs/
164+
│ │ ├── dynamodb/
165+
│ │ ├── lambda/
166+
│ │ ├── iam/
167+
│ │ └── sts/
168+
│ ├── services/ # Service implementations
169+
│ │ ├── s3/ # FileSystem + SQLite
170+
│ │ ├── sqs/ # In-memory queues
171+
│ │ ├── dynamodb/ # BadgerDB
172+
│ │ ├── lambda/ # SQLite + filesystem (stub runtime)
173+
│ │ └── iam/ # SQLite (IAM + STS)
174+
│ ├── dashboard/ # Dashboard REST API + WebSocket
175+
│ ├── eventbus/ # In-memory event pub/sub
176+
│ └── storage/ # Shared storage abstractions
177+
├── web/ # Next.js dashboard (React, TypeScript, Tailwind)
178+
├── docker/ # Dockerfile, docker-compose.yml
179+
├── smithy-models/ # AWS Smithy JSON model files
180+
├── tests/compatibility/ # Python/boto3 compatibility tests
181+
├── docs/ # Documentation
182+
├── Makefile
183+
├── devcloud.yaml # Default configuration
184+
└── go.mod
185+
```
186+
187+
## Startup Flow
188+
189+
1. Load config from `devcloud.yaml` (or specified path)
190+
2. Initialize structured logger (slog)
191+
3. Create plugin registry
192+
4. Register service factories (S3, SQS, DynamoDB, IAM, Lambda)
193+
5. Initialize services in order: S3 → SQS → DynamoDB → IAM → STS → Lambda
194+
6. IAM store is shared with STS via plugin config options
195+
7. Set up event bus, log collector, dashboard API
196+
8. Create gateway with middleware chain and service router
197+
9. Start HTTP server on configured port
198+
10. Wait for shutdown signal (SIGINT/SIGTERM), graceful shutdown with 15s timeout

‎docs/configuration.md‎

Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
# Configuration
2+
3+
**Configuration is optional.** DevCloud ships with built-in defaults: running `devcloud` with no flags enables all 96 services on port 4747, storing data under `./data/<service>/`. The embedded defaults are compiled into the binary from [`internal/config/default.yaml`](https://github.com/skyoo2003/devcloud/blob/main/internal/config/default.yaml).
4+
5+
To override defaults, provide a YAML file. DevCloud looks for config in this order:
6+
7+
1. `--config <path>` flag (explicit; the file must exist)
8+
2. `./devcloud.yaml` in the current working directory (auto-detected)
9+
3. Embedded defaults (used when neither of the above is present)
10+
11+
Environment variables override YAML values for selected keys (see [Environment Variable Overrides](#environment-variable-overrides)).
12+
13+
## Configuration File
14+
15+
### Server
16+
17+
| Key | Default | Description |
18+
|-----|---------|-------------|
19+
| `server.port` | `4747` | HTTP server port |
20+
21+
### Services
22+
23+
Each service has the following options:
24+
25+
| Key | Default | Description |
26+
|-----|---------|-------------|
27+
| `services.<name>.enabled` | `true` | Enable or disable the service |
28+
| `services.<name>.data_dir` | varies | Data directory for persistent storage |
29+
30+
Service-specific options:
31+
32+
| Key | Default | Description |
33+
|-----|---------|-------------|
34+
| `services.lambda.runtime` | `""` | Lambda runtime configuration |
35+
| `services.lambda.warm_containers` | `0` | Number of warm containers to keep |
36+
| `services.iam.enforce_policies` | `false` | Enforce IAM policies (experimental) |
37+
38+
### Auth
39+
40+
| Key | Default | Description |
41+
|-----|---------|-------------|
42+
| `auth.enabled` | `false` | Enable SigV4 signature validation |
43+
44+
### Dashboard
45+
46+
| Key | Default | Description |
47+
|-----|---------|-------------|
48+
| `dashboard.enabled` | `false` | Enable the web dashboard |
49+
50+
### Logging
51+
52+
| Key | Default | Description |
53+
|-----|---------|-------------|
54+
| `logging.level` | `info` | Log level: `debug`, `info`, `warn`, `error` |
55+
| `logging.format` | `text` | Log format: `text` or `json` |
56+
57+
## Full Example
58+
59+
```yaml
60+
server:
61+
port: 4747
62+
63+
services:
64+
s3:
65+
enabled: true
66+
data_dir: ./data/s3
67+
sqs:
68+
enabled: true
69+
dynamodb:
70+
enabled: true
71+
data_dir: ./data/dynamodb
72+
iam:
73+
enabled: true
74+
data_dir: ./data/iam
75+
sts:
76+
enabled: true
77+
data_dir: ./data/sts
78+
lambda:
79+
enabled: true
80+
data_dir: ./data/lambda
81+
82+
auth:
83+
enabled: false
84+
85+
dashboard:
86+
enabled: false
87+
88+
logging:
89+
level: info
90+
format: text
91+
```
92+
93+
## Environment Variable Overrides
94+
95+
### `DEVCLOUD_SERVICES`
96+
97+
Comma-separated list of services to enable. All other services listed in the config file are disabled. Accepts individual service names and tier shortcuts.
98+
99+
**Tier shortcuts** (expand to predefined service groups — see [`internal/config/config.go`](https://github.com/skyoo2003/devcloud/blob/main/internal/config/config.go) for the exact list):
100+
101+
| Token | Expands to |
102+
|-------|------------|
103+
| `tier1` | Big 6 + core integration: s3, sqs, dynamodb, iam, sts, lambda, sns, kms, secretsmanager, ssm, cloudwatchlogs, cloudwatch, eventbridge, ec2, ecs, ecr, route53, acm |
104+
| `tier2` | Extended services: cognito, elasticloadbalancingv2, ebs, efs, states, apigateway, apigatewayv2, kinesis, firehose, ses, sesv2, rds, cloudformation |
105+
| `tier3` | Analytics & platform: elasticache, cloudfront, wafv2, glue, athena, organizations, cloudtrail, eks, autoscaling, appsync, emr, batch |
106+
| `all` | Disables the env-var filter (all enabled services in the config stay enabled) |
107+
108+
Examples:
109+
110+
```bash
111+
# Enable only S3 and SQS
112+
DEVCLOUD_SERVICES=s3,sqs ./dist/devcloud
113+
114+
# Enable all Tier 1 services
115+
DEVCLOUD_SERVICES=tier1 ./dist/devcloud
116+
117+
# Enable Tier 1 + a few extras
118+
DEVCLOUD_SERVICES=tier1,kinesis,firehose ./dist/devcloud
119+
120+
# With Docker
121+
docker run -p 4747:4747 -e DEVCLOUD_SERVICES=tier1 ghcr.io/skyoo2003/devcloud:latest
122+
```
123+
124+
### `DEVCLOUD_DATA_DIR`
125+
126+
Overrides the base data directory for **all** services. Each service's `data_dir` becomes `<DEVCLOUD_DATA_DIR>/<service_name>` regardless of what's in the YAML file.
127+
128+
```bash
129+
# Put all service data under /tmp/devcloud-local
130+
DEVCLOUD_DATA_DIR=/tmp/devcloud-local ./dist/devcloud
131+
132+
# With Docker (mount the host directory)
133+
docker run -p 4747:4747 \
134+
-e DEVCLOUD_DATA_DIR=/app/data \
135+
-v $(pwd)/devcloud-data:/app/data \
136+
ghcr.io/skyoo2003/devcloud:latest
137+
```
138+
139+
Useful for:
140+
- CI jobs that need ephemeral per-run data dirs
141+
- Quickly relocating state without editing `devcloud.yaml`
142+
143+
## Data Directories
144+
145+
Services with persistent storage create data under their configured `data_dir`:
146+
147+
| Service | Default `data_dir` | Storage Backend | Contents |
148+
|---------|-------------------|-----------------|----------|
149+
| S3 | `./data/s3` | Filesystem + SQLite | Object files, `metadata.db` |
150+
| DynamoDB | `./data/dynamodb` | BadgerDB | BadgerDB data files |
151+
| IAM | `./data/iam` | SQLite | `iam.db` (users, roles, keys) |
152+
| STS | `./data/sts` | Shared with IAM | Uses IAM's database |
153+
| Lambda | `./data/lambda` | SQLite + Filesystem | `lambda.db`, `code/` directory |
154+
| SQS | — | In-memory | No persistence |

0 commit comments

Comments
 (0)