Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 35 additions & 15 deletions .github/workflows/validate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,15 @@ on:
pull_request:
branches: [main]

permissions:
contents: read

jobs:
shellcheck:
name: Shell Script Lint
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
- name: Install shellcheck
run: sudo apt-get install -y shellcheck
- name: Lint shell scripts
Expand All @@ -27,9 +30,9 @@ jobs:
name: Terraform Validate
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
- name: Setup OpenTofu
uses: opentofu/setup-opentofu@v1
uses: opentofu/setup-opentofu@9d84900f3238fab8cd84ce47d658d25dd008be2f # v1.0.8
- name: Validate GCP provider
working-directory: providers/gcp/infra
run: |
Expand All @@ -41,7 +44,7 @@ jobs:
name: YAML Lint
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
- name: Install yamllint
run: pip install yamllint
- name: Lint YAML templates
Expand All @@ -53,14 +56,31 @@ jobs:
name: Secret Detection
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Check for secrets in code
run: |
# Check for common secret patterns (exclude docs and .github — they document the patterns)
if grep -rE 'sk-ant-|sk-proj-|AKIA[A-Z0-9]{16}' \
--include='*.sh' --include='*.tf' --include='*.yml' --include='*.yaml' --include='*.md' \
--exclude-dir=docs --exclude-dir=.github --exclude-dir=.git .; then
echo "ERROR: Possible secrets found in code!"
exit 1
fi
echo "No secrets detected"
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
with:
fetch-depth: 0
- name: Run gitleaks
uses: gitleaks/gitleaks-action@ff98106e4c7b2bc287b24eaf42907196329070c7 # v2.3.9
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITLEAKS_CONFIG: .gitleaks.toml

image-scan:
name: Container Image Scan
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
- name: Scan Qdrant image
uses: aquasecurity/trivy-action@b6643a29fecd7f34b3597bc6acb0a98b03d33ff8 # v0.33.1
with:
image-ref: 'qdrant/qdrant:v1.13.2'
format: 'table'
severity: 'CRITICAL,HIGH'
exit-code: '1'
- name: Scan Chrome image
uses: aquasecurity/trivy-action@b6643a29fecd7f34b3597bc6acb0a98b03d33ff8 # v0.33.1
with:
image-ref: 'chromedp/headless-shell:stable'
format: 'table'
severity: 'CRITICAL'
exit-code: '1'
15 changes: 15 additions & 0 deletions .gitleaks.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Gitleaks configuration for create-openclaw-agent
# Extends the default ruleset with project-specific allowlists

title = "create-openclaw-agent gitleaks config"

[allowlist]
description = "Global allowlist"
paths = [
'''docs/.*\.md$''',
'''.github/.*\.md$''',
'''AGENTS\.md$''',
'''CLAUDE\.md$''',
'''README\.md$''',
'''.gitleaks\.toml$''',
]
17 changes: 17 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Pre-commit hooks for create-openclaw-agent
# Install: pip install pre-commit && pre-commit install
repos:
- repo: https://github.com/gitleaks/gitleaks
rev: v8.21.2
hooks:
- id: gitleaks
- repo: https://github.com/koalaman/shellcheck-precommit
rev: v0.10.0
hooks:
- id: shellcheck
args: ["-x"]
- repo: https://github.com/gruntwork-io/pre-commit
rev: v0.1.23
hooks:
- id: terraform-fmt
- id: terraform-validate
43 changes: 38 additions & 5 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,15 @@ CLI tool for deploying self-hosted OpenClaw AI agents on cloud infrastructure. S
│ └── agent-config.example.yml # Portable agent config template
├── docs/
│ ├── gcp-guide.md # Secret Manager, IAM, IAP, troubleshooting
│ └── mem0-setup.md # Mem0 plugin setup guide
│ ├── mem0-setup.md # Mem0 plugin setup guide
│ └── security.md # Full security architecture document
├── .github/workflows/
│ └── validate.yml # CI: shellcheck, tofu validate, smoke tests
├── scripts/
│ ├── run-e2e.sh # E2E test runner (env-based API keys)
│ └── verify-e2e-cleanup.sh # Post-E2E resource cleanup verification
├── .gitleaks.toml # Gitleaks secret scanning config
├── .pre-commit-config.yaml # Pre-commit hooks (gitleaks, shellcheck, tofu)
├── .gitignore
├── CLAUDE.md # Same as AGENTS.md
├── AGENTS.md # This file
Expand All @@ -45,15 +51,29 @@ CLI tool for deploying self-hosted OpenClaw AI agents on cloud infrastructure. S

- **Cloud-agnostic**: Provider interface in `providers/<cloud>/provider.sh`. GCP first; community adds AWS/Azure by implementing the same functions.
- **No external IP**: VM accessible only via IAP tunnel (GCP) or equivalent.
- **3 containers**: OpenClaw gateway + Qdrant vector store + Chrome headless (CDP). Total resource needs: 2.5 CPU, 3GB RAM. Requires e2-medium or larger.
- **3 containers on bridge network**: OpenClaw gateway + Qdrant vector store + Chrome headless (CDP) on an isolated Docker bridge network (`openclaw-net`). Gateway ports bound to `127.0.0.1` only. Read-only root filesystems (`read_only: true`), all Linux capabilities dropped (`cap_drop: ALL`), `no-new-privileges`. Total resource needs: 2.5 CPU, 3GB RAM. Requires e2-medium or larger.
- **Secrets in Secret Manager**: Zero plaintext on disk. Startup script fetches secrets into tmpfs (`/run/openclaw-secrets/`). Symlinked as `.env` for Docker Compose.
- **agent-config.yml**: Portable personal config (no secrets). Single source of truth for infrastructure, LLMs, plugins, channels. Generates `terraform.tfvars` and Docker config.
- **Mem0 memory**: Qdrant vector store running as Docker sidecar. Data at `~/.openclaw/memory/qdrant/`. LLM extraction via Anthropic Haiku.
- **Auto-restore on fresh VM**: Startup script checks if `~/.openclaw/openclaw.json` exists; if not, downloads and restores from latest GCS backup.
- **Egress firewall**: Outbound traffic restricted to ports 80, 443, and 53 only. All other egress denied. Cloud NAT provides outbound connectivity (no external IP).
- **Backup encryption**: Backups encrypted client-side with [age](https://age-encryption.org/) before upload to GCS. Encryption key stored in Secret Manager.
- **Auto-restore on fresh VM**: Startup script checks if `~/.openclaw/openclaw.json` exists; if not, downloads and restores from latest GCS backup. Handles both encrypted and unencrypted backups.
- **File ownership**: Container runs as UID 1000 (`node` user). Host files chown'd to 1000:1000.

## Security — CRITICAL Rules

Full security architecture documented in `docs/security.md`.

### Defense layers

1. **Secrets**: Secret Manager → tmpfs at boot → `.env` symlink. Zero plaintext on persistent disk.
2. **Network**: No external IP. IAP-only SSH. Egress firewall allows only ports 80/443/53.
3. **Containers**: Bridge network, `read_only: true`, `cap_drop: ALL`, `no-new-privileges`, resource limits, images pinned by SHA256 digest.
4. **Infrastructure**: Shielded VM + Secure Boot, OS Login (IAM-based SSH), least-privilege service account, `unattended-upgrades`.
5. **Backups**: Client-side `age` encryption before GCS upload. Key in Secret Manager. Bucket versioning enabled.
6. **Supply chain**: GitHub Actions SHA-pinned, Docker images digest-pinned, gitleaks + Trivy in CI, `install.sh` checksum verification with hard-fail.
7. **Pre-commit hooks**: gitleaks (secrets), shellcheck (shell bugs), terraform-fmt, terraform-validate. Install: `pip install pre-commit && pre-commit install`.

### NEVER commit secrets

The `.gitignore` blocks all sensitive files. Before ANY commit, verify:
Expand Down Expand Up @@ -92,11 +112,11 @@ git diff --cached | grep -iE 'sk-|api.key|secret|token.*=.*[a-z0-9]{20}'
All values come from `agent-config.yml` via `config_generate_tfvars()` in `lib/config.sh`. Users never edit `.tfvars` directly.

Required: `project_id`, `backup_bucket_name`.
Optional with defaults: `region`, `zone`, `machine_type`, `disk_size_gb`, `timezone`, `vm_name`, `network`, `backup_retention_days`, `backup_cron_interval_hours`, `secrets_prefix`.
Optional with defaults: `region`, `zone`, `machine_type`, `disk_size_gb`, `timezone`, `vm_name`, `service_account_id`, `network`, `backup_retention_days`, `backup_cron_interval_hours`, `secrets_prefix`.

### startup.sh is a templatefile

Uses Terraform `${}` interpolation for: `${backup_bucket}`, `${timezone}`, `${backup_hours}`, `${secrets_prefix}`. Shell variables use standard `$VAR` syntax (no `$${}` escaping needed since we rewrote it).
Uses Terraform `${}` interpolation for: `${backup_bucket}`, `${timezone}`, `${backup_hours}`, `${secrets_prefix}`, `${backup_retention}`. Shell variables use `$${VAR}` syntax (Terraform renders `$$` as a literal `$`). Inside single-quoted heredocs (`<< 'EOF'`), plain `$VAR` works because Terraform only interpolates `${...}` patterns.

## Provider Interface

Expand All @@ -119,6 +139,8 @@ provider_check_resources() # Warn if VM too small for containers

## Backup Contents

Backups are encrypted with `age` before upload to GCS. The encryption key is stored in Secret Manager. Restore handles both encrypted and unencrypted (legacy) backups.

What the backup script (`openclaw-backup.sh`) saves:

- `openclaw.json` — full config
Expand Down Expand Up @@ -154,6 +176,17 @@ What the backup script (`openclaw-backup.sh`) saves:
4. Add same in `lib/backup.sh` `restore_from_backup()` function
5. Update backup table in README.md

### Modifying Docker Compose security

When adding containers to `docker-compose.override.example.yml`, always include:
- `networks: [openclaw-net]` (bridge network, not host)
- `read_only: true` + `tmpfs` for writable paths
- `cap_drop: [ALL]` (add back only what's strictly required)
- `security_opt: [no-new-privileges:true]`
- `deploy.resources.limits` (CPU + memory)
- Image pinned by version + SHA256 digest
- Health check with interval/timeout/retries

### Modifying .gitignore

Only ADD patterns, never remove them. Use `!filename` to explicitly allow files.
49 changes: 35 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,16 +67,18 @@ bash restore.sh YOUR_BUCKET_NAME
## Architecture

```
+-- GCE VM (e2-medium, IAP only) -------------------+
+-- GCE VM (e2-medium, IAP only, egress 80/443/53) -+
| |
| Docker |
| Docker (bridge network: openclaw-net) |
| +- openclaw-gateway (Claude Sonnet 4) |
| | ports: 127.0.0.1 only |
| | read_only, cap_drop:ALL, no-new-privileges |
| | +- Memory: Mem0 OSS (Qdrant vectors) |
| | +- Audio: Voxtral Mini (Mistral) |
| | +- Browser: Chrome CDP (headless) |
| | +- Channel: WhatsApp |
| +- qdrant (vector store sidecar) |
| +- chrome (headless browser sidecar) |
| +- qdrant (vector store, read_only, cap_drop) |
| +- chrome (headless browser, read_only, cap_drop)|
| |
| /run/openclaw-secrets/ (tmpfs, RAM only) |
| +- secrets.env (fetched from SM) |
Expand All @@ -86,7 +88,7 @@ bash restore.sh YOUR_BUCKET_NAME
| +- browser/chrome-data/ |
| +- workspace/ (SOUL.md, IDENTITY.md, etc.) |
| |
| Cron: backup -> GCS every 6h + on reboot |
| Cron: backup (age-encrypted) -> GCS every 6h |
+----------------------------------------------------+
| ^
v | Secrets at boot
Expand All @@ -100,12 +102,25 @@ bash restore.sh YOUR_BUCKET_NAME

## Security

Defense-in-depth hardening across every layer. See [`docs/security.md`](docs/security.md) for the full architecture and threat model.

- VM has **no external IP** — access only via [IAP tunnel](https://cloud.google.com/iap/docs/using-tcp-forwarding)
- **Egress firewall** — outbound restricted to ports 80, 443, and 53 only (all other egress denied)
- All API keys in **Secret Manager** — fetched into tmpfs (RAM) at boot, never persisted to disk
- Service account follows **least privilege** (logging, monitoring, storage, secretAccessor)
- Docker runs with `no-new-privileges` security option
- Shielded VM with Secure Boot enabled
- Backups exclude secrets (`.env` is a symlink to tmpfs)
- **Container hardening** — bridge network (not host), read-only filesystems, all capabilities dropped (`cap_drop: ALL`), `no-new-privileges`, resource limits, images pinned by SHA256 digest
- Service account follows **least privilege** (logging, monitoring, storage, secretAccessor — no delete)
- Shielded VM with Secure Boot, OS Login (IAM-based SSH)
- **Backups encrypted** with [age](https://age-encryption.org/) before upload to GCS; key in Secret Manager
- **Pre-commit hooks** — gitleaks, shellcheck, terraform-fmt, terraform-validate
- **CI scanning** — gitleaks (secrets), Trivy (container vulnerabilities), all GitHub Actions SHA-pinned

### Pre-commit hooks

Install to catch secrets and lint errors before they reach the repo:

```bash
pip install pre-commit && pre-commit install
```

## Cost Estimate

Expand All @@ -126,7 +141,7 @@ With 1-year VM commitment: ~$26/mo. GCP offers $300 free trial (~12 months free)

## Backup & Restore

Backups happen automatically every 6 hours and on every VM reboot. Last 30 backups retained.
Backups happen automatically every 6 hours and on every VM reboot. Last 30 backups retained. Backups are **encrypted with [age](https://age-encryption.org/)** before upload — the encryption key is stored in Secret Manager, never on disk.

### What's backed up

Expand All @@ -144,7 +159,7 @@ Backups happen automatically every 6 hours and on every VM reboot. Last 30 backu
| `agent-config.yml` | Portable configuration |
| Docker config | docker-compose.yml + override |

> **Note:** API keys are NOT in backups — they're in Secret Manager. WhatsApp session may need re-pairing after restore.
> **Note:** API keys are NOT in backups — they're in Secret Manager. Backups are encrypted at rest with `age`. Restore handles both encrypted and unencrypted (legacy) backups. WhatsApp session may need re-pairing after restore.

## File Structure

Expand All @@ -168,9 +183,15 @@ Backups happen automatically every 6 hours and on every VM reboot. Last 30 backu
│ ├── docker-compose.override.example.yml
│ ├── env.example
│ └── agent-config.example.yml
└── docs/
├── gcp-guide.md # GCP setup, Secret Manager, IAM
└── mem0-setup.md # Mem0 plugin configuration
├── scripts/
│ ├── run-e2e.sh # E2E test runner
│ └── verify-e2e-cleanup.sh # Post-E2E resource cleanup verification
├── docs/
│ ├── gcp-guide.md # GCP setup, Secret Manager, IAM
│ ├── mem0-setup.md # Mem0 plugin configuration
│ └── security.md # Full security architecture
├── .gitleaks.toml # Gitleaks secret scanning config
└── .pre-commit-config.yaml # Pre-commit hooks
```

## Contributing
Expand Down
Loading
Loading