From e91fb40efc1e409aa0df1476faec25f4baa0f620 Mon Sep 17 00:00:00 2001 From: Tristan Escalada <355457+tescalada@users.noreply.github.com> Date: Mon, 21 Sep 2026 20:18:44 -0400 Subject: [PATCH 1/2] Make sparknet-http an opt-in profile; drivers are registered at runtime The webapp no longer takes a driver URL from the environment: meter drivers are registered under Global Settings > Meter Drivers and stored in the database, and the webapp boots normally with none registered. sparknet-http moves to the sparknet profile, ground drops METERING_PROVIDER_URL and its depends_on on the driver, and CLAUDE.md and README.md describe the registration flow and the new profile. The sparknet-http service keeps stdin open, since the binary exits when stdin reaches end-of-file, and its health check probes /v1/healthz. --- CLAUDE.md | 35 +++++++++++++++++++++++------------ README.md | 32 +++++++++++++++++++++----------- docker-compose.yml | 25 ++++++++++++++++--------- 3 files changed, 60 insertions(+), 32 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index c706848..d69eff7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -32,15 +32,19 @@ ground it's called **GroundBolt**. Same code, different deployment target. |---|---| | `thundercloud/` | The webapp (Python/Flask, managed with `uv`). Carries no dev compose file — only `docker-compose.test.yml`, the self-contained test harness its CI runs. | | `symmetricds/` | Docker image build for SymmetricDS, the bidirectional DB-sync engine between the ground and cloud Postgres databases. Configured via env vars (`ENGINE_NAME`, `GROUP_ID`, `REGISTRATION_URL`, …); see its README. | -| `sparknet-http/` | Distribution repo for the SparkNet-Http meter driver: publishes release binaries and builds the container image. The service source and `.proto` contract live elsewhere. | +| `sparknet-http/` | Distribution repo for SparkNet-Http, an optional meter driver: publishes release binaries and builds the container image. The service source and `.proto` contract live elsewhere. Not in the default dev stack; opt in with `--profile sparknet` and register it from the webapp. | | `ansible/` | Provisions a real GroundBolt host: `bootstrap_groundbolt.sh` runs on the target device, resolves config into `/etc/groundbolt/inventory.ini`, and runs `playbook.yml` locally to bring up the compose stack. | ## System shape -- The **ground webapp** manages a micro-grid. It talks to a **metering - provider** over an OpenAPI HTTP+SSE contract (`METERING_PROVIDER_URL`); the - provider drives the gateway radio that reaches the meters. The provider is - `sparknet-http` (port 8080), which can run against a real gateway on a +- The **ground webapp** manages a micro-grid. It reaches meters through a + **meter driver** spoken to over an OpenAPI HTTP+SSE contract; the driver + drives the gateway radio that reaches the meters. Drivers are not + configured by environment variable: they are registered at runtime under + **Global Settings > Meter Drivers** (by base URL) and stored in the + database. With none registered the webapp boots normally and logs + `metering provider is not configured; skipping startup`. `sparknet-http` + (port 8080) is one such driver; it can run against a real gateway on a serial device or with `SPARKNET_HTTP_SIMULATE_GATEWAY=1` for dev. - Each side (ground, cloud) has its own Postgres. A **SymmetricDS node runs next to each database** (`symds-ground`, `symds-cloud`) and the pair syncs @@ -56,21 +60,28 @@ ground it's called **GroundBolt**. Same code, different deployment target. Lives at `docker-compose.yml` in this directory — it's here because it spans component repos: the webapp builds from `./thundercloud`, SymmetricDS from -`./symmetricds`, and `sparknet-http` is pulled as the published image (its -repo distributes prebuilt binaries; there's no source to build). Run compose -commands from the workspace root. Profiles: +`./symmetricds`, and `sparknet-http` (opt-in) is pulled as the published +image (its repo distributes prebuilt binaries; there's no source to build). +Run compose commands from the workspace root. Profiles: -- **default** — the ground stack: `ground` (webapp, http://localhost:8765), - `postgres-ground` (host port 5440), `sparknet-http` (8080, gateway - simulator on), `symds-ground`. +- **default** — the ground stack, with no meter driver: `ground` (webapp, + http://localhost:8765), `postgres-ground` (host port 5440), `symds-ground`. + `ground` has no `depends_on` on any driver and no driver URL variable. - **`--profile cloud`** — adds `cloud` (webapp, http://localhost:5010), `postgres-cloud` (5441), `symds-cloud` (31415). Only with this profile up does ground↔cloud sync run; without it `symds-ground` retries until the cloud side appears. +- **`--profile sparknet`** — adds `sparknet-http` (8080, gateway simulator + on via `SPARKNET_HTTP_SIMULATE_GATEWAY=1`). It is not wired to `ground` + by compose; register it from the ground webapp under **Global Settings > + Meter Drivers > Register driver** with base URL `http://sparknet-http:8080`. + A failing driver image shows only as an unhealthy `sparknet-http`; + `ground` is unaffected. ```sh -docker compose up -d # ground stack +docker compose up -d # ground stack (no driver) docker compose --profile cloud up -d # + cloud stack +docker compose --profile sparknet up -d # + sparknet-http driver; then register it in the webapp docker compose exec ground uv run flask user create # flask CLI ``` diff --git a/README.md b/README.md index 9ea305a..9122d63 100644 --- a/README.md +++ b/README.md @@ -22,27 +22,37 @@ Python package — called **ThunderCloud** when deployed to the cloud and |---|---| | `thundercloud/` | The webapp (Python/Flask). Its own `docker-compose.test.yml` holds only the self-contained test harness that its CI runs. | | `symmetricds/` | Docker image build for SymmetricDS, which syncs the ground and cloud databases bidirectionally. | -| `sparknet-http/` | Distribution repo for the SparkNet-Http meter driver: release binaries plus the container image the ground webapp talks to for meter operations. | +| `sparknet-http/` | Distribution repo for SparkNet-Http, an optional meter driver: release binaries plus the container image. The webapp runs without it; register it from the webapp when you want meter operations against a (simulated) gateway. | | `ansible/` | Provisions a real GroundBolt host from a single bootstrap command run on the device. | -How they fit together: the ground webapp calls the metering provider -(`sparknet-http`, run in gateway-simulation mode in local dev) over an -HTTP+SSE API to reach the meters; each of ground and cloud has its own -Postgres, with a SymmetricDS node beside each keeping the two databases in -sync; the cloud webapp is the same application pointed at the cloud database. +How they fit together: the ground webapp reaches meters through whichever +meter driver is registered with it — drivers are registered at runtime under +**Global Settings > Meter Drivers** and spoken to over an HTTP+SSE API; with +none registered the webapp runs normally, just without meter operations. +`sparknet-http` is one such driver (run in gateway-simulation mode in local +dev). Each of ground and cloud has its own Postgres, with a SymmetricDS node +beside each keeping the two databases in sync; the cloud webapp is the same +application pointed at the cloud database. ## Quickstart The local dev stack is this repo's `docker-compose.yml`: it builds the webapp -from `./thundercloud` and SymmetricDS from `./symmetricds`, and pulls the -published `sparknet-http` image. From the workspace root: +from `./thundercloud` and SymmetricDS from `./symmetricds`; the optional +`sparknet` profile pulls the published `sparknet-http` image. From the +workspace root: ```sh -./clone.sh # clone/update all component repos -docker compose up -d # ground stack: webapp at localhost:8765 -docker compose --profile cloud up -d # + cloud stack: webapp at localhost:5010 +./clone.sh # clone/update all component repos +docker compose up -d # ground stack: webapp at localhost:8765 +docker compose --profile cloud up -d # + cloud stack: webapp at localhost:5010 +docker compose --profile sparknet up -d # + sparknet-http driver (gateway simulator) on :8080 ``` +The default stack has no meter driver. To use one, bring up the `sparknet` +profile, then log in to the ground webapp at http://localhost:8765 and register +`http://sparknet-http:8080` under **Global Settings > Meter Drivers > Register +driver**. The registered driver then becomes selectable on the meter form. + Tests stay self-contained in thundercloud (its CI runs them with no sibling checkouts): `cd thundercloud && docker compose -f docker-compose.test.yml run --rm test`. diff --git a/docker-compose.yml b/docker-compose.yml index 57e46a3..d802319 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -24,7 +24,6 @@ services: - .env environment: DATABASE_URL: postgresql://spark:spark@postgres-ground/ground - METERING_PROVIDER_URL: http://sparknet-http:8080 # The ground row created on first boot derives its UUID from `serial` # via `as_uuid(serial)`. To make this local deployment **be** a # specific site (so meters/customers/etc. created here are anchored @@ -39,8 +38,6 @@ services: depends_on: postgres-ground: condition: service_started - #sparknet-http: - # condition: service_healthy healthcheck: test: ["CMD", "curl", "-f", "http://localhost:5000/health"] interval: 30s @@ -52,14 +49,24 @@ services: - path: ./thundercloud action: rebuild - # Metering provider: the SparkNet controller core with an HTTP+SSE API, - # which the ground webapp reaches via METERING_PROVIDER_URL. Pulled as the - # published image — ./sparknet-http is a distribution repo whose Dockerfile - # wraps prebuilt release binaries, so there is nothing to build from source - # here. Same service name as the production (ansible) deployment. + # Optional meter driver: SparkNet-Http, the SparkNet controller core with + # an HTTP+SSE API. Not part of the default stack — the webapp boots with + # no driver and drivers are registered at runtime. Start it with + # `docker compose --profile sparknet up -d`, then in the ground webapp + # register `http://sparknet-http:8080` under Global Settings > Meter + # Drivers > Register driver. Pulled as the published image — + # ./sparknet-http is a distribution repo whose Dockerfile wraps prebuilt + # release binaries, so there is nothing to build from source here. Same + # service name as the production (ansible) deployment. sparknet-http: container_name: "sparknet-http" image: ghcr.io/earthspark/sparknet-http:latest + profiles: + - sparknet + # The binary serves a legacy line protocol on stdin and exits with status 1 + # when stdin reaches end-of-file, which is what a non-interactive container + # gets. Keeping stdin open holds it running. + stdin_open: true environment: # Truthy SPARKNET_HTTP_SIMULATE_GATEWAY makes the provider run its # in-memory gateway simulator; unset it to talk to a real gateway over @@ -69,7 +76,7 @@ services: ports: - "8080:8080" healthcheck: - test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:8080/healthz >/dev/null 2>&1 || exit 1"] + test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:8080/v1/healthz >/dev/null 2>&1 || exit 1"] interval: 5s timeout: 5s retries: 5 From 185ac991894ef07796da0c0238b6e2a20d6506bc Mon Sep 17 00:00:00 2001 From: Tristan Escalada <355457+tescalada@users.noreply.github.com> Date: Thu, 24 Sep 2026 16:06:50 -0400 Subject: [PATCH 2/2] Run the dev stack on published latest images The webapp, SymmetricDS and meter-driver-emulator services name the ghcr.io/earthspark latest images and keep their build sections, with no pull_policy: a plain up uses the local image or pulls it when absent, up --build builds from the sibling checkouts, and docker compose pull fetches the newest published images. cloud gains the webapp build section. The webapp env file is optional, with its development defaults as ${VAR:-default} entries in the compose file; ground's SERIAL defaults to INIT_GROUND_SERIAL. meter-driver-emulator and sparknet-http run behind the driver-emulator and driver-sparknet profiles. The unused METERING_PROVIDER_URL is removed from ground. repos lists meter-driver-emulator, and clone.sh checks out submodules after each clone or fast-forward and reports a failed submodule step separately. The docs state Docker Compose 2.24.0 as the minimum version. --- CLAUDE.md | 97 ++++++++++++++++++++++----------- README.md | 97 ++++++++++++++++++++++++--------- clone.sh | 27 +++++++--- docker-compose.yml | 130 +++++++++++++++++++++++++++++++++++---------- repos | 12 +++-- 5 files changed, 265 insertions(+), 98 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index d69eff7..c7c9d0d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -32,20 +32,23 @@ ground it's called **GroundBolt**. Same code, different deployment target. |---|---| | `thundercloud/` | The webapp (Python/Flask, managed with `uv`). Carries no dev compose file — only `docker-compose.test.yml`, the self-contained test harness its CI runs. | | `symmetricds/` | Docker image build for SymmetricDS, the bidirectional DB-sync engine between the ground and cloud Postgres databases. Configured via env vars (`ENGINE_NAME`, `GROUP_ID`, `REGISTRATION_URL`, …); see its README. | -| `sparknet-http/` | Distribution repo for SparkNet-Http, an optional meter driver: publishes release binaries and builds the container image. The service source and `.proto` contract live elsewhere. Not in the default dev stack; opt in with `--profile sparknet` and register it from the webapp. | +| `meter-driver-emulator/` | An emulator of the meter-driver HTTP+SSE contract (port 18080). Runs in the local dev stack with `--profile driver-emulator`. Its Docker build copies the `meter-driver-spec` submodule. | +| `sparknet-http/` | Distribution repo for the SparkNet-Http meter driver: publishes release binaries and builds the container image. The service source and `.proto` contract live elsewhere. Runs in the local dev stack with `--profile driver-sparknet`. | | `ansible/` | Provisions a real GroundBolt host: `bootstrap_groundbolt.sh` runs on the target device, resolves config into `/etc/groundbolt/inventory.ini`, and runs `playbook.yml` locally to bring up the compose stack. | ## System shape -- The **ground webapp** manages a micro-grid. It reaches meters through a - **meter driver** spoken to over an OpenAPI HTTP+SSE contract; the driver - drives the gateway radio that reaches the meters. Drivers are not - configured by environment variable: they are registered at runtime under - **Global Settings > Meter Drivers** (by base URL) and stored in the - database. With none registered the webapp boots normally and logs - `metering provider is not configured; skipping startup`. `sparknet-http` - (port 8080) is one such driver; it can run against a real gateway on a - serial device or with `SPARKNET_HTTP_SIMULATE_GATEWAY=1` for dev. +- The **ground webapp** manages a micro-grid. It talks to a **meter + driver** over an OpenAPI HTTP+SSE contract; the driver reaches the meters. + Drivers are registered from the running ground app under Global Settings > + Meter Drivers > Register driver by the base URL of their HTTP service. In + the local dev stack a driver runs only when its profile is chosen: + `--profile driver-emulator` runs `meter-driver-emulator`, registered as + `http://meter-driver-emulator:18080`; `--profile driver-sparknet` runs + `sparknet-http` (port 8080), registered as `http://sparknet-http:8080`, + which can run against a real gateway on a serial device or with + `SPARKNET_HTTP_SIMULATE_GATEWAY=1`. With neither profile the stack runs + without a driver and the webapp boots normally with none registered. - Each side (ground, cloud) has its own Postgres. A **SymmetricDS node runs next to each database** (`symds-ground`, `symds-cloud`) and the pair syncs them bidirectionally. `GROUP_ID` values must match the node-group link @@ -59,30 +62,55 @@ ground it's called **GroundBolt**. Same code, different deployment target. ## Local dev stack Lives at `docker-compose.yml` in this directory — it's here because it spans -component repos: the webapp builds from `./thundercloud`, SymmetricDS from -`./symmetricds`, and `sparknet-http` (opt-in) is pulled as the published -image (its repo distributes prebuilt binaries; there's no source to build). -Run compose commands from the workspace root. Profiles: - -- **default** — the ground stack, with no meter driver: `ground` (webapp, - http://localhost:8765), `postgres-ground` (host port 5440), `symds-ground`. - `ground` has no `depends_on` on any driver and no driver URL variable. +component repos. Requires Docker Compose 2.24.0 or newer, the first release +that accepts `required: false` on `env_file` entries (it is the first built +on compose-go v2.0.0-beta.3, which added it; 2.23.3 used compose-go +v1.20.2). Run compose commands from the workspace root. + +Images: the webapp (`ground`, `cloud`), SymmetricDS (`symds-ground`, +`symds-cloud`) and `meter-driver-emulator` name +`ghcr.io/earthspark/{thundercloud,symmetricds,meter-driver-emulator}:latest`, +which each component's CI publishes from every build of `main` (a release +tag never moves `latest`). Each also keeps a `build` section. No +`pull_policy` is set, so Compose's default, `missing`, applies: a plain `up` +runs whatever image of that name exists locally and pulls only when it is +absent. + +- The first `up` pulls the published `latest`, so it needs only the metarepo. +- `docker compose up --build` builds from the `./thundercloud`, + `./symmetricds` and `./meter-driver-emulator` checkouts, which `clone.sh` + provides (with submodules; the emulator's build needs `meter-driver-spec`). + The build carries the same image name, so later plain `up` runs keep using + it. +- Nothing re-pulls `latest` on its own. `docker compose pull` fetches the + newest published images and replaces a local build of the same name. +- A service whose image is not published yet fails to pull and falls back to + building from its checkout, so on a fresh workspace run `./clone.sh` first + if an image is missing from the registry. +- To build one component and pull the rest: `docker compose up -d --build + ground`, or `docker compose build ground cloud`. `develop.watch` rebuilds + also need that component's checkout from `./clone.sh`. + +Profiles: + +- **default** — the ground stack: `ground` (webapp, http://localhost:8765), + `postgres-ground` (host port 5440), `symds-ground`. No meter driver. +- **`--profile driver-emulator`** — adds `meter-driver-emulator` (18080). + Register it as `http://meter-driver-emulator:18080`. +- **`--profile driver-sparknet`** — adds `sparknet-http` (8080, gateway + simulator on), pulled as the published + `ghcr.io/earthspark/sparknet-http:latest` image (its repo distributes + prebuilt binaries; there's no source to build). Register it as + `http://sparknet-http:8080`. - **`--profile cloud`** — adds `cloud` (webapp, http://localhost:5010), `postgres-cloud` (5441), `symds-cloud` (31415). Only with this profile up does ground↔cloud sync run; without it `symds-ground` retries until the cloud side appears. -- **`--profile sparknet`** — adds `sparknet-http` (8080, gateway simulator - on via `SPARKNET_HTTP_SIMULATE_GATEWAY=1`). It is not wired to `ground` - by compose; register it from the ground webapp under **Global Settings > - Meter Drivers > Register driver** with base URL `http://sparknet-http:8080`. - A failing driver image shows only as an unhealthy `sparknet-http`; - `ground` is unaffected. ```sh -docker compose up -d # ground stack (no driver) -docker compose --profile cloud up -d # + cloud stack -docker compose --profile sparknet up -d # + sparknet-http driver; then register it in the webapp -docker compose exec ground uv run flask user create # flask CLI +docker compose --profile driver-emulator up -d # ground stack + emulator +docker compose --profile driver-emulator --profile cloud up -d # + cloud stack +docker compose exec ground uv run flask user create # flask CLI ``` The **test harness is not here**: it stays self-contained in @@ -97,10 +125,15 @@ docker compose -f docker-compose.test.yml run --rm test uv run pytest # s ``` The webapp's `.env` lives at the workspace root: local (gitignored), seeded -by `clone.sh` from the tracked `.env.example` when missing. It feeds both -the webapp containers (via `env_file:`) and compose interpolation, so -overrides (a specific site serial, real cloud SymmetricDS endpoint) also go -in it; the comments in `docker-compose.yml` document the variables. +by `clone.sh` from the tracked `.env.example` when missing, and optional — +`env_file` sets `required: false`. The webapp's development defaults live in +the `x-webapp-environment` block of `docker-compose.yml` as `${VAR:-default}` +entries equal to the `.env.example` values. For each variable, a value +exported in the shell that runs compose wins, then `.env`, then the compose +default. The `.env` feeds both the webapp containers (via +`env_file:`) and compose interpolation, so overrides (a specific site serial, +real cloud SymmetricDS endpoint) also go in it; the comments in +`docker-compose.yml` document the variables. Non-Docker local dev (uv, flask CLI, database reset, demo data) is covered in `thundercloud/README.md`. diff --git a/README.md b/README.md index 9122d63..0892f8e 100644 --- a/README.md +++ b/README.md @@ -22,36 +22,72 @@ Python package — called **ThunderCloud** when deployed to the cloud and |---|---| | `thundercloud/` | The webapp (Python/Flask). Its own `docker-compose.test.yml` holds only the self-contained test harness that its CI runs. | | `symmetricds/` | Docker image build for SymmetricDS, which syncs the ground and cloud databases bidirectionally. | -| `sparknet-http/` | Distribution repo for SparkNet-Http, an optional meter driver: release binaries plus the container image. The webapp runs without it; register it from the webapp when you want meter operations against a (simulated) gateway. | +| `meter-driver-emulator/` | An emulator of the meter-driver HTTP+SSE contract. Runs in the local dev stack with `--profile driver-emulator`. | +| `sparknet-http/` | Distribution repo for the SparkNet-Http meter driver: release binaries plus its container image. Runs in the local dev stack with `--profile driver-sparknet`. | | `ansible/` | Provisions a real GroundBolt host from a single bootstrap command run on the device. | -How they fit together: the ground webapp reaches meters through whichever -meter driver is registered with it — drivers are registered at runtime under -**Global Settings > Meter Drivers** and spoken to over an HTTP+SSE API; with -none registered the webapp runs normally, just without meter operations. -`sparknet-http` is one such driver (run in gateway-simulation mode in local -dev). Each of ground and cloud has its own Postgres, with a SymmetricDS node -beside each keeping the two databases in sync; the cloud webapp is the same -application pointed at the cloud database. +How they fit together: the ground webapp calls a meter driver over an +HTTP+SSE API to reach the meters (in local dev, `meter-driver-emulator` with +`--profile driver-emulator`, or `sparknet-http` in gateway-simulation mode with +`--profile driver-sparknet`); each of ground and cloud has its own Postgres, with a +SymmetricDS node beside each keeping the two databases in sync; the cloud +webapp is the same application pointed at the cloud database. ## Quickstart -The local dev stack is this repo's `docker-compose.yml`: it builds the webapp -from `./thundercloud` and SymmetricDS from `./symmetricds`; the optional -`sparknet` profile pulls the published `sparknet-http` image. From the -workspace root: +Requires Docker Compose 2.24.0 or newer, the first release that accepts +`required: false` on `env_file` entries. + +The local dev stack is this repo's `docker-compose.yml`. From the workspace +root: + +```sh +docker compose --profile driver-emulator up -d # ground stack + emulator driver: webapp at localhost:8765 +docker compose --profile driver-emulator --profile cloud up -d # + cloud stack: webapp at localhost:5010 +``` + +Choose a meter driver with a profile, then register it in the ground app +under **Global Settings > Meter Drivers > Register driver**: + +| Profile | Driver | Base URL to register | +|---|---|---| +| `--profile driver-emulator` | `meter-driver-emulator` | `http://meter-driver-emulator:18080` | +| `--profile driver-sparknet` | `sparknet-http` (gateway simulator on) | `http://sparknet-http:8080` | + +With neither profile the stack runs without a meter driver; the webapp boots +normally with none registered. + +### Which images run + +The webapp, SymmetricDS and meter-driver-emulator services name the `latest` +images that each component's CI publishes to GHCR from every build of `main`, +and each also has a `build` section pointing at its checkout. No +`pull_policy` is set, so Compose uses its default, `missing`: a plain `up` +runs whatever image of that name exists locally and pulls only when there is +none. + +- The first `up` pulls the published `latest` images, so it needs only this + repo checked out. +- `docker compose up -d --build` (after `./clone.sh`) builds the images from + `./thundercloud`, `./symmetricds` and `./meter-driver-emulator`. The build + carries the same image name, so later plain `up` runs keep using your local + build. +- `up` never re-pulls `latest` on its own. `docker compose pull` fetches the + newest published images and replaces a local build of the same name. +- A service whose image is not published yet fails to pull and is built from + its checkout instead. On a fresh workspace, run `./clone.sh` first if an + image is missing from the registry. + +To build one component and pull the rest, name its services: ```sh -./clone.sh # clone/update all component repos -docker compose up -d # ground stack: webapp at localhost:8765 -docker compose --profile cloud up -d # + cloud stack: webapp at localhost:5010 -docker compose --profile sparknet up -d # + sparknet-http driver (gateway simulator) on :8080 +./clone.sh # clone/update all component repos +docker compose up -d --build ground # build the webapp, pull everything else +docker compose build ground cloud # or build the webapp image without starting it ``` -The default stack has no meter driver. To use one, bring up the `sparknet` -profile, then log in to the ground webapp at http://localhost:8765 and register -`http://sparknet-http:8080` under **Global Settings > Meter Drivers > Register -driver**. The registered driver then becomes selectable on the meter form. +`develop.watch` rebuilds (`docker compose watch`) also build from the +component's checkout, so they need `./clone.sh` too. Tests stay self-contained in thundercloud (its CI runs them with no sibling checkouts): @@ -68,14 +104,23 @@ details. This clones (or updates) every repo you can access. Each reachable repo is cloned on `main`; if it's already present, it's fast-forwarded instead of -re-cloned, so the script is safe to run repeatedly. Repos you can't access -(private or unreachable with your current SSH key) are **skipped** with a -note — the run continues and exits successfully. A summary at the end reports -how many were cloned, updated, and skipped. +re-cloned, so the script is safe to run repeatedly. After either step the +script checks out the repo's submodules (`git submodule update --init +--recursive`); the meter-driver-emulator build needs its `meter-driver-spec` +submodule. Repos you can't access (private or unreachable with your current +SSH key) are **skipped** with a note — the run continues and exits +successfully. A repo whose submodule step fails is still cloned or updated +and is reported as `(; submodule update failed)`. A summary at the +end reports how many were cloned, updated, skipped, and had a failed +submodule step. The script also seeds a local `.env` (the webapp's env file, read by `docker-compose.yml`) from the tracked `.env.example` if you don't have one -yet; an existing `.env` is never touched. +yet; an existing `.env` is never touched. The `.env` is optional: without it +the webapp runs on the development defaults in `docker-compose.yml`, which +equal the `.env.example` values. For each variable, a value exported in the +shell that runs `docker compose` wins, then the value in `.env`, then the +default in `docker-compose.yml`. ## Workflow diff --git a/clone.sh b/clone.sh index 24ca4b9..b775bbe 100755 --- a/clone.sh +++ b/clone.sh @@ -2,9 +2,12 @@ # # Clone or update every component repo listed in the `repos` manifest. # -# Each repo you can reach is cloned (or fast-forwarded if already present). -# Repos you can't access (private / unreachable) are skipped with a note; -# the run continues and exits 0. +# Each repo you can reach is cloned (or fast-forwarded if already present), +# and its submodules are then checked out with +# `git submodule update --init --recursive`. Repos you can't access +# (private / unreachable) are skipped with a note; a repo whose submodule +# step fails is still counted as cloned or updated and is reported with +# "submodule update failed". The run continues and exits 0. set -u @@ -22,6 +25,7 @@ cd "$script_dir" || exit 1 cloned=0 updated=0 skipped=0 +submodule_failed=0 while IFS= read -r line; do # Skip blank lines. @@ -59,22 +63,33 @@ while IFS= read -r line; do if git -C "$name" fetch origin "$branch" >/dev/null 2>&1 \ && git -C "$name" checkout "$branch" >/dev/null 2>&1 \ && git -C "$name" merge --ff-only "origin/$branch" >/dev/null 2>&1; then - echo "updated: $name ($branch)" + action="updated" updated=$((updated + 1)) else echo "skip: $name (no access or unreachable)" skipped=$((skipped + 1)) + continue fi else # Not present: clone the branch. if git clone --branch "$branch" "$url" "$name" >/dev/null 2>&1; then - echo "cloned: $name ($branch)" + action="cloned" cloned=$((cloned + 1)) else echo "skip: $name (no access or unreachable)" skipped=$((skipped + 1)) + continue fi fi + + # Check out the submodules at the commits the branch records. A failure + # here leaves the repo cloned or updated, and is reported on its own. + if git -C "$name" submodule update --init --recursive >/dev/null 2>&1; then + echo "$action: $name ($branch)" + else + echo "$action: $name ($branch; submodule update failed)" + submodule_failed=$((submodule_failed + 1)) + fi done < "$manifest" # Seed the webapp env file from the tracked example on first run. Never @@ -85,5 +100,5 @@ if [ ! -f "$script_dir/.env" ]; then fi echo "" -echo "summary: cloned=$cloned updated=$updated skipped=$skipped" +echo "summary: cloned=$cloned updated=$updated skipped=$skipped submodule_failed=$submodule_failed" exit 0 diff --git a/docker-compose.yml b/docker-compose.yml index d802319..6e033ee 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,40 +1,99 @@ # Full local dev stack for the GroundBolt/ThunderCloud system. # -# This file lives in the workspace metarepo because it spans component repos: -# it builds the webapp from ./thundercloud and SymmetricDS from ./symmetricds, -# relying on the sibling layout that ./clone.sh guarantees. Run compose -# commands from the workspace root. +# This file lives in the workspace metarepo because it spans component repos. +# Run compose commands from the workspace root. +# +# Images: the webapp, SymmetricDS and the meter-driver-emulator name the +# `latest` image that each component's CI publishes to GHCR from every build +# of its `main` branch, and each keeps a `build` section. No `pull_policy` is +# set, so Compose uses its default, "missing": `docker compose up` runs +# whatever image of that name exists locally and pulls only when none does. +# - On the first run `up` pulls the published `latest`, so it needs only this +# metarepo checked out. +# - `docker compose up --build` builds the images from ./thundercloud, +# ./symmetricds and ./meter-driver-emulator, the sibling checkouts that +# ./clone.sh provides. The local build carries the same image name, so +# later plain `up` runs keep using it. +# - `up` never re-pulls `latest` on its own. `docker compose pull` fetches the +# newest published images and replaces a local build of the same name. +# - A service whose image is not published yet fails to pull and is built +# from its checkout, so run ./clone.sh first in that case. +# - To build one component and pull the rest: +# `docker compose up -d --build ground` (or `docker compose build ground +# cloud`). `develop.watch` rebuilds also need that component's checkout. +# +# Meter drivers: choose one with `--profile driver-emulator` (meter-driver-emulator) +# or `--profile driver-sparknet` (sparknet-http), then register it in the ground +# webapp under Global Settings > Meter Drivers at +# http://meter-driver-emulator:18080 or http://sparknet-http:8080. With +# neither profile the stack runs without a meter driver; the webapp boots +# normally with none registered. +# +# Webapp settings: `ground` and `cloud` take their development defaults from +# the `x-webapp-environment` block below. A `.env` at the workspace root is +# optional. Precedence for each variable: a value exported in the shell that +# runs compose wins, then the project `.env`, then the default here. # # The self-contained test harness (postgres-test + test runner) stays in # thundercloud/docker-compose.test.yml — thundercloud's CI runs it with no # sibling checkouts. Run tests from ./thundercloud. +# Development defaults for the webapp, shared by `ground` and `cloud`. Each +# value is `${VAR:-default}`: Compose substitutes VAR from the calling shell +# when it is exported there, else from the project `.env` when it is set +# there, else the default. A literal value here would +# hide the `.env` value, because `environment` takes precedence over +# `env_file`. +x-webapp-environment: &webapp-environment + SM_API_ENDPOINT: "${SM_API_ENDPOINT:-http://localhost:8080/api/v0}" + SM_CURRENCY: "${SM_CURRENCY:-USD}" + SM_DEFAULT_PHONE_COUNTRY_CODE: "${SM_DEFAULT_PHONE_COUNTRY_CODE:-1}" + SM_SERIAL: "${SM_SERIAL:-1234}" + SM_SPARKCLOUD_API_KEY: "${SM_SPARKCLOUD_API_KEY:-test}" + SM_HEROKU: "${SM_HEROKU:-false}" + SM_OFFLINE: "${SM_OFFLINE:-true}" + SM_DEMO_METERS: "${SM_DEMO_METERS:-1234}" + INIT_CREATE_DEFAULTS: "${INIT_CREATE_DEFAULTS:-true}" + SM_INIT_ADMIN_USERNAME: "${SM_INIT_ADMIN_USERNAME:-admin}" + INIT_ADMIN_USERNAME: "${INIT_ADMIN_USERNAME:-admin}" + INIT_ADMIN_PASSWORD: "${INIT_ADMIN_PASSWORD:-password}" + INIT_ADMIN_EMAIL: "${INIT_ADMIN_EMAIL:-admin@sparkmeter.io}" + INIT_GROUND_NAME: "${INIT_GROUND_NAME:-Ground}" + INIT_GROUND_SERIAL: "${INIT_GROUND_SERIAL:-1234}" + HEARTBEAT_PERIOD: "${HEARTBEAT_PERIOD:-1}" + SM_SECURITY_PASSWORD_SALT: "${SM_SECURITY_PASSWORD_SALT:-dev-salt-not-secret}" + SM_SECRET_KEY: "${SM_SECRET_KEY:-dev-not-secret}" + services: ground: container_name: "ground" - build: + build: &webapp-build context: './thundercloud' dockerfile: deploy/Dockerfile target: production - image: sparkmeter + image: ghcr.io/earthspark/thundercloud:latest command: /app/run/webapp ports: - "8765:5000" env_file: - - .env + - path: .env + required: false environment: + <<: *webapp-environment DATABASE_URL: postgresql://spark:spark@postgres-ground/ground # The ground row created on first boot derives its UUID from `serial` - # via `as_uuid(serial)`. To make this local deployment **be** a - # specific site (so meters/customers/etc. created here are anchored - # to the same ground row the cloud has), set these in your `.env` - # alongside the SYMDS_* values: + # via `as_uuid(serial)`. INIT_GROUND_SERIAL and INIT_GROUND_NAME come + # from `x-webapp-environment` (defaults `1234` and `Ground`), and + # SERIAL, which drives the default-ground lookup, defaults to the same + # serial. To make this local deployment **be** a specific site (so + # meters/customers/etc. created here are anchored to the same ground + # row the cloud has), override these in your `.env` alongside the + # SYMDS_* values: # INIT_GROUND_SERIAL= # INIT_GROUND_NAME= # GROUND_SERIAL= # matches INIT_GROUND_SERIAL; drives default-ground lookup - INIT_GROUND_SERIAL: ${INIT_GROUND_SERIAL:-} - INIT_GROUND_NAME: ${INIT_GROUND_NAME:-} - SERIAL: ${GROUND_SERIAL:-} + # Without GROUND_SERIAL, SERIAL follows INIT_GROUND_SERIAL. + SERIAL: ${GROUND_SERIAL:-${INIT_GROUND_SERIAL:-1234}} depends_on: postgres-ground: condition: service_started @@ -49,20 +108,15 @@ services: - path: ./thundercloud action: rebuild - # Optional meter driver: SparkNet-Http, the SparkNet controller core with - # an HTTP+SSE API. Not part of the default stack — the webapp boots with - # no driver and drivers are registered at runtime. Start it with - # `docker compose --profile sparknet up -d`, then in the ground webapp - # register `http://sparknet-http:8080` under Global Settings > Meter - # Drivers > Register driver. Pulled as the published image — - # ./sparknet-http is a distribution repo whose Dockerfile wraps prebuilt - # release binaries, so there is nothing to build from source here. Same - # service name as the production (ansible) deployment. + # Metering provider: the SparkNet controller core with an HTTP+SSE API. + # Register it in the ground webapp under Global Settings > Meter Drivers + # with the base URL http://sparknet-http:8080. Pulled as the + # published image — ./sparknet-http is a distribution repo whose Dockerfile + # wraps prebuilt release binaries, so there is nothing to build from source + # here. Same service name as the production (ansible) deployment. sparknet-http: container_name: "sparknet-http" image: ghcr.io/earthspark/sparknet-http:latest - profiles: - - sparknet # The binary serves a legacy line protocol on stdin and exits with status 1 # when stdin reaches end-of-file, which is what a non-interactive container # gets. Keeping stdin open holds it running. @@ -81,6 +135,21 @@ services: timeout: 5s retries: 5 start_period: 5s + profiles: ["driver-sparknet"] + + # Meter driver: an emulator of the meter-driver HTTP+SSE contract, started + # with `--profile driver-emulator`. Register + # it in the ground webapp under Global Settings > Meter Drivers > Register + # driver with the base URL http://meter-driver-emulator:18080. Its image + # carries its own HEALTHCHECK on /v1/healthz. Building it needs the + # meter-driver-spec submodule, which ./clone.sh checks out. + meter-driver-emulator: + container_name: "meter-driver-emulator" + build: './meter-driver-emulator' + image: ghcr.io/earthspark/meter-driver-emulator:latest + ports: + - "18080:18080" + profiles: ["driver-emulator"] # POSSIBLE IMPROVEMENTS: # - Healthcheck uses psql which requires a full DB connection; could use @@ -111,7 +180,7 @@ services: container_name: "symds-ground" build: context: './symmetricds' - image: symmetricds + image: ghcr.io/earthspark/symmetricds:latest environment: DATABASE_URL: postgresql://spark:spark@postgres-ground/ground ENGINE_NAME: ground-engine @@ -159,13 +228,16 @@ services: cloud: container_name: "cloud" - image: sparkmeter + build: *webapp-build + image: ghcr.io/earthspark/thundercloud:latest command: /app/run/webapp ports: - "5010:5000" env_file: - - .env + - path: .env + required: false environment: + <<: *webapp-environment DATABASE_URL: postgresql://spark:spark@postgres-cloud/cloud SM_HEROKU: "true" depends_on: @@ -188,7 +260,7 @@ services: container_name: "symds-cloud" build: context: './symmetricds' - image: symmetricds + image: ghcr.io/earthspark/symmetricds:latest environment: DATABASE_URL: postgresql://spark:spark@postgres-cloud/cloud PROTOCOL: http diff --git a/repos b/repos index 67986fc..bf67a84 100644 --- a/repos +++ b/repos @@ -10,9 +10,11 @@ # # To add a repo: append a line with its name and clone URL (and a branch # if it lives somewhere other than main). To remove one: delete its line. -# Then run ./clone.sh — it clones new entries and fast-forwards existing ones. +# Then run ./clone.sh — it clones new entries and fast-forwards existing ones, +# and checks out each repo's submodules at the commits its branch records. -ansible git@github.com:EarthSpark/ansible.git -sparknet-http git@github.com:EarthSpark/sparknet-http.git -symmetricds git@github.com:EarthSpark/symmetricds.git -thundercloud git@github.com:EarthSpark/thundercloud.git +ansible git@github.com:EarthSpark/ansible.git +meter-driver-emulator git@github.com:EarthSpark/meter-driver-emulator.git +sparknet-http git@github.com:EarthSpark/sparknet-http.git +symmetricds git@github.com:EarthSpark/symmetricds.git +thundercloud git@github.com:EarthSpark/thundercloud.git