Skip to content

Run the dev stack on published images, with the meter driver chosen by profile #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tescalada merged 2 commits into from
Sep 24, 2026
Merged

Run the dev stack on published images, with the meter driver chosen by profile #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e91fb40
Make sparknet-http an opt-in profile; drivers are registered at runtime
tescalada Sep 22, 2026
185ac99
Run the dev stack on published latest images
tescalada Sep 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 63 additions & 19 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,16 +32,23 @@ ground it's called **GroundBolt**. Same code, different deployment target.
|---|---|
| `thundercloud/` | The webapp (Python/Flask, managed with `uv`). Carries no dev compose file — only `docker-compose.test.yml`, the self-contained test harness its CI runs. |
| `symmetricds/` | Docker image build for SymmetricDS, the bidirectional DB-sync engine between the ground and cloud Postgres databases. Configured via env vars (`ENGINE_NAME`, `GROUP_ID`, `REGISTRATION_URL`, …); see its README. |
| `sparknet-http/` | Distribution repo for the SparkNet-Http meter driver: publishes release binaries and builds the container image. The service source and `.proto` contract live elsewhere. |
| `meter-driver-emulator/` | An emulator of the meter-driver HTTP+SSE contract (port 18080). Runs in the local dev stack with `--profile driver-emulator`. Its Docker build copies the `meter-driver-spec` submodule. |
| `sparknet-http/` | Distribution repo for the SparkNet-Http meter driver: publishes release binaries and builds the container image. The service source and `.proto` contract live elsewhere. Runs in the local dev stack with `--profile driver-sparknet`. |
| `ansible/` | Provisions a real GroundBolt host: `bootstrap_groundbolt.sh` runs on the target device, resolves config into `/etc/groundbolt/inventory.ini`, and runs `playbook.yml` locally to bring up the compose stack. |

## System shape

- The **ground webapp** manages a micro-grid. It talks to a **metering
provider** over an OpenAPI HTTP+SSE contract (`METERING_PROVIDER_URL`); the
provider drives the gateway radio that reaches the meters. The provider is
`sparknet-http` (port 8080), which can run against a real gateway on a
serial device or with `SPARKNET_HTTP_SIMULATE_GATEWAY=1` for dev.
- The **ground webapp** manages a micro-grid. It talks to a **meter
driver** over an OpenAPI HTTP+SSE contract; the driver reaches the meters.
Drivers are registered from the running ground app under Global Settings >
Meter Drivers > Register driver by the base URL of their HTTP service. In
the local dev stack a driver runs only when its profile is chosen:
`--profile driver-emulator` runs `meter-driver-emulator`, registered as
`http://meter-driver-emulator:18080`; `--profile driver-sparknet` runs
`sparknet-http` (port 8080), registered as `http://sparknet-http:8080`,
which can run against a real gateway on a serial device or with
`SPARKNET_HTTP_SIMULATE_GATEWAY=1`. With neither profile the stack runs
without a driver and the webapp boots normally with none registered.
- Each side (ground, cloud) has its own Postgres. A **SymmetricDS node runs
next to each database** (`symds-ground`, `symds-cloud`) and the pair syncs
them bidirectionally. `GROUP_ID` values must match the node-group link
Expand All @@ -55,23 +62,55 @@ ground it's called **GroundBolt**. Same code, different deployment target.
## Local dev stack

Lives at `docker-compose.yml` in this directory — it's here because it spans
component repos: the webapp builds from `./thundercloud`, SymmetricDS from
`./symmetricds`, and `sparknet-http` is pulled as the published image (its
repo distributes prebuilt binaries; there's no source to build). Run compose
commands from the workspace root. Profiles:
component repos. Requires Docker Compose 2.24.0 or newer, the first release
that accepts `required: false` on `env_file` entries (it is the first built
on compose-go v2.0.0-beta.3, which added it; 2.23.3 used compose-go
v1.20.2). Run compose commands from the workspace root.

Images: the webapp (`ground`, `cloud`), SymmetricDS (`symds-ground`,
`symds-cloud`) and `meter-driver-emulator` name
`ghcr.io/earthspark/{thundercloud,symmetricds,meter-driver-emulator}:latest`,
which each component's CI publishes from every build of `main` (a release
tag never moves `latest`). Each also keeps a `build` section. No
`pull_policy` is set, so Compose's default, `missing`, applies: a plain `up`
runs whatever image of that name exists locally and pulls only when it is
absent.

- The first `up` pulls the published `latest`, so it needs only the metarepo.
- `docker compose up --build` builds from the `./thundercloud`,
`./symmetricds` and `./meter-driver-emulator` checkouts, which `clone.sh`
provides (with submodules; the emulator's build needs `meter-driver-spec`).
The build carries the same image name, so later plain `up` runs keep using
it.
- Nothing re-pulls `latest` on its own. `docker compose pull` fetches the
newest published images and replaces a local build of the same name.
- A service whose image is not published yet fails to pull and falls back to
building from its checkout, so on a fresh workspace run `./clone.sh` first
if an image is missing from the registry.
- To build one component and pull the rest: `docker compose up -d --build
ground`, or `docker compose build ground cloud`. `develop.watch` rebuilds
also need that component's checkout from `./clone.sh`.

Profiles:

- **default** — the ground stack: `ground` (webapp, http://localhost:8765),
`postgres-ground` (host port 5440), `sparknet-http` (8080, gateway
simulator on), `symds-ground`.
`postgres-ground` (host port 5440), `symds-ground`. No meter driver.
- **`--profile driver-emulator`** — adds `meter-driver-emulator` (18080).
Register it as `http://meter-driver-emulator:18080`.
- **`--profile driver-sparknet`** — adds `sparknet-http` (8080, gateway
simulator on), pulled as the published
`ghcr.io/earthspark/sparknet-http:latest` image (its repo distributes
prebuilt binaries; there's no source to build). Register it as
`http://sparknet-http:8080`.
- **`--profile cloud`** — adds `cloud` (webapp, http://localhost:5010),
`postgres-cloud` (5441), `symds-cloud` (31415). Only with this profile up
does ground↔cloud sync run; without it `symds-ground` retries until the
cloud side appears.

```sh
docker compose up -d # ground stack
docker compose --profile cloud up -d # + cloud stack
docker compose exec ground uv run flask user create # flask CLI
docker compose --profile driver-emulator up -d # ground stack + emulator
docker compose --profile driver-emulator --profile cloud up -d # + cloud stack
docker compose exec ground uv run flask user create # flask CLI
```

The **test harness is not here**: it stays self-contained in
Expand All @@ -86,10 +125,15 @@ docker compose -f docker-compose.test.yml run --rm test uv run pytest <path> # s
```

The webapp's `.env` lives at the workspace root: local (gitignored), seeded
by `clone.sh` from the tracked `.env.example` when missing. It feeds both
the webapp containers (via `env_file:`) and compose interpolation, so
overrides (a specific site serial, real cloud SymmetricDS endpoint) also go
in it; the comments in `docker-compose.yml` document the variables.
by `clone.sh` from the tracked `.env.example` when missing, and optional —
`env_file` sets `required: false`. The webapp's development defaults live in
the `x-webapp-environment` block of `docker-compose.yml` as `${VAR:-default}`
entries equal to the `.env.example` values. For each variable, a value
exported in the shell that runs compose wins, then `.env`, then the compose
default. The `.env` feeds both the webapp containers (via
`env_file:`) and compose interpolation, so overrides (a specific site serial,
real cloud SymmetricDS endpoint) also go in it; the comments in
`docker-compose.yml` document the variables.

Non-Docker local dev (uv, flask CLI, database reset, demo data) is covered in
`thundercloud/README.md`.
Expand Down
87 changes: 71 additions & 16 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,27 +22,73 @@ Python package — called **ThunderCloud** when deployed to the cloud and
|---|---|
| `thundercloud/` | The webapp (Python/Flask). Its own `docker-compose.test.yml` holds only the self-contained test harness that its CI runs. |
| `symmetricds/` | Docker image build for SymmetricDS, which syncs the ground and cloud databases bidirectionally. |
| `sparknet-http/` | Distribution repo for the SparkNet-Http meter driver: release binaries plus the container image the ground webapp talks to for meter operations. |
| `meter-driver-emulator/` | An emulator of the meter-driver HTTP+SSE contract. Runs in the local dev stack with `--profile driver-emulator`. |
| `sparknet-http/` | Distribution repo for the SparkNet-Http meter driver: release binaries plus its container image. Runs in the local dev stack with `--profile driver-sparknet`. |
| `ansible/` | Provisions a real GroundBolt host from a single bootstrap command run on the device. |

How they fit together: the ground webapp calls the metering provider
(`sparknet-http`, run in gateway-simulation mode in local dev) over an
HTTP+SSE API to reach the meters; each of ground and cloud has its own
Postgres, with a SymmetricDS node beside each keeping the two databases in
sync; the cloud webapp is the same application pointed at the cloud database.
How they fit together: the ground webapp calls a meter driver over an
HTTP+SSE API to reach the meters (in local dev, `meter-driver-emulator` with
`--profile driver-emulator`, or `sparknet-http` in gateway-simulation mode with
`--profile driver-sparknet`); each of ground and cloud has its own Postgres, with a
SymmetricDS node beside each keeping the two databases in sync; the cloud
webapp is the same application pointed at the cloud database.

## Quickstart

The local dev stack is this repo's `docker-compose.yml`: it builds the webapp
from `./thundercloud` and SymmetricDS from `./symmetricds`, and pulls the
published `sparknet-http` image. From the workspace root:
Requires Docker Compose 2.24.0 or newer, the first release that accepts
`required: false` on `env_file` entries.

The local dev stack is this repo's `docker-compose.yml`. From the workspace
root:

```sh
docker compose --profile driver-emulator up -d # ground stack + emulator driver: webapp at localhost:8765
docker compose --profile driver-emulator --profile cloud up -d # + cloud stack: webapp at localhost:5010
```

Choose a meter driver with a profile, then register it in the ground app
under **Global Settings > Meter Drivers > Register driver**:

| Profile | Driver | Base URL to register |
|---|---|---|
| `--profile driver-emulator` | `meter-driver-emulator` | `http://meter-driver-emulator:18080` |
| `--profile driver-sparknet` | `sparknet-http` (gateway simulator on) | `http://sparknet-http:8080` |

With neither profile the stack runs without a meter driver; the webapp boots
normally with none registered.

### Which images run

The webapp, SymmetricDS and meter-driver-emulator services name the `latest`
images that each component's CI publishes to GHCR from every build of `main`,
and each also has a `build` section pointing at its checkout. No
`pull_policy` is set, so Compose uses its default, `missing`: a plain `up`
runs whatever image of that name exists locally and pulls only when there is
none.

- The first `up` pulls the published `latest` images, so it needs only this
repo checked out.
- `docker compose up -d --build` (after `./clone.sh`) builds the images from
`./thundercloud`, `./symmetricds` and `./meter-driver-emulator`. The build
carries the same image name, so later plain `up` runs keep using your local
build.
- `up` never re-pulls `latest` on its own. `docker compose pull` fetches the
newest published images and replaces a local build of the same name.
- A service whose image is not published yet fails to pull and is built from
its checkout instead. On a fresh workspace, run `./clone.sh` first if an
image is missing from the registry.

To build one component and pull the rest, name its services:

```sh
./clone.sh # clone/update all component repos
docker compose up -d # ground stack: webapp at localhost:8765
docker compose --profile cloud up -d # + cloud stack: webapp at localhost:5010
docker compose up -d --build ground # build the webapp, pull everything else
docker compose build ground cloud # or build the webapp image without starting it
```

`develop.watch` rebuilds (`docker compose watch`) also build from the
component's checkout, so they need `./clone.sh` too.

Tests stay self-contained in thundercloud (its CI runs them with no sibling
checkouts):
`cd thundercloud && docker compose -f docker-compose.test.yml run --rm test`.
Expand All @@ -58,14 +104,23 @@ details.

This clones (or updates) every repo you can access. Each reachable repo is
cloned on `main`; if it's already present, it's fast-forwarded instead of
re-cloned, so the script is safe to run repeatedly. Repos you can't access
(private or unreachable with your current SSH key) are **skipped** with a
note — the run continues and exits successfully. A summary at the end reports
how many were cloned, updated, and skipped.
re-cloned, so the script is safe to run repeatedly. After either step the
script checks out the repo's submodules (`git submodule update --init
--recursive`); the meter-driver-emulator build needs its `meter-driver-spec`
submodule. Repos you can't access (private or unreachable with your current
SSH key) are **skipped** with a note — the run continues and exits
successfully. A repo whose submodule step fails is still cloned or updated
and is reported as `(<branch>; submodule update failed)`. A summary at the
end reports how many were cloned, updated, skipped, and had a failed
submodule step.

The script also seeds a local `.env` (the webapp's env file, read by
`docker-compose.yml`) from the tracked `.env.example` if you don't have one
yet; an existing `.env` is never touched.
yet; an existing `.env` is never touched. The `.env` is optional: without it
the webapp runs on the development defaults in `docker-compose.yml`, which
equal the `.env.example` values. For each variable, a value exported in the
shell that runs `docker compose` wins, then the value in `.env`, then the
default in `docker-compose.yml`.

## Workflow

Expand Down
27 changes: 21 additions & 6 deletions clone.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,12 @@
#
# Clone or update every component repo listed in the `repos` manifest.
#
# Each repo you can reach is cloned (or fast-forwarded if already present).
# Repos you can't access (private / unreachable) are skipped with a note;
# the run continues and exits 0.
# Each repo you can reach is cloned (or fast-forwarded if already present),
# and its submodules are then checked out with
# `git submodule update --init --recursive`. Repos you can't access
# (private / unreachable) are skipped with a note; a repo whose submodule
# step fails is still counted as cloned or updated and is reported with
# "submodule update failed". The run continues and exits 0.

set -u

Expand All @@ -22,6 +25,7 @@ cd "$script_dir" || exit 1
cloned=0
updated=0
skipped=0
submodule_failed=0

while IFS= read -r line; do
# Skip blank lines.
Expand Down Expand Up @@ -59,22 +63,33 @@ while IFS= read -r line; do
if git -C "$name" fetch origin "$branch" >/dev/null 2>&1 \
&& git -C "$name" checkout "$branch" >/dev/null 2>&1 \
&& git -C "$name" merge --ff-only "origin/$branch" >/dev/null 2>&1; then
echo "updated: $name ($branch)"
action="updated"
updated=$((updated + 1))
else
echo "skip: $name (no access or unreachable)"
skipped=$((skipped + 1))
continue
fi
else
# Not present: clone the branch.
if git clone --branch "$branch" "$url" "$name" >/dev/null 2>&1; then
echo "cloned: $name ($branch)"
action="cloned"
cloned=$((cloned + 1))
else
echo "skip: $name (no access or unreachable)"
skipped=$((skipped + 1))
continue
fi
fi

# Check out the submodules at the commits the branch records. A failure
# here leaves the repo cloned or updated, and is reported on its own.
if git -C "$name" submodule update --init --recursive >/dev/null 2>&1; then
echo "$action: $name ($branch)"
else
echo "$action: $name ($branch; submodule update failed)"
submodule_failed=$((submodule_failed + 1))
fi
done < "$manifest"

# Seed the webapp env file from the tracked example on first run. Never
Expand All @@ -85,5 +100,5 @@ if [ ! -f "$script_dir/.env" ]; then
fi

echo ""
echo "summary: cloned=$cloned updated=$updated skipped=$skipped"
echo "summary: cloned=$cloned updated=$updated skipped=$skipped submodule_failed=$submodule_failed"
exit 0
Loading