Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ See the CHANGELOG rule in [AGENTS.md](../AGENTS.md).

## 2026-10-19

- A redeploy with unchanged deployer-owned secrets skips their six GitHub writes: a lock-fenced Redis
digest receipt certifies the last complete write; `REGISTRY_*` are still written every deploy.
- New products scaffold from kit tag `0.11.0` (commit `59a0752c`): parallel PR CI behind fail-closed
`lint-and-test`, native SSH deploy with `up --wait`; existing products keep their workflow until updated.
- The deployer rereads a pending deploy run and an accepted failed-job rerun every 3 s instead of
Expand Down
2 changes: 1 addition & 1 deletion docs/ERROR_HANDLING.md
Original file line number Diff line number Diff line change
Expand Up @@ -343,7 +343,7 @@ unknown failures use the bounded `RETRY` path. A future remediation agent may di
asynchronously and propose a tested code fix outside the deploy path.

### Deploy Deduplication and the Deploy Fence
Atomic `SET NX EX` Redis lock per project (`deploy:<project_id>:lock`, TTL `deploy.deploy_lock_ttl`) prevents duplicate deploys. Its value is a token unique to the holder (`<task_id>:<uuid>`), and the claim is the deploy's `DeployFence` (`services/langgraph/src/deploy_fence.py`), threaded explicitly through the consumer, the DevOps subgraph state (`deploy_fence`) and the result handlers. Every deploy write calls `DeployFence.ensure_held(<DeployWrite>)` — a Lua GET-compare — immediately before it is made: secret persistence and GitHub secret writes, the pin tag, stopping older runs, workflow dispatch and rerun, the deployment record, the post-deploy product writes (owner grant, settings seed), remote execution, and the deploy's Run and application state. Two rules govern the checks: nothing is awaited between a check and its effect (an API claim, read or wait in between gets its own check after it, so the GitHub dispatch and rerun check again after the dispatch claim), and a Run's terminal outcome is written last (a lifecycle action moves the application's status before it records its Run). A deploy whose lock expired or was replaced raises `DeployFenceLost`, makes no further write, and records its own Run once as `DeployOutcome.DEPLOY_LOCK_LOST` — never over a Run that already holds a terminal outcome, which the API refuses to rewrite; the supervisor redeploys the story under the retry bound. Release is a compare-and-delete in `finally`, so a deploy never deletes a lock another deploy now holds. The check narrows the window to one Redis round trip; it cannot stop a write already in flight when the key expires.
Atomic `SET NX EX` Redis lock per project (`deploy:<project_id>:lock`, TTL `deploy.deploy_lock_ttl`) prevents duplicate deploys. Its value is a token unique to the holder (`<task_id>:<uuid>`), and the claim is the deploy's `DeployFence` (`services/langgraph/src/deploy_fence.py`), threaded explicitly through the consumer, the DevOps subgraph state (`deploy_fence`) and the result handlers. Every deploy write calls `DeployFence.ensure_held(<DeployWrite>)` — a Lua GET-compare — immediately before it is made: secret persistence, GitHub secret writes (each PUT attempt) and the deploy-secrets receipt (`docs/SECRETS.md`), the pin tag, stopping older runs, workflow dispatch and rerun, the deployment record, the post-deploy product writes (owner grant, settings seed), remote execution, and the deploy's Run and application state. Two rules govern the checks: nothing is awaited between a check and its effect (an API claim, read or wait in between gets its own check after it, so the GitHub dispatch and rerun check again after the dispatch claim), and a Run's terminal outcome is written last (a lifecycle action moves the application's status before it records its Run). A deploy whose lock expired or was replaced raises `DeployFenceLost`, makes no further write, and records its own Run once as `DeployOutcome.DEPLOY_LOCK_LOST` — never over a Run that already holds a terminal outcome, which the API refuses to rewrite; the supervisor redeploys the story under the retry bound. Release is a compare-and-delete in `finally`, so a deploy never deletes a lock another deploy now holds. The check narrows the window to one Redis round trip; it cannot stop a write already in flight when the key expires.

### Stale Worker Cleanup
`_check_project_lock()` in the engineering consumer verifies `worker:status` in Redis. Workers in terminal states (`DEAD`/`FAILED`/`STOPPED`) get their Redis keys cleaned up automatically, unblocking new task dispatch without manual intervention.
Expand Down
7 changes: 4 additions & 3 deletions docs/SECRETS.md
Original file line number Diff line number Diff line change
Expand Up @@ -273,9 +273,10 @@ Application deployment is fully delegated to GitHub Actions. This allows secure
2. **Secret Injection** (three stages):
* **Scaffolder**: Sets `REGISTRY_URL`, `REGISTRY_USER`, `REGISTRY_PASSWORD` immediately after repo creation (before first CI push)
* **scheduler-pipeline**: The PR poller rewrites the same three immediately before every merge, and on every tick a PR is still armed for GitHub auto-merge from an earlier release; it is the only automated merger (no GitHub auto-merge), so push-main CI never starts on stale or absent registry secrets; a failed write blocks the merge
* **DeployerNode**: Sets 9 secrets total — `DOTENV`, `DEPLOY_HOST`, `DEPLOY_USER`, `DEPLOY_SSH_KEY`, `DEPLOY_PORT`, `PROJECT_NAME`, `REGISTRY_URL`, `REGISTRY_USER`, `REGISTRY_PASSWORD`
3. **CI workflow** (`ci.yml`, on push): lint → test → build images → push to self-hosted Docker registry
4. **Deploy workflow** (`deploy.yml`, on `workflow_dispatch` from Orchestrator): SCP compose files → write `.env` from DOTENV → pull images → `docker compose up`
* **DeployerNode** (`subgraphs/devops/deploy_secrets.py`): every deploy needs 9 secrets set before it dispatches `deploy.yml`. It writes the three `REGISTRY_*` on every deploy, and is the only writer of the other six: `DOTENV`, `DEPLOY_HOST`, `DEPLOY_USER`, `DEPLOY_SSH_KEY`, `DEPLOY_PORT`, `PROJECT_NAME`. Those six are skipped, with their public-key reads, while the deploy-secrets receipt vouches for them (below)
3. **Deploy-secrets receipt**: Redis key `deploy:<project_id>:github-secrets` holds one SHA-256 over the GitHub repository id (one `get_repo` read per deploy) and the six resolved names and values in canonical order; never a value, key or per-value digest. A deploy whose digest matches writes only `REGISTRY_*`. Any other receipt (absent, corrupt, another digest, a recreated repository) is a miss: the deploy deletes the receipt, writes all nine, and sets the new digest only after all nine succeeded. Reading, deleting and setting it are Lua scripts that first compare the project deploy lock with this deploy's token, and every secret PUT attempt checks the lock, so a deploy that lost the lock fails `deploy_lock_lost` and certifies nothing. A partial or interrupted write leaves no receipt, so the next deploy writes all nine again; a Redis or repository-read failure refuses the deploy rather than counting as a hit. Losing the receipt costs one full write. The digest attests what the orchestrator last wrote, not what GitHub holds: after a managed secret is edited or deleted outside the orchestrator, delete the receipt before the next deploy with `docker compose exec -T redis redis-cli DEL deploy:<project_id>:github-secrets`. As with every deploy write, a PUT already in flight when the lock expires is not revoked.
4. **CI workflow** (`ci.yml`, on push): lint → test → build images → push to self-hosted Docker registry
5. **Deploy workflow** (`deploy.yml`, on `workflow_dispatch` from Orchestrator): SCP compose files → write `.env` from DOTENV → pull images → `docker compose up`

**Privilege Separation:**
* **Infra Service**: Can create/destroy servers (Root access via Ansible). Cannot see App Secrets.
Expand Down
18 changes: 11 additions & 7 deletions services/langgraph/src/deploy_fence.py
Original file line number Diff line number Diff line change
Expand Up @@ -121,13 +121,17 @@ async def ensure_held(self, write: DeployWrite) -> None:
"""Refuse `write` unless the lock still stores this deploy's token."""
held = await self.redis.eval(_HOLDS_DEPLOY_LOCK, 1, self.lock_key, self.token)
if held != 1:
logger.warning(
"deploy_fence_lost",
project_id=self.project_id,
lock_key=self.lock_key,
refused_write=write.value,
)
raise DeployFenceLost(self.project_id, write)
raise self.lost(write)

def lost(self, write: DeployWrite) -> DeployFenceLost:
"""The refusal of `write`, logged, for a check that found another holder."""
logger.warning(
"deploy_fence_lost",
project_id=self.project_id,
lock_key=self.lock_key,
refused_write=write.value,
)
return DeployFenceLost(self.project_id, write)

async def release(self) -> None:
"""Delete the lock only while it is still this deploy's."""
Expand Down
134 changes: 134 additions & 0 deletions services/langgraph/src/subgraphs/devops/deploy_secrets.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
"""The GitHub Actions repository secrets a deploy needs, and the receipt that skips unchanged ones.

The deployer is the only writer of six secrets (`DOTENV`, `DEPLOY_HOST`,
`DEPLOY_USER`, `DEPLOY_SSH_KEY`, `DEPLOY_PORT`, `PROJECT_NAME`). Redis key
`deploy:<project_id>:github-secrets` holds the SHA-256 of the last complete
successful write of all six, bound to the GitHub repository id, so a redeploy
with the same values to the same repository writes only the registry secrets.
The receipt is read, deleted and set only inside a Lua script that first
compares the project deploy lock with this deploy's token.
"""

from collections.abc import Iterable
from functools import partial
import hashlib
import json

import structlog

from shared.clients.github import (
GitHubAppClient,
RegistrySecretsNotRefreshedError,
registry_repository_secrets,
)
from shared.diagnostics import redact_diagnostic

from ...deploy_fence import DeployFence, DeployFenceLost, DeployWrite

logger = structlog.get_logger()

_NOT_HELD = -1

# A hit leaves the receipt; anything else is deleted before the write starts, so
# an interrupted write can never leave a receipt for values it replaced.
_CLAIM_RECEIPT = """
if redis.call('GET', KEYS[1]) ~= ARGV[1] then
return -1
end
if redis.call('GET', KEYS[2]) == ARGV[2] then
return 1
end
redis.call('DEL', KEYS[2])
return 0
"""

_PUBLISH_RECEIPT = """
if redis.call('GET', KEYS[1]) ~= ARGV[1] then
return -1
end
redis.call('SET', KEYS[2], ARGV[2])
return 1
"""


def receipt_key(project_id: str) -> str:
return f"deploy:{project_id}:github-secrets"


def payload_digest(repository_id: int, secrets: dict[str, str]) -> str:
"""SHA-256 of the repository id and every resolved secret name and value, order-free."""
canonical = json.dumps(
{"repository_id": repository_id, "secrets": secrets},
sort_keys=True,
separators=(",", ":"),
ensure_ascii=False,
)
return hashlib.sha256(canonical.encode("utf-8")).hexdigest()


async def _fenced_receipt(fence: DeployFence, script: str, digest: str) -> int:
result = await fence.redis.eval(
script, 2, fence.lock_key, receipt_key(fence.project_id), fence.token, digest
)
if result == _NOT_HELD:
raise fence.lost(DeployWrite.GITHUB_SECRETS)
return result


async def write_deploy_secrets(
github: GitHubAppClient,
owner: str,
repo: str,
deployer_secrets: dict[str, str],
*,
deploy_fence: DeployFence,
diagnostic_secrets: Iterable[str] = (),
) -> bool:
"""Make the deployer's six secrets and the three registry secrets current in `owner/repo`.

True only when all nine are known set: written now, or the six certified by a
matching receipt. A lost deploy lock raises `DeployFenceLost`; every other
failure, Redis and the repository read included, is False and publishes nothing.
"""
# The scaffolder and the PR poller write the registry secrets too, so no
# deploy-only receipt can vouch for them; rewriting them on every deploy keeps
# an administrative or capability redeploy's repository current as well.
try:
registry_secrets = registry_repository_secrets()
except RegistrySecretsNotRefreshedError as error:
logger.error("registry_env_missing", reason=error.reason.value, detail=error.detail)
return False

guard = partial(deploy_fence.ensure_held, DeployWrite.GITHUB_SECRETS)
try:
# A repository deleted and recreated under the same name gets a new id.
repository = await github.get_repo(owner, repo)
await guard()
digest = payload_digest(repository.id, deployer_secrets)
cached = await _fenced_receipt(deploy_fence, _CLAIM_RECEIPT, digest) == 1
pending = registry_secrets if cached else {**deployer_secrets, **registry_secrets}
written = await github.set_repository_secrets(owner, repo, pending, guard=guard)
logger.info(
"deploy_secrets_configured",
owner=owner,
repo=repo,
secrets_count=written,
cached_count=len(deployer_secrets) if cached else 0,
total=len(deployer_secrets) + len(registry_secrets),
)
if written != len(pending):
return False
if not cached:
await _fenced_receipt(deploy_fence, _PUBLISH_RECEIPT, digest)
return True
except DeployFenceLost:
raise
except Exception as e:
logger.error(
"deploy_secrets_setup_failed",
owner=owner,
repo=repo,
error=redact_diagnostic(e, secrets=diagnostic_secrets),
error_type=type(e).__name__,
)
return False
87 changes: 14 additions & 73 deletions services/langgraph/src/subgraphs/devops/deployer.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,7 @@
from langchain_core.messages import AIMessage
import structlog

from shared.clients.github import (
GitHubAppClient,
RegistrySecretsNotRefreshedError,
RequestGuard,
deploy_pin_tag,
registry_repository_secrets,
)
from shared.clients.github import GitHubAppClient, RequestGuard, deploy_pin_tag
from shared.clients.registry import RegistryError
from shared.contracts.dto.application import ApplicationStatus
from shared.contracts.dto.deploy_dispatch import DeployDispatchClaim
Expand All @@ -26,6 +20,7 @@
from ...deploy_fence import DeployFence, DeployFenceLost, DeployWrite
from ...nodes.base import FunctionalNode
from ...runtime_identity import project_spec_runtime_slug
from .deploy_secrets import write_deploy_secrets
from .deploy_workflow import require_backend_workflow
from .dotenv_builder import build_dotenv, encode_dotenv
from .image_gate import ImagesNotPublishedError, image_references, verify_published_images
Expand Down Expand Up @@ -294,62 +289,6 @@ def _encode_deploy_payload(
return dotenv_b64


async def _write_deploy_secrets( # noqa: PLR0913
github_client: GitHubAppClient,
owner: str,
repo: str,
server_ip: str,
port: int,
project_name: str,
dotenv_b64: str,
ssh_key: str,
ssh_user: str,
*,
deploy_fence: DeployFence,
diagnostic_secrets: Iterable[str] = (),
) -> bool:
"""Write deployment secrets to GitHub repository for deploy.yml workflow."""
# Registry credentials for CI docker push. The PR poller already wrote them
# before the merge that started this commit's build; rewriting them here keeps
# an administrative or capability redeploy's repository current as well.
try:
registry_secrets = registry_repository_secrets()
except RegistrySecretsNotRefreshedError as error:
logger.error("registry_env_missing", reason=error.reason.value, detail=error.detail)
return False

secrets_map = {
"DOTENV": dotenv_b64,
"DEPLOY_HOST": server_ip,
"DEPLOY_USER": ssh_user,
"DEPLOY_SSH_KEY": ssh_key,
"DEPLOY_PORT": str(port),
"PROJECT_NAME": project_name,
**registry_secrets,
}

await deploy_fence.ensure_held(DeployWrite.GITHUB_SECRETS)
try:
count = await github_client.set_repository_secrets(owner, repo, secrets_map)
logger.info(
"deploy_secrets_configured",
owner=owner,
repo=repo,
secrets_count=count,
total=len(secrets_map),
)
return count == len(secrets_map)
except Exception as e:
logger.error(
"deploy_secrets_setup_failed",
owner=owner,
repo=repo,
error=redact_diagnostic(e, secrets=diagnostic_secrets),
error_type=type(e).__name__,
)
return False


class DeployerNode(FunctionalNode):
"""Deploy via GitHub Actions: write secrets, dispatch deploy.yml, wait for completion."""

Expand Down Expand Up @@ -768,16 +707,18 @@ async def _deploy(self, state: DevOpsState) -> dict: # noqa: PLR0911
repo=repo,
dotenv_len=len(dotenv_b64),
)
secrets_ok = await _write_deploy_secrets(
github_client=github,
owner=owner,
repo=repo,
server_ip=server_ip,
port=port,
project_name=project_name,
dotenv_b64=dotenv_b64,
ssh_key=ssh_key,
ssh_user=server.ssh_user,
secrets_ok = await write_deploy_secrets(
github,
owner,
repo,
{
"DOTENV": dotenv_b64,
"DEPLOY_HOST": server_ip,
"DEPLOY_USER": server.ssh_user,
"DEPLOY_SSH_KEY": ssh_key,
"DEPLOY_PORT": str(port),
"PROJECT_NAME": project_name,
},
deploy_fence=deploy_fence,
diagnostic_secrets=diagnostic_secrets,
)
Expand Down
11 changes: 11 additions & 0 deletions services/langgraph/tests/service/_public_deploy_transport.py
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,17 @@ def respond(self, request):
)
assert request.method == "GET"
assert request.headers["Authorization"] == "token content-fixture-token"
if path == "/repos/fixture/public":
return httpx.Response(
200,
json={
"id": 1001,
"name": "public",
"full_name": "fixture/public",
"html_url": "https://github.com/fixture/public",
"clone_url": "https://github.com/fixture/public.git",
},
)
assert request.headers["Accept"] == "application/vnd.github.raw+json"
assert dict(request.url.params) == {"ref": self.built}
assert path in {
Expand Down
Loading