Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 26 additions & 12 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,15 +8,21 @@ One client for everything you ask traust; one runner for the work you do for it.

```go
c, err := api.NewBoundClient(api.ClientConfig{BaseURL: coreURL, Tokens: api.FileTokenSource{Path: tokenPath}})
r, err := ops.NewRunner(c, myExecutor) // your Executor runs the skill; the runner does the rest
pack, err := ops.LoadPack(packYAML)
r, err := ops.NewRunner(ops.RunnerConfig{
Client: c, Pack: pack, Executor: myExecutor, Assets: []string{"ledger_audit"},
LeaseSeconds: 30, PollInterval: time.Second,
})
err = r.Run(ctx)
```

| I want to… | Use | Core operations |
|---|---|---|
| onboard a product and its repositories | `c` | `PlanOnboarding` (dry run) → `Onboard` |
| browse inventory | `c` | `ListProducts`, `ListRepositories`, `ListRefs`, `ListOwners`, `ListProductsOf` |
| run skills | `ops.Runner` + your `Executor` | `Claim`/`Start` → `Prepare*` → your run → `Record*` → `Complete` (the runner calls these, you don't) |
| ask for a run (a person's token) | `c` | `Submit` with the pack version it must run under |
| run queued pack commands | `ops.Runner` + your `Executor` | `Claim` → `GetOrderManifest` → your run → `Complete` (the runner calls these, you don't) |
| run and record a dispatched order | `ops.RecordingRun` + your `Executor` | `Start` → `Prepare*` → your run → `Record*` → `Complete` (same) |
| read what a run produced | `c` | `ListRepositoryAssessments`, `GetAssessment`, `GetArtifact`, `GetArtifactContent` |
| work findings (a person's token) | `c` | `ReviewQueue`, `GetFinding`, `Decide`, `ListDecisions`, `Restate` |
| check the ledger | `c` | `ListLayers`, `ListEvents`, `VerifyLayer`, `Checkpoint`; core writes ledger events when records and decisions land, you never append them |
Expand All @@ -27,36 +33,44 @@ Method names are core's operation ids. A long or unclear name is fixed in core's

## How it works

One lifecycle, two ways in. Both hold the order's lease the same way (`v1/ops/lease.go`); nothing else in the SDK heartbeats, records or completes.

```mermaid
sequenceDiagram
participant R as SDK runner (v1/ops)
participant R as SDK (v1/ops)
participant API as traust-core REST API
participant X as consumer Executor
R->>API: claim (or start a dispatched order)
R->>API: prepare → skill input
R->>X: run(input), heartbeating the lease
X-->>R: output document
R->>R: validate against schema
R->>API: record(document)
alt queued order: ops.Runner
R->>API: claim(assets)
R->>API: manifest: planned with my pinned pack?
else dispatched order: ops.RecordingRun
R->>API: start
R->>API: prepare → skill input
end
R->>X: run, heartbeating the lease
X-->>R: output + JobResult
R->>API: record(output), when the order records a document
R->>API: complete(JobResult)
```

Each step is one core endpoint. A step core doesn't serve is not a step: never stand one in locally.

| Step | Core endpoint |
|---|---|
| submit (a person) | `POST /v1/orders` with the config (pack version) the order must run under |
| claim / start | `POST /v1/orders/claim` (filtered by `assets`; 204 = nothing queued) · `POST /v1/orders/start` (an explicitly dispatched order) |
| pinned pack | `GET /v1/orders/{order_id}/manifest`: a runner runs only orders planned with exactly its pack's digest |
| heartbeat | `POST /v1/orders/{order_id}/heartbeat`; a 409 means the lease is lost: stop, record nothing |
| prepare | `GET /v1/assessments/{assessment_id}/triage-input`, and each domain's input route as core adds it |
| record | `POST /v1/assessments/{assessment_id}/triage-records` · `/v1/validation-records` · `/v1/verification-records` · `/v1/assessment-records` |
| complete | `POST /v1/orders/{order_id}/complete` with `JobResult` (outcome, degraded, usage); repeat-safe |
| complete | `POST /v1/orders/{order_id}/complete` with `JobResult` (outcome, degraded, usage, and a `RunOrderReason` from core's closed vocabulary); repeat-safe |
| feed | `GET /v1/feed`, cursor-paged |
| config resolve, preflight | not served by core: no such step |

| Package | Owns | Consumer writes |
|---|---|---|
| `v1/api` | the client, types and enums generated from core's OpenAPI; auth, read retries, typed problem errors, one page iterator | a `TokenSource` |
| `v1/ops` | the runner: claim, heartbeat, lease loss, prepare, validate, record, complete | an `Executor`; skill registrations |
| `v1/ops` | the lease (heartbeat, lease loss, complete) and its two ways in: `Runner` (claim, pinned pack, the asset's command) and `RecordingRun` (start, prepare, record) | an `Executor`; the pack file |
| `v1/feed` | reading `GET /v1/feed` once per event, resuming from a cursor | a cursor store |
| `v1/<pkg>/<pkg>test` | test doubles for each extension point; the conformance suite every `Executor` passes against `traust serve` | — |

Expand All @@ -67,7 +81,7 @@ Packages not in this table (`storage`, `ledger`, `skills`, `types`, `enums`, `va
- **Every change is one of three things:** an extension point the SDK calls, a lifecycle step in `ops`, or a regenerated `v1/api`. Anything else is glue; don't add it. A missing endpoint is a traust-core change, not a workaround here
- **Core decides; the SDK enforces.** Authorization, ordering, budget, classification, identity, IDs and timestamps come from core; the SDK never derives, caches or defaults them. Approvals and decisions come from verified people through core; the runner's identity only claims, heartbeats, records and completes. The SDK's one local check is schema validation, which core re-checks
- **Core's REST API is the only way in.** No database handle, DDL, contracts tables, URL building or path splicing: call the generated method
- **Generated, never committed by hand.** `make api-generate` builds core's OpenAPI at the pinned core commit, generates `client.gen.go`, and deletes the spec. Commit the generated Go, the codegen config and the pin, nothing else. Never edit `*.gen.go`; never hand-write a type or enum core already publishes; never wrap generated methods one-for-one
- **Generated, never committed by hand.** Everything the SDK knows about core comes from core at `CORE_REF`: `make api-generate` builds its OpenAPI, generates `client.gen.go` and deletes the spec, and writes `v1/ops/pack.schema.json` from the contracts core validates packs with. Commit those two generated files and the pin, nothing else; there is no second contracts pin. Never edit `*.gen.go`; never hand-write a type or enum core already publishes; never wrap generated methods one-for-one
- **Fail closed.** No answer means no run: 5xx, timeout, network failure, malformed body, an unrecognised problem all stop with a typed error. Errors map on HTTP status and, where core sends one, the problem `code`; never on `detail` text. Retry only idempotent reads
- **No fallbacks.** A missing or invalid dependency is a constructor error, never a default. Unknown is `nil`, never a zero value
- **One config.** The consumer passes core URL, credential source and stage identity; nothing else. Library code never reads env vars or files. No settings that choose between behaviours
Expand Down
5 changes: 4 additions & 1 deletion OWNERS
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# Prow (openshift-ci) approval config; see OWNERS_ALIASES for membership.
# The root governs the framework's shape (AGENTS.md, ci/, .github/, go/Makefile, go/go.mod,
# go/internal/generate, go/v1/ops, go/v1/api bindings, new packages). Directories with their own
# OWNERS hand routine changes to the whole team.
approvers:
- traust-admins
- sdk-framework
reviewers:
- traust-admins
4 changes: 4 additions & 0 deletions OWNERS_ALIASES
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@
# (https://github.com/orgs/traust-security/teams/traust-admins), which also gates
# merges via branch protection (CODEOWNERS + required code-owner review).
aliases:
# Approves the framework's shape; matches CODEOWNERS.
sdk-framework:
- chubirka
- smith-xyz
traust-admins:
- chubirka
- girish1430
Expand Down
3 changes: 3 additions & 0 deletions go/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,9 @@ All notable changes to the Go SDK are documented here.
### Changed

- `make api-generate` builds core's OpenAPI at `CORE_REF` and deletes it; the spec is no longer committed.
- `JobResult.Reason` is `*api.RunOrderReason` (core's closed vocabulary, `api.RunOrderReason*` constants), not free text.
- `api.Decode` returns a generated response's body or its `*ProblemError`; `ops.LoadPack` validates against the pack schema from core at `CORE_REF`.
- `ops.RecordingRun` completes failed, without recording, when the executor returns an unknown outcome.

## [0.26.0]

Expand Down
55 changes: 17 additions & 38 deletions go/Makefile
Original file line number Diff line number Diff line change
@@ -1,25 +1,23 @@
CONTRACTS_REPO := https://github.com/traust-security/traust-contracts.git
CONTRACTS_REF := ac517047c6566b019e63b911f6b7522000826b48
PACK_SCHEMA_REF := 4b7532bf044571a3a53077b03cf2955066a83374
CONTRACTS_CACHE := $(or $(XDG_CACHE_HOME),$(HOME)/.cache)/traust-sdk/contracts
CONTRACTS_SOURCE := $(CONTRACTS_CACHE)/checkout-$(CONTRACTS_REF)
PACK_SCHEMA_SOURCE := $(CONTRACTS_CACHE)/checkout-$(PACK_SCHEMA_REF)
ENUMS_DIR := $(CONTRACTS_SOURCE)/enums/v1
SCHEMAS_DIR := $(CONTRACTS_SOURCE)/schemas/v1
STORAGE_DIR := $(CONTRACTS_SOURCE)/storage/v1
FIXTURES_DIR := $(CONTRACTS_SOURCE)/tests/fixtures/storage
STORAGE_SAMPLES := $(CONTRACTS_SOURCE)/tests/storage_samples.py
CORE_REPO := https://github.com/traust-security/traust-core.git
CORE_REF := 977e8bf22117c2d58ba3c7bdfd38518274bb4f5b
CORE_REF := d06d3afd2b84cb8f4f7aa215172a4c802c108bc7
CORE_SOURCE := $(or $(XDG_CACHE_HOME),$(HOME)/.cache)/traust-sdk/core/checkout-$(CORE_REF)
CORE_PYTHON := uv run --no-dev --quiet --extra rest python
CORE_PYTHON := cd "$(CORE_SOURCE)" && uv run --quiet python
OPENAPI_DIR := ./internal/generate/openapi
OUT_DIR := ./v1/enums
ROOT := ..
GO_CI := python3 $(ROOT)/ci/lang/go/ci.py
export PATH := $(shell go env GOPATH)/bin:$(PATH)

.PHONY: contracts pack-schema core generate api-generate e2e e2e-runner test test-integration check-drift fmt fmt-check vet lint security tools
.PHONY: contracts core generate api-generate e2e test test-integration check-drift fmt fmt-check vet lint security tools

contracts:
@set -eu; \
Expand All @@ -38,28 +36,14 @@ contracts:
generate: contracts
go run ./internal/generate -enums "$(ENUMS_DIR)" -out "$(OUT_DIR)" -schemas "$(SCHEMAS_DIR)" -types ./v1/types -validate ./v1/validate/schemas_gen.go -storage "$(STORAGE_DIR)" -storage-out ./v1/storage -storage-fixtures "$(FIXTURES_DIR)" -storage-samples "$(STORAGE_SAMPLES)" -contracts-ref "$(CONTRACTS_REF)"

pack-schema:
@set -eu; \
if [ ! -d "$(PACK_SCHEMA_SOURCE)/.git" ]; then \
rm -rf "$(PACK_SCHEMA_SOURCE)"; \
mkdir -p "$(PACK_SCHEMA_SOURCE)"; \
git -C "$(PACK_SCHEMA_SOURCE)" init --quiet; \
git -C "$(PACK_SCHEMA_SOURCE)" remote add origin "$(CONTRACTS_REPO)"; \
fi; \
if ! git -C "$(PACK_SCHEMA_SOURCE)" cat-file -e "$(PACK_SCHEMA_REF)^{commit}" 2>/dev/null; then \
git -C "$(PACK_SCHEMA_SOURCE)" fetch --quiet --no-tags --depth=1 origin "$(PACK_SCHEMA_REF)"; \
fi; \
git -C "$(PACK_SCHEMA_SOURCE)" checkout --quiet --detach --force "$(PACK_SCHEMA_REF)"; \
cp "$(PACK_SCHEMA_SOURCE)/config/v1/pack.schema.json" ./v1/ops/pack.schema.json

test:
go test ./...

test-integration:
go test -count=1 -v ./v1/storage/...

check-drift: generate api-generate pack-schema
@git diff --exit-code $(OUT_DIR) ./v1/types/ ./v1/validate/schemas_gen.go ./v1/storage/operations.go ./v1/storage/projection_identifiers.go ./v1/storage/schema.go ./v1/storage/queries.go ./v1/storage/storagetest/testdata/ ./v1/api/client.gen.go ./v1/ops/pack.schema.json || (echo "ERROR: generated files are out of date; run 'make generate api-generate pack-schema' and commit" && exit 1)
check-drift: generate api-generate
@git diff --exit-code $(OUT_DIR) ./v1/types/ ./v1/validate/schemas_gen.go ./v1/storage/operations.go ./v1/storage/projection_identifiers.go ./v1/storage/schema.go ./v1/storage/queries.go ./v1/storage/storagetest/testdata/ ./v1/api/client.gen.go ./v1/ops/pack.schema.json || (echo "ERROR: generated files are out of date; run 'make generate api-generate' and commit" && exit 1)

core:
@set -eu; \
Expand All @@ -75,15 +59,20 @@ core:
git -C "$(CORE_SOURCE)" checkout --quiet --detach --force "$(CORE_REF)"; \
git -C "$(CORE_SOURCE)" clean -ffdq

# The spec is built from core at CORE_REF, used once and deleted; only client.gen.go is committed.
# Everything the SDK knows about core comes from core at CORE_REF: the client from its OpenAPI
# (built, used once, deleted) and the pack schema from the contracts core validates packs with.
api-generate: core
@set -eu; \
tmp="$$(mktemp -d)"; trap 'rm -rf "$$tmp"' EXIT; \
$(MAKE) -s -C "$(CORE_SOURCE)" api-check; \
python3 $(OPENAPI_DIR)/normalize.py "$(CORE_SOURCE)/.openapi/openapi.json" "$$tmp/openapi.json"; \
go tool oapi-codegen --config $(OPENAPI_DIR)/oapi-codegen.yaml "$$tmp/openapi.json"
go tool oapi-codegen --config $(OPENAPI_DIR)/oapi-codegen.yaml "$$tmp/openapi.json"; \
($(CORE_PYTHON) -c "import json; from traust_core.v1.contracts import config_schema; print(json.dumps(config_schema('pack'), indent=2))") > ./v1/ops/pack.schema.json

# A local-issuer token for the e2e server: $(1) identity, $(2) machine (True/False), $(3) file.
mint = ($(CORE_PYTHON) -c "from pathlib import Path; from traust_core.v1.context import Config, Context; print(Context(Config.from_file(Path('$$home/config.yaml'))).local_issuer.mint('$(1)', machine=$(2)).token.get_secret_value())") > "$$home/$(3)"

# The generated client against traust-server built from core at CORE_REF (SQLite, local issuer).
# The generated client and the runner against traust-server built from core at CORE_REF.
e2e: core
@set -eu; \
home="$$(mktemp -d)"; port=18080; \
Expand All @@ -92,20 +81,10 @@ e2e: core
trap 'kill $$pid 2>/dev/null || true; rm -rf "$$home"' EXIT; \
for _ in $$(seq 60); do curl -sf -o /dev/null http://127.0.0.1:$$port/v1/healthz && break; sleep 1; done; \
curl -sf -o /dev/null http://127.0.0.1:$$port/v1/healthz || { cat "$$home/server.log"; exit 1; }; \
(cd "$(CORE_SOURCE)" && uv run --quiet python -c "from pathlib import Path; from traust_core.v1.context import Config, Context; print(Context(Config.from_file(Path('$$home/config.yaml'))).local_issuer.mint('e2e@example.com').token.get_secret_value())") > "$$home/token"; \
TRAUST_E2E_URL=http://127.0.0.1:$$port TRAUST_E2E_TOKEN_FILE="$$home/token" go test -count=1 -tags e2e -run E2E -v ./v1/api/

e2e-runner: core
@set -eu; \
home="$$(mktemp -d)"; port=18081; \
pack="$(CURDIR)/v1/ops/testdata/local-command.yaml"; \
printf 'home: %s\nidentity:\n trust_local: true\n' "$$home" > "$$home/config.yaml"; \
(cd "$(CORE_SOURCE)" && exec $(CORE_PYTHON) -c 'from traust_core.v1.rest.server import main; main()' --config "$$home/config.yaml" --port $$port) > "$$home/server.log" 2>&1 & pid=$$!; \
trap 'kill $$pid 2>/dev/null || true; rm -rf "$$home"' EXIT; \
for _ in $$(seq 60); do curl -sf -o /dev/null http://127.0.0.1:$$port/v1/healthz && break; sleep 1; done; \
curl -sf -o /dev/null http://127.0.0.1:$$port/v1/healthz || { cat "$$home/server.log"; exit 1; }; \
(cd "$(CORE_SOURCE)" && $(CORE_PYTHON) "$(CURDIR)/v1/ops/testdata/seed_order.py" "$$home/config.yaml" "$$pack" "$$home/token"); \
TRAUST_E2E_URL=http://127.0.0.1:$$port TRAUST_E2E_TOKEN_FILE="$$home/token" go test -count=1 -tags e2e -run TestE2ERunnerLocalCommand -v ./v1/ops/
$(call mint,e2e@example.com,False,person); \
$(call mint,e2e-runner,True,runner); \
TRAUST_E2E_URL=http://127.0.0.1:$$port TRAUST_E2E_TOKEN_FILE="$$home/person" TRAUST_E2E_RUNNER_TOKEN_FILE="$$home/runner" \
go test -count=1 -tags e2e -run E2E -v ./v1/api/ ./v1/ops/

fmt:
gofmt -w .
Expand Down
4 changes: 4 additions & 0 deletions go/OWNERS
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
filters:
"^(README\\.md|CHANGELOG\\.md|VERSION)$":
approvers:
- traust-admins
23 changes: 12 additions & 11 deletions go/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,17 +56,18 @@ or cross-field policy are introduced.

## Queued fleet commands

`ops.LoadPack` validates the installed pack against the pinned Contracts
schema and retains its byte digest. Configure `ops.NewRunner` with a
`BoundClient`, the pack, an `Executor`, eligible asset names, lease duration
and poll interval. `RunOnce` claims one order; `Run` keeps claiming until
cancelled. The runner checks Core's run-manifest pack digest, resolves
`{run_id}`, maintains the lease, and completes from the executor's native
`api.JobResult`. The executor runs `Execution.Argv`, returns its outcome, and
stops all work when its context is cancelled. This path currently accepts only fleet-wide
commands without model or repository inputs, such as `ledger_audit`.
The queued fleet-command runner sends only `ProducedReport.Result` to Core;
`RecordingRun` uses `ProducedReport.Payload` for native record doors.
`ops.LoadPack` validates a pack against the schema core loads packs with and pins it by
the digest of its bytes. `ops.NewRunner` takes a `BoundClient`, the pack, an `Executor`,
the assets this host runs, a lease and a poll interval. `RunOnce` claims one order and
`Run` keeps claiming until cancelled. The runner runs an order only when core planned it
with exactly this pack (`GetOrderManifest`), resolves `{run_id}` in the asset's command,
holds the lease while the executor runs `Execution.Argv`, and completes with the
executor's `api.JobResult`. Refusals carry a `api.RunOrderReason`. Fleet-wide commands
only: an asset with partitions, a model or params is refused at `NewRunner`.
The runner sends only `ProducedReport.Result` to core; `RecordingRun` sends
`ProducedReport.Payload` to its record door.

A person asks for a run with `Submit`, naming the pack version it must run under.

## Native report recording

Expand Down
23 changes: 14 additions & 9 deletions go/internal/generate/openapi/normalize.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,7 @@
appear in core's document: exclusiveMinimum/exclusiveMaximum as numbers
instead of the 3.0 boolean flag, and pydantic's Optional as
anyOf: [schema, {type: "null"}] instead of 3.0's nullable: true. This
rewrites those forms and namespaces SourceType to preserve existing Go
SourceChannel constants. Wire values and validation constraints are unchanged.
rewrites those forms and namespaces the enums in NAMESPACED. Wire values and validation constraints are unchanged.

Usage: normalize.py <openapi.json> <openapi-3.0.json>
"""
Expand Down Expand Up @@ -52,20 +51,26 @@ def normalize(node: object) -> None:
normalize(item)


# Enums whose constants carry their type name, so a value shared with another enum never renames
# existing constants (SourceType beside SourceChannel; RunOrderReason's not_found, bad_request).
NAMESPACED = ("SourceType", "RunOrderReason")


def _camel(value: str) -> str:
return "".join(part.title() for part in value.replace("-", "_").split("_"))


def main(argv: list[str]) -> int:
if len(argv) != 3:
print(__doc__, file=sys.stderr)
return 2
with open(argv[1], encoding="utf-8") as handle:
spec = json.load(handle)
normalize(spec)
# Avoid renaming existing SourceChannel constants when SourceType is added.
source_type = spec["components"]["schemas"].get("SourceType")
if source_type is not None:
source_type["x-enum-varnames"] = [
"SourceType" + "".join(part.title() for part in value.split("_"))
for value in source_type["enum"]
]
for name in NAMESPACED:
schema = spec["components"]["schemas"].get(name)
if schema is not None:
schema["x-enum-varnames"] = [name + _camel(value) for value in schema["enum"]]
spec["openapi"] = "3.0.3"
with open(argv[2], "w", encoding="utf-8") as handle:
json.dump(spec, handle, indent=2)
Expand Down
8 changes: 8 additions & 0 deletions go/v1/api/OWNERS
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# The regenerated client and its tests are routine; the bindings in client.go are the root's.
filters:
"\\.gen\\.go$":
approvers:
- traust-admins
"_test\\.go$":
approvers:
- traust-admins
Loading
Loading