GitHub as a backing store for CALM

[rocketstack-matt]

Feature Proposal

Target Project:

Open question — this proposal identifies two possible target projects (calm-hub or cli) rather than assuming one. See "Proposed Implementation" below; I would like input from the other maintainers here on which is the better fit, or whether both are worth pursuing.

Description of Feature:

Allow a GitHub repository to serve as a persistence backend for CALM documents (namespaces, architectures, patterns, flows, etc.), so a team can get PR review, diffs, and git history "for free" without operating a database. The core motivation is a zero-ops option: a GitHub repo + a PAT as the entire backing store, aimed at small teams, demos, and GitOps-oriented shops.

User Stories:

• As a small team evaluating CALM, I want to store our architecture documents without standing up MongoDB or managing a NitriteDB data volume, so that adoption has near-zero infrastructure cost.
• As an architecture reviewer, I want changes to CALM documents to go through the same PR review my team already uses for code, so that architecture changes get the same scrutiny.
• As a CalmHub operator who already runs the server, I want an alternative to Mongo/Nitrite that stores data in a repo I control, so that my system of record is plain git rather than a database I have to back up and administer.

Current Limitations:

• calm-hub's storage layer is pluggable (org.finos.calm.store.*) but only has mongo and standalone (embedded NitriteDB) modes today — both require running the Quarkus server, and neither gives git-native history/PR review over the data itself.
• The CLI's calm workspace command is already a local, git-rooted bundle of CALM documents, but its only remote sync target is CalmHub via push/check/bump (one-way, no pull yet — tracked as TODO from feat(cli): Initial calm workspace commands #2378). It has no notion of syncing directly to an arbitrary GitHub repo.

Proposed Implementation:

Two structurally different connection points were scoped out; they solve related but distinct problems and are presented here as alternatives rather than a single recommendation.

Option A — CalmHub storage backend. Add a github mode to calm-hub's pluggable store layer, alongside mongo/standalone, following the existing "Adding a New Storage Backend" runbook in calm-hub/AGENTS.md.

• New package org.finos.calm.store.github implementing the 13 "content" store interfaces (NamespaceStore, ArchitectureStore, PatternStore, FlowStore, AdrStore, ControlStore, DomainStore, InterfaceStore, CoreSchemaStore, DecoratorStore, StandardStore, TimelineStore, ResourceMappingStore) plus a GithubCounterStore for ID generation. UserAccessStore, AuditLogStore, SearchStore are explicitly out of scope for phase 1 (see below).
• One configured repo per Hub instance; namespaces as top-level folders; one file per version ({namespace}/{resourceType}/{id}/{version}.json), mapping 1:1 onto the existing Mongo versions-map.
• Direct commits to a configured branch (synchronous, matching existing Mongo/Nitrite write semantics). The one-file-per-version layout means GitHub's own Contents API optimistic-concurrency check (sha matching, create-only PUTs) is the conflict detector — no custom locking needed. New resource IDs use a best-effort max+1-and-retry-on-409 scheme (not a hard atomic guarantee like Mongo's counter).
• In-memory cache with TTL over tree listings/file contents; versions are immutable once written so per-version content is cacheable indefinitely.
• A thin hand-written @RegisterRestClient interface for the specific GitHub REST endpoints needed (rather than a full GitHub SDK object model), matching the codebase's existing small-mockable-interface test style.
• Auth via a fine-grained PAT scoped to the one configured repo.
• Pro: every existing Hub consumer (calm-hub-ui, VSCode extension, calm-studio, workspace push) keeps working unchanged. Con: doesn't remove the operational footprint of running the Quarkus server — only the database goes away, so "zero-ops" is partial.

Option B — CLI workspace push/pull to GitHub. Add GitHub as a second sync target for calm workspace, alongside CalmHub, with no server involved at all.

• Introduce a RemoteStore-style abstraction (mirroring calm-hub's own pluggable-store pattern, but at the CLI layer): a CalmHubRemoteStore (the existing CalmHubClient behavior, refactored behind the interface) and a new GithubRemoteStore.
• .calm-workspace/config.json gains a remote target config (repo owner/name, branch, PAT via env var, namespace→path prefix). push commits tracked documents using the same one-file-per-version layout as Option A, via the GitHub Contents API.
• Requires building pull (reading the repo structure back and materializing local bundle files), which doesn't exist today for any remote target — this is genuinely new scope, not a side effect of adding a backing store.
• Pro: genuinely zero-server, best fit for the stated zero-ops motivation; smaller conceptual leap since a workspace is already git-rooted. Con: only serves the CLI — calm-hub-ui/VSCode/Studio get nothing from this unless they grow their own GitHub client, or CalmHub is later taught to read from a workspace-populated repo (a possible future convergence of both options, not designed here).

Alternatives Considered:

The two options above are the alternatives; neither is recommended over the other in this proposal. A third possibility — CalmHub reading from a GitHub repo that a workspace populates, effectively layering Option A on top of Option B's data — was noted as a possible future convergence but is out of scope for an initial implementation.

Testing Strategy:

• Option A: Unit tests mock the thin GitHub REST client interface directly (consistent with existing Mongo/Nitrite unit tests, which mock typed interfaces rather than using @SuppressWarnings("unchecked")). Integration tests (GithubArchitectureIntegration.java etc., mirroring the existing Mongo*Integration/Nitrite*Integration naming pattern) run against a local WireMock instance with scripted GitHub API responses — no TestContainers image exists for GitHub, and this avoids live network dependency or CI secrets.
• Option B: Unit tests around the new RemoteStore abstraction and GithubRemoteStore, likely also backed by a mocked/stubbed GitHub client; push/pull round-trip tests against a local git repo or stubbed API responses.

Documentation Requirements:

• calm-hub/AGENTS.md (Option A) or cli/AGENTS.md (Option B) updated with the new mode/target, config properties, and known limitations.
• User-facing docs on calm.finos.org describing setup (PAT scope, repo layout expectations).
• Explicit documentation of the open questions below as known gaps, not silent omissions.

Implementation Checklist:

• Design reviewed and approved (including which option — A, B, or both — is chosen)
• Implementation completed
• Tests written and passing
• Documentation updated
• Relevant workflows updated (if needed)
• Performance impact assessed

Additional Context:

Open questions raised by this proposal that need resolution regardless of which option is chosen:

• UserAccessStore (Option A only): could map to GitHub collaborator/team permissions, but Hub's namespace-scoped role model doesn't line up cleanly with GitHub's repo-level permission model.
• AuditLogStore (Option A only): git commit history is a natural fit for "what changed when," but commits would be authored by the Hub's service PAT, not the actual end user — actor attribution is lost unless Hub embeds user identity in commit metadata/trailers.
• SearchStore (Option A only): no query engine — would need an in-memory index built from cached repo contents; full-text semantics from Mongo/Nitrite likely won't translate directly.
• Rate limits at scale: GitHub's 5000 req/hr (PAT-authenticated) is shared across all callers using that token — more of a concern for Option A (many concurrent Hub users through one server) than Option B (single CLI invocation at a time).

[karlmoll]

@robmoffat - per discussion yesterday.

[willosborne]

I think if we want workspaces to push to github - why not just commit the file changes it makes via bump and push as normal?
if the argument is - so users don't need to look at git - then maybe we need an opinionated command that auto-commits stuff.
I worry that directly pushing with PATs is going to be difficult with enterprise deployments of Git + may also present more issues with e.g. rotating the PATs, as they do tend to expire/need rotating.

Maybe the github cli can help here? Not sure how that handles auth, but if we can piggyback on that with 'gh api' it could make things easier

That said though, I'm kind of curious how this git backing store works. Are there controls on who can commit? What's the branch structure?
How will we resolve issues with e.g. merge conflicts, in case of multiple users working on the same document?

I guess my first reaction is, this would be very cool, but Calm workspace/Hub is basically driven by a database right now; Git is not really a database, so we need a pretty opinionated workflow to make it a good user experience. Would be very interested to hear more about how GitHub is used as a backing store in the existing systems works today.

[rocketstack-matt]

I'd be interested in those who have voiced this feature as something they would like envisage interacting with it . . . is it as simple as we want CALM Hub to be able to read from GitHub but all the writes would be done using Git CLI natively - that would presumably resolve any concerns you may have @willosborne?

If we did that, the pain point perhaps becomes the layout of the repository, as it would need to be a structure CALM Hub would be expecting, which is perhaps where calm workspaces comes back into the picture as the local management layer to ensure anything committed into the repo is in the correct layout?

[YoofiTT96]

• 1 on the workflow. Option B is technically possible today albeit through native git commands and scoped to the .calm-workspace. Committing + pushing and pulling as native git commands allow you to use git as a backed store. An enhancement appears to be some command that acts as a facade over the gh cli. Option A syncing with Github is quite interesting and the so you get the PR reviews and git history + Calm hub versioning. The Layering down the line feels necessary.

I'm thinking merging to your main or default branch triggering a workflow that syncs your designated workspace to calm with calm workspaces once the layouting details are ironed out. However, that means if you push to Calm hub directly you would now have the Github store as only a subset of the archs with Calm Hub possessing the full set. Possibly having the UI show the delineation clearly

[rocketstack-matt]

Following the discussion above, here is a refined proposal that converges Options A and B into the shape @willosborne and @YoofiTT96 were circling: the Hub is a reader, native git is the writer, and calm workspace is the layout manager. No PATs for writes, no custom sync protocol — the repo's working tree is the store and git push is publish.

Shape

A new read-only calm.database.mode=git in calm-hub:

• The Hub keeps a JGit shallow clone of a configured repo/branch, fetches on a poll interval, and serves everything from an immutable in-memory snapshot pinned to a commit SHA. Reads go over the git protocol against a local clone — no REST API rate limits, and the mode is host-agnostic (GitHub, GHE, GitLab, Bitbucket all work unchanged; PR review comes from whichever host you use).
• Git mode forces calm.readonly=true — the exact mechanism the existing read-only mode already uses — so every write endpoint refuses writes through code paths that already exist, and no counter store is needed (counters only exist on create paths).
• Writes happen only via native git: branch → PR → merge. This directly answers the control questions raised above — who can commit is branch protection + CODEOWNERS + repo permissions; merge conflicts are git's own semantics (two branches adding the same version path with different content is a visible conflict, which is effectively the create-only 409 check expressed in git terms); actor attribution is better than the database modes, because authors commit as themselves and change audit is git log.

Layout convention (where calm workspace comes in)

The repo path mirrors the document $id URL path exactly, so file path ⇄ $id is a pure bidirectional transform:

calm/namespaces/{ns}/{architectures|patterns|standards|interfaces}/{name}/versions/{version}.json
calm/namespaces/{ns}/domains/{domain}/controls/{control}/requirement/versions/{version}.json
calm/mappings.json

calm workspace gains a canonical-layout mode: add/new place files at the convention path and allocate numeric IDs in a checked-in calm/mappings.json (this is how the Hub's numeric routes and ResourceMappingStore stay stable — and a duplicate ID allocation from two branches surfaces as a merge conflict in the PR, not a silent collision). bump copies the current version file to the new version path and edits the copy, so old versions stay immutable and PRs show per-version diffs. workspace check grows a layout-conformance check (every file's path matches its $id, mappings consistent) as the CI gate that keeps the repo Hub-readable. In this mode check/bump read versions from the tree, so they work offline — and no workspace pull is needed at all, because sync is just git pull.

Scope and deferrals

• Phase 1 types: exactly what the workspace $id grammar can express today — architectures, patterns, standards, interfaces, plus domains/controls. Flows, ADRs, timelines, decorators, and core schemas return empty/404 in git mode as documented limitations; extending the grammar (including ADRs' integer revisions) is a fast-follow.
• UserAccessStore: not backed in phase 1 — read authz stays whatever the Hub's security config provides; write authz is the repo's own permission model, since writes only happen via git. AuditLogStore: no-op; git history is the audit. Both deliberate, documented.
• Freshness: reads may lag the branch by up to the poll interval; webhook-driven refresh is a possible later addition.
• Search: in-memory over the snapshot, reusing the existing SearchTextMatcher.

Testing

Because the read path is JGit rather than a REST client, integration tests construct local bare repos in temp directories — no WireMock fixtures, no network, no CI secrets. A shared fixture repo produced by the workspace commands doubles as input for the Hub tests, exercising the contract from both sides.

Phasing

1. 1a — convention + CLI: define the layout, implement the workspace canonical-layout mode and conformance check. Testable with nothing but a git repo.
2. 1b — Hub git mode: the clone/index component plus the phase-1 stores, reading repos that 1a produces.

Open questions

I would like input from the other maintainers on two points:

1. Mode naming/framing — (a) host-agnostic git as proposed here, working with any git host; or (b) github, narrower but matching the issue title and leaving room for GitHub-specific features (webhook refresh, Checks integration) in the mode itself.
2. Canonical in-place layout — (a) the workspace places files in the fixed calm/namespaces/... tree, accepting the constraint as the price of zero drift between working files and served content; or (b) a publish step materialises the layout from free-form working files, keeping file-location freedom at the cost of generated output, double commits, and "which copy is real" questions.

[jpgough-ms]

As discussed on #2984 it would be great to flesh out the user stories here so that we can understand what is either missing or what problems this will solve, outside of it solving storage dependencies.

It be good to have a deeper dive once that's fleshed out.

[byrash]

Design reviewed and shared in WG on August 06 2026 #2925

#2925 (comment)
#2925 (comment)

[byrash]

Upon request from @rocketstack-matt & comment above, we will re review the design along with PR #3001 in tomorrow WG.

Thank You.

FYI: @jpgough-ms @markscott-ms

[byrash]

---

Story 1: Authenticate Architects with Enterprise Identity

▎ As an Enterprise Architecture function, we want CALM Hub to authenticate users through our existing OIDC identity provider, so that architecture asset access is governed by the same identity and entitlement system as every other enterprise tool — no separate credentials, no shadow user management, and audit trails tie back to real people.

Story 2: Git as the Single Source of Truth for Architecture Assets

▎ As an Enterprise Architecture team, we want CALM Hub to serve architecture assets directly from our GitHub repositories without requiring a separate database or manual import, so that the Pull Request workflow — where architects author, peer-review, and approve changes — remains the single governance process, and the Hub always reflects the approved, merged state.

Story 3: Immutable, SHA-Pinned Architecture Versions

▎ As an architect designing a solution, I want every reference to a building block, standard, or control to be pinned to a specific, immutable Git commit, so that my solution is reproducible — I know exactly what it was designed against, regardless of how the source asset evolves later — and I can upgrade deliberately, not silently.

Story 4: Federated Architecture Across Enterprise and Business Units

▎ As a Business Unit Architecture lead, I want my team to consume and extend Enterprise building blocks and standards by reference — without copying them into our repo — while also publishing BU-specific assets in our own namespace, so that we stay aligned with Enterprise governance while retaining autonomy over BU-specific architecture decisions.