DIAL 2.0 — Tenant-rooted storage, unified authorization, and native complex resources

[DmytroZaichenkoDev]

DIAL 2.0 — Tenant-rooted storage, unified authorization, and native complex resources

Type: EPIC / Design

Issue tracker

Statuses: ⬜ todo · 🚧 in progress · 👀 in review · ✅ merged

Child issue	Status
#1863 — P0 Foundation: storage-layout seam, tenant-rooted layout behind a setting	👀 in review — #1866, #1889
#1870 — P1 Verification: replay differ, access-decision differ, bucket verifier	👀 in review — #1891, #1892, #1893 (approved, stacked on #1866)
#1920 — P2: per-bucket migration state, write pause + forced cache-flush primitive	🚧 in progress
TBD — P2: bucket migrator (copy, verify, promote) + CEK fail-loud guard	⬜ todo
TBD — P2: cutover runbook automation (pre-flight scans, drain checks, rollback drill)	⬜ todo
TBD — P3: legacy-state translation through owning services (rules, publications, roles)	⬜ todo
TBD — P3: unified assignment/policy engine, proven equivalent by the P1 differ	⬜ todo
TBD — P3: engine swap, access/* endpoints, share/* deprecation	⬜ todo
TBD — P4: tenants and org-units (/v1/tenants, /v1/org-units), tenant-scoped addressing	⬜ todo
TBD — P4: limit policies and the config-role → authZ-role + limits fan-out	⬜ todo
TBD — P5: complex resources — package layout, dependency API, resource roles	⬜ todo
TBD — P5: links primitive, export/import, re-addressed publish/copy/move	⬜ todo
TBD — P6: versioning/tagging via link indirection	⬜ todo

Problem

DIAL Core stores every resource under a bucket-rooted physical layout: Users/{sub}/conversations/..., Keys/{project}/files/..., public/..., platform/.... The principal's identity is baked into the physical path, there is no tenant boundary anywhere in the tree, and platform-internal state (api_key_data/, background_jobs/, ...) sits interleaved at the same level as user content.

Authorization is an implicit union over an 11-rule chain (AccessService): if any of own-resources, admin, global-reader, auto-shared, per-request, appdata, review, public, shared, app-self, or chained-schema rules grants an action, access is allowed. Two properties make this hard to evolve:

• The public rules document fails open (RuleMatcher): no rules → readable by everyone. Omission grants access.
• No decision traces to an explicit, stored grant — several rules derive access from path shapes and identity conventions.

Above the substrate, applications and toolsets that are really multi-file packages have no native representation — no package boundary, no dependency declaration, no per-resource roles, no versioning.

DIAL 2.0 needs multi-tenancy (a tenant-rooted tree with hard boundaries), a default-deny, assignment-based authorization model where every outcome traces to a literal stored rule, and a native complex-resource model — without changing observable behavior for existing users until each change is deliberate and announced.

Use cases

1. Place every bucket under a tenant root (.org/{tenant}/...) so tenant isolation is structural, not conventional.
2. Migrate a populated deployment bucket-at-a-time, with per-bucket rollback and no full-store freeze.
3. Prove — not assert — that the new layout changes nothing a caller can observe, before any data moves.
4. Replace the legacy access chain with one engine (assignments + policies, default-deny) proven to decide identically on the pinned behavior corpus, with explain answering "why does this subject have access".
5. Real tenants and org-units with scoped addressing (/v1/files/{tenant}/{org-path}/resources/...) and limit policies.
6. Applications/toolsets as packages: declared dependencies, resource roles (manage.*/execute.*), export/import, and versioning on top.
7. Keep every previously issued identifier (bucket tokens, resource URLs) resolving across every phase.

Rejected alternatives

• A. API adaptation layer (old surface → new substrate, preserving legacy response shapes). Rejected: the layout seam sits below the API — a legacy request builds the same descriptor, resolves to a tenant-rooted path, and returns an unchanged URL. The seam is the adaptation layer.
• B. Copy-while-running + change-feed catch-up. Rejected: requires a reliable list of what changed during the copy; resource events ride at-most-once Redis pub/sub with no replay, so a dropped event is a silently stale object no verifier can detect.
• C. Lazy migrate-on-access. Rejected: a permanent fallback branch in the hot read path, and no bounded completion.
• D. Global big-bang freeze. Kept only as a fast path for deployments small enough that the freeze is minutes; rejected as the default — freeze time scales with object count, and DIAL has no read-only mode.
• E. Complex resources first / in parallel. Deferred on capacity, not rejected on merit: packaging rides the new substrate (scoped roles, links, tenancy), and a parallel bridged track has no bandwidth today.

Delivery tracks

Every phase belongs to exactly one track:

• Invisible — behavioral identity with today under the zero config; identity is measured by the P1 harness, never asserted.
• Opt-in — additive; a deployment that never enables it never experiences it.
• Announced — observable change; requires deprecation discipline and public communication.

graph TD
    P0["P0 Storage layout seam (#1863)<br/><i>invisible — in review</i>"]
    P1["P1 Verification instruments (#1870)<br/><i>invisible — in review</i>"]
    P2["P2 Migration + cutover<br/><i>invisible; measured, not assumed — DATA MOVES HERE</i>"]
    P3["P3 Unified authorization<br/><i>invisible + first announced deprecation</i>"]
    P4["P4 Tenancy: tenants, org-units, limits<br/><i>opt-in</i>"]
    P5["P5 Complex resources (native)<br/><i>opt-in + announced tightenings</i>"]
    P6["P6 Versioning / tagging<br/><i>opt-in</i>"]
    P0 --> P1 --> P2 --> P3 --> P4 --> P5 --> P6

Loading

Technical requirements

1. The tenant-rooted layout ships behind a setting, off by default (storage.layout.tenantRooted); enabling it on a populated deployment requires migration.
2. Logical URLs and bucket tokens are invariant. Only physical paths change; everything the caller observes — status, body, headers, etags — stays identical through P2.
3. Nothing durable derives from a physical path. Identifiers handed to users and encryption AAD bind to the layout-independent legacy path (getLegacyFilePath); a physical path is free to change, a stored artifact is not.
4. The physical mapping is total and reversible: every legacy location maps to exactly one tenant-rooted location and back; unmapped locations throw rather than pass through.
5. Every behavioral difference is machine-checked. A checked-in expected-divergences file governs the comparison: an unlisted divergence fails the build, and a stale entry fails too.
6. Migration is copy, never move, bucket-at-a-time behind a per-bucket write pause, with per-bucket rollback via the reverse transform and the old tree retained read-only through a deprecation window.
7. Exactly one live authorization engine at any time. The legacy chain stays authoritative until translated state is proven equivalent, then is replaced wholesale — never both at once across different endpoints.
8. The target model is default-deny; today's fail-open behaviors (unruled public folders) are materialized as explicit grants during translation, or deliberately and announcedly tightened.
9. Every resource type has an explicit migration verdict (migrate / regenerate / leave / drop) before the migrator runs; a type with no verdict blocks the migrator.
10. The resource API shape is preserved throughout: {resource-type} stays the first segment after /v1/, and the resource verbs are unchanged; later phases add addressing forms and sub-APIs, never break existing ones.

Solution overview

P0 — the seam (built, in review — #1863)

Every physical path is composed through a two-method StorageLayout seam (location prefix + type folder), consulted at one choke point in ResourceDescriptor. Two implementations:

same request:     PUT /v1/conversations/{bucket}/folder/chat1

legacy layout:    Users/u1/conversations/folder/chat1
tenant-rooted:    .org/default/.users/u1/.conversations/folder/chat1

Dotted names are reserved structure (.org, .users, .keys, .system, dotted type folders); the transformer rejects collisions, dotted principal ids, and malformed tenant ids outright. System buckets (SystemResourceRegistry) relocate to a root-level .system/ segment, preserving whole-bucket scans and globally unique keys.

Correction (2026-09-08, from #1920): an earlier version of this paragraph said P2's per-bucket flag needs no interface change, because the layout receives the bucket location as an argument. That is one method short — a physical path is composed from a location prefix and a resource-type folder, and only the first is resolved with the bucket in hand. The layout is therefore chosen per bucket, one level up, so both parts come from the same one; the two-method interface itself is unchanged.

P1 — verification instruments (built, in review — #1870)

The layout is process-wide and chosen at startup, so comparison is between two runs, not within one: both instances replay the same corpus from empty and everything observable is diffed.

1. Replay differ — one checked-in corpus through both layouts; diffs status, body, headers, and etags (content-derived, so they check bytes without comparing paths). Captured values that both runs derive from shared inputs are compared by value; pagination is walked to the end with each layout consuming its own tokens.
2. Access-decision differ — a subject × resource × action matrix over HTTP covering 10 of the 11 rules (the 11th is unreachable and recorded as such, with the reason enforced). Grant cells assert expected permissions; denial cells assert full denial. Built in P1, reused in P3 as the equivalence specification for the new engine — captured while a working reference implementation still exists.
3. Bucket verifier — migrate a seeded bucket, boot a core onto the copy, read everything back: inventory, content, blob metadata, plus named checks for the two artifacts whose silent loss is catastrophic — public/rules/rules (fails open when absent) and an AAD-encrypted secret (proves ciphertext survives migration).

All three run in CI as an isolated Gradle task (:server:layoutDiffTest), quarantined because the suite's job is flipping process-wide static state.

P2 — migration and cutover (next; data moves here)

Per bucket: seal it against writes, flush its write-behind entries, copy its objects to tenant-rooted paths, promote it to the new layout, release. Nothing changes inside a bucket while it is copied, so staleness is impossible by construction; the freeze is one principal for seconds, not the store for hours. No change feed is needed — that is the point of the mechanism. Reads are never paused: the copy goes to a separate tree, so a sealed bucket keeps serving its original content until it is promoted.

Correction (2026-09-08, from #1920): the seal is not the existing bucket lock, as an earlier version of this paragraph assumed. Ordinary resource writes lock the individual resource, not the bucket; the bucket lock is a convention observed by sharing, invitations, publications, move/copy, complex resources and the admin config surface, so copying under it would not stop a plain file upload. The write pause is a state on the per-bucket flag, checked on the write path — which is why #1920 covers the flag and the barrier as one piece of work.

Bucket order: user buckets (canary — many, small, independent) → Keys/ (blocked on the project-vs-deployment disambiguation) → public/ (the hard one: holds the rules document and publication state) → platform (last — config entities are addressed by short name and never read from blob on the request path).

Key constraints established by review of the shipped code:

• Encrypted resources byte-copy like everything else — the AAD binds to the layout-independent path — with one ordering rule: the per-bucket CEK (encryption_keys) is the first type copied, and CEK absence during migration must fail loud (today's getOrCreateKey would silently mint a fresh key and orphan every ciphertext in the bucket).
• Redis keys resources, dirty-queue entries, and per-resource locks by physical path. A layout change therefore requires a drained sync queue and a cold (non-rolling) restart for the global flag; for the per-bucket flag it makes strongly-consistent flag reads a design requirement, not an optimization. Rollback drills must flush cache — legacy-keyed entries become addressable again on a flip back.
• Per-type verdicts are a checked-in decision table (all 31 types: migrate / deferred / drop) — the migrator's spec. Notables: response mappings and background jobs migrate (long-lived, user-visible); per-request key data, cost accumulators, limit counters, live sessions and channels drop. Existing shares, share state, and pending invitations are dropped at cutover (product decision): they would survive relocation, but dropping them means P3's engine only has to honor shares created under its own model.
• Pre-flight scans: path-length headroom (tenant-rooting adds ~15–25 bytes against the 900-byte cap; a resource inside that margin migrates fine and then fails validation on every subsequent access, including its own delete).

Compatibility rules: previously issued bucket tokens keep resolving (they are persisted in conversations, app ids, and third-party storage we cannot survey); newly issued tokens may change shape; public and platform stay literal aliases through the deprecation window. Deleting the old tree is a separate, announced, much later decision.

P3 — unified authorization

The legacy 11-rule union retires in favor of one model: assignments and policies, stored as files, evaluated identically at every scope, default-deny, every outcome tracing to a literal rule, introspectable via explain.

• The hard invariant: exactly one live engine. Split enforcement (new endpoints on the new model, old endpoints on the chain) produces a resource readable through one surface and denied through the other — a security incident with a rollout plan, not a migration strategy.
• Translation goes through the owning services — read the old DTO, map, write via the service that enforces the new model's invariants — never a blob copy or payload rewrite. In scope: rules, publication state, role/limit fan-out. Out of scope by the share-amnesty decision: legacy share documents.
• The P1 access differ is the acceptance test. Divergences are allowed but each must be classified: a bug being fixed, or a deliberate tightening that gets announced. An unexplained difference is a stop condition. The named hazard: unruled public folders are readable-by-everyone today; unless materialized as explicit grants, they all go dark at the swap — silently, for everyone.
• API: adds /v1/ops/resource/access/{assign,revoke,list,explain,check}; deprecates /v1/ops/resource/share/* (superseded by access/* with "sharing": true, kept operational through the window) — the programme's first announced deprecation.

P4 — tenancy: tenants, org-units, limits (opt-in)

Populates the latent tenancy the substrate has carried since P0: real tenants beyond the default, org-units, /v1/tenants and /v1/org-units (+ /join), limit policies, and the config-role fan-out (each config role splits into an authZ role + limit policies + guardrails). Shared resources gain the tenant/org-scoped addressing form — /v1/files/public/policies/risk.pdf → /v1/files/{tenant}/{org-path}/resources/policies/risk.pdf, with resources as the reserved delimiter. Behavior shift to watch: public → default tenant is behavior-preserving only while one tenant exists; the second tenant is where "public" starts meaning "public within tenant" — gated behind opt-in adoption.

P5 — complex resources, native (opt-in + announced tightenings)

Applications and toolsets become packages: a package layout and boundary, the dependency API (app / current-user / current-call), the links primitive (first consumed as external dependencies), per-resource roles (manage.* / execute.*, including custom), export/import, per-request-key runtime surface bound to its issuing resource, and publish/copy/move re-addressed off public/.... The announced sub-track carries the deliberate tightenings (attachment authority, share-graph exposure, grant lifetime, invocation consent) — never smuggled in as side effects. This phase is deliberately last: packaging rides scoped roles, links, and tenancy from P3–P4.

P6 — versioning / tagging (opt-in)

Link-indirection versioning on top of P5's links primitive.

API surface by phase

Phase	API surface
P0–P2	None. The old surface stays frozen and is served from the new substrate; the entire migration is invisible at the API
P3	Added /v1/ops/resource/access/*; deprecated /v1/ops/resource/share/* (kept through the window)
P4	Added /v1/tenants/..., /v1/org-units/... (+ /join), limits endpoints; tenant/org-scoped addressing form for shared resources
P5	Added per-resource dependencies/... and roles/... sub-APIs, /v1/ops/resource/links/..., /v1/ops/resource/{export,import}; publications and copy/move re-addressed
P6	None — rides P5's link indirection

Operational changes

• Static settings: storage.layout.tenantRooted (default false), storage.layout.defaultTenant (default "default") — documented in the README settings table.
• CI: storage_layout_comparison job runs the three instruments as a sibling of run_tests; the comparison corpus and the expected-divergences file are checked in and reviewable like code.
• Storage layout (when enabled): new top-level prefixes .org/{tenant}/ and .system/; .dial-tmp and complex_resource_refs remain at the root (recorded decision, revisited if per-tenant staging isolation is required).
• P2 adds: the per-bucket migrated flag (strongly consistent reads), a forced-flush primitive, the migrator, and pre-flight/drain checks in the runbook.

References

1. P0: [DIAL 2.0] Introduce tenant-rooted storage layout (addressing only) #1863, PRs feat: add tenant-rooted storage layout behind a setting #1863 #1866, fix: map system buckets and keep durable artifacts stable under the tenant-rooted layout #1863 #1889 · P1: [DIAL 2.0] Verify behaviour is unchanged on the new storage layout #1870, PRs test: compare the two storage layouts by replaying one corpus through both #1870 #1891, test: compare access decisions across the two storage layouts #1870 #1892, test: verify a bucket migrated to the tenant-rooted layout #1870 #1893
2. Key code: StorageLayout / StorageLayouts / TenantLayoutTransformer / SystemResourceRegistry (storage module), ResourceDescriptor.getLegacyFilePath, AccessService / RuleMatcher (legacy chain), server/src/test/.../layout/ + server/src/test/resources/layout-diff/ (instruments and corpus)
3. Detailed phase designs, decision records (delivery sequencing, coexistence posture, cutover mechanism, invitation-link policy, share amnesty), the conceptual target model (§3 storage, §6 authorization, §7 complex resources), and the per-type migration verdict table are maintained in the internal design repository.