feat: add tenant-rooted storage layout behind a setting #1863

[DmytroZaichenkoDev]

Part 1 of 2 for #1863 (addressing only). Part 2 is #1868.

What

Adds a tenant-rooted storage layout, off by default, and routes physical path composition through it.

Users/u1/files/documents/notes.txt   →  .org/<tenant>/.users/u1/.files/documents/notes.txt
public/files/policies/risk.pdf       →  .org/<tenant>/.files/policies/risk.pdf
Keys/<project>/files/report.csv      →  .org/<tenant>/.keys/<project>/.files/report.csv
platform/files/x                     →  .files/x        (platform scope is the root)

• TenantLayoutTransform — the conversion, total and reversible in both directions.
• StorageLayout + LegacyStorageLayout — ResourceDescriptor composed the bucket location and the resource-type folder inline in two places (getAbsoluteFilePath and resolveByPath); both now go through one layout.
• TenantRootedStorageLayout + StorageLayouts — the new layout and the holder for the active one.
• AiDial selects it at start-up from a new settings section:

"storageLayout": {
  "tenantRooted": false,
  "defaultTenant": "default"
}

Invariant

Off by default, so physical paths are unchanged. ResourceDescriptorPathTest pins the current paths; TenantRootedLayoutApiTest drives the resource API with the flag on through the real stack — round trip, listing, deletion, and the blob landing under .org/test-tenant/… in a reserved type folder — which is the only way to show the descriptor, the cache and the blob store agree on the same paths.

Notes for review

• The active layout is process-wide mutable state. ResourceDescriptor is constructed in ~63 files and carries no configuration, so there is nowhere else to put it without touching every construction site. Open to alternatives.
• AiDial sets the layout unconditionally on every start, including the legacy case, so it is always determined by settings rather than inherited from a previous start() in the same JVM.
• Applications and toolsets keep separate folders (.applications, .toolsets) rather than both mapping to .deployments as the design sketch shows: merging two groups into one folder is not invertible, and a working reverse transform is required for migration.
• storageLayout is a new top-level section rather than a field on storage — Storage is jclouds provider configuration, this is a logical concern — and deliberately static settings, not hot-reloaded config.
• defaultTenant defaults to "default"; that string becomes a physical path segment, so it is worth an explicit decision.
• Users/ and Keys/ prefixes are repeated in TenantLayoutTransform because storage cannot depend on server's BucketBuilder.

Not included

No migration sits behind this flag. Enabling it against populated storage re-addresses everything and the existing data is not reachable at the new paths — the README says so. Three things must land before it can be flipped anywhere real: re-encryption of path-bound secrets (see #1868 for the first half), a decision on invitation ids, which embed the physical path, and a measurement of the existing path-length distribution against the 900-byte ceiling, which tenant-rooting eats into.

[ai-dial-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

[astsiapanay]

Can we switch between layouts in runtime?

[DmytroZaichenkoDev]

No, by design: the layout is fixed at startup before any resource I/O, because a runtime flip re-addresses live data out from under the Redis cache, the dirty write-behind queue, and per-resource locks (all keyed by physical path). The setter exists so the comparison tests can boot two instances in one JVM. Per-bucket migration in P2 will come as a composite layout consulted per resolution, not a runtime switch.

[astsiapanay]

That's a dangerous method by nature. I'm thinking how we can make us safe. Can we come up with more robust mechanism so anyone couldn't change the layout in runtime?