diff --git a/content/docs/data-modeling/drivers.mdx b/content/docs/data-modeling/drivers.mdx index 5c5b6dfc53..81c1093b33 100644 --- a/content/docs/data-modeling/drivers.mdx +++ b/content/docs/data-modeling/drivers.mdx @@ -166,6 +166,19 @@ export OS_DATABASE_URL=file:./data/app.db export OS_DATABASE_URL=:memory: ``` +### Space Reclamation & Table Rotation (ADR-0057) + +The SQL driver connects SQLite with `auto_vacuum=INCREMENTAL` and exposes +`reclaimSpace()` (`PRAGMA incremental_vacuum`), which the platform +LifecycleService calls after every sweep that deleted rows — so the database +file actually shrinks instead of pinning at its high-water mark. Objects +declaring `lifecycle.storage.strategy: 'rotation'` are physically +time-sharded on SQLite: writes land in the current shard, reads go through a +view under the table's name, and expired shards are reclaimed with a single +`DROP TABLE`. On PostgreSQL/MySQL both are no-ops — those engines manage +space with their own vacuum machinery, and rotation falls back to an +equivalent age-based reap. + ## Memory Driver The in-memory driver keeps records in plain in-process objects (queried via diff --git a/content/docs/data-modeling/objects.mdx b/content/docs/data-modeling/objects.mdx index 719247d9c3..529008d651 100644 --- a/content/docs/data-modeling/objects.mdx +++ b/content/docs/data-modeling/objects.mdx @@ -138,6 +138,73 @@ versioning: { } ``` +#### Data Lifecycle (Retention & Rotation) + +Declares how long the object's data lives and how its space is reclaimed +(ADR-0057). The platform **LifecycleService** (registered by +`ObjectQLPlugin`, default-on) sweeps declared policies hourly. Objects +without a `lifecycle` block keep permanent `record` semantics — nothing is +ever deleted. + +```typescript +// High-frequency telemetry: rotation window + age reap +lifecycle: { + class: 'telemetry', + retention: { maxAge: '14d' }, + storage: { strategy: 'rotation', shards: 14, unit: 'day' }, +} + +// Ephemeral rows: TTL on the natural expiry field +lifecycle: { + class: 'transient', + ttl: { field: 'expires_at', expireAfter: '1d' }, +} + +// Compliance ledger: hot window, then cold storage +lifecycle: { + class: 'audit', + retention: { maxAge: '90d' }, + archive: { after: '90d', to: 'archive', keep: '7y' }, +} +``` + +| Key | Description | +| :--- | :--- | +| `class` | Persistence contract (see table below); `record` is the implicit default | +| `retention.maxAge` | Age-based reap on `created_at` | +| `ttl` | Per-row expiry: `field` + `expireAfter` | +| `storage` | `{ strategy: 'rotation', shards, unit }` — time-shard + O(1) shard `DROP` (SQLite) | +| `archive` | Cold-store hand-off: `after` (must equal `retention.maxAge`) + `to` (datasource name) + optional `keep` | +| `reclaim` | Space reclamation after sweeps (default on for non-`record`) | + +| Class | Contract | Typical use | +| :--- | :--- | :--- | +| `record` | Business truth — permanent, no policies allowed | accounts, invoices | +| `audit` | Compliance ledger — retain → archive → delete | audit logs | +| `telemetry` | High-frequency log — rotation / short retention | activity streams, job runs | +| `transient` | Ephemeral state — TTL auto-expire | receipts, device codes | +| `event` | Bus messages — very short TTL (hours) | scheduled fan-out | + +Enforcement rules: + +- A non-`record` class **must** declare at least one bounding policy + (`retention`, `ttl`, or rotation `storage`) — rejected at parse time + otherwise. Policies on `record` are rejected too. +- Duration literals are `` + `h`/`d`/`w`/`y` (e.g. `'6h'`, `'14d'`, `'7y'`); + `archive.after` must equal `retention.maxAge`. +- An `archive`-declared object is **never** hot-deleted before the archive + copy succeeded. No datasource registered under the `archive.to` name ⇒ + rows are retained (safe default), not dropped. +- Registering a datasource named `telemetry` routes every + `telemetry`/`event`/`audit` object to it — separate storage, opt-in purely + by the datasource's existence. + +Operations knobs live in the `lifecycle` settings namespace: a runtime +`enabled` switch, tenant-scoped `retention_overrides` (a regulated tenant +sets years while dev keeps days), row `quotas` and `growth_alert_rows` +(observe-and-alert only). `OS_LIFECYCLE_DISABLED=1` disables sweeping +entirely. + ### Record Name Auto-generate unique record identifiers: diff --git a/content/docs/deployment/production-readiness.mdx b/content/docs/deployment/production-readiness.mdx index 36f720ae1c..50428a2e26 100644 --- a/content/docs/deployment/production-readiness.mdx +++ b/content/docs/deployment/production-readiness.mdx @@ -83,6 +83,12 @@ the [HARDENING.md recipes](https://github.com/objectstack-ai/framework/blob/main checklist in HARDENING.md). - [ ] Cross-tenant negative tests in CI. - [ ] Backup / restore drill documented and tested. +- [ ] Data-retention windows reviewed (ADR-0057): the platform's default + `lifecycle` declarations bound telemetry (activity 14d, job runs 30d, + notifications 90d, audit 90d-hot); extend per environment/tenant via + the `lifecycle.retention_overrides` setting **before** go-live if your + compliance regime needs longer, and register an `archive` datasource + if audit data must move to cold storage instead of being retained hot. ## What's NOT in the runtime (yet) diff --git a/content/docs/kernel/services.mdx b/content/docs/kernel/services.mdx index 737e80ee16..e786f930ac 100644 --- a/content/docs/kernel/services.mdx +++ b/content/docs/kernel/services.mdx @@ -90,6 +90,7 @@ The core ecosystem defines several standard service contracts: | `auth` | `IAuthService` | `plugin-auth` | | `api-registry` | `ApiRegistry` | `@objectstack/core` | | `cache` | `ICacheService` | Redis, Memcached, or in-memory | +| `lifecycle` | `LifecycleService` (`@objectstack/objectql`) | Registered by `ObjectQLPlugin` — enforces object `lifecycle` declarations (ADR-0057 retention/rotation/archival); call `sweep()` for an on-demand pass | The logger is not a registered service — it is exposed directly as `ctx.logger` (the `Logger` contract). Inter-plugin events also do not go through a service: diff --git a/skills/objectstack-data/SKILL.md b/skills/objectstack-data/SKILL.md index fe354ff8de..7eea84aff3 100644 --- a/skills/objectstack-data/SKILL.md +++ b/skills/objectstack-data/SKILL.md @@ -81,6 +81,7 @@ database table and exposes automatic CRUD APIs. | `titleFormat` | — | **Retired (ADR-0079)** — a render-only template the server can't return or query. Use `nameField`; for a composite title, designate a `returnType: 'text'` formula field as `nameField` | | `enable` | — | Capability flags (trackHistory, searchable, apiEnabled, etc.) | | `fieldGroups` | — | Ordered list of logical field groups for forms/detail pages (see [Field Groups](#field-groups-mvp)) | +| `lifecycle` | `record` semantics (permanent) | Data retention/rotation/archival contract (ADR-0057). **Required for append-only, high-write-rate objects** — a `telemetry`/`transient`/`event`/`audit` class must declare a bounding policy or parsing fails (see [Data Lifecycle & Retention](./rules/lifecycle.md)) | ### Object Capabilities (`enable`) @@ -205,6 +206,7 @@ For comprehensive documentation with incorrect/correct examples: - **[Relationships](./rules/relationships.md)** — lookup vs master_detail, junction patterns, delete behaviors - **[Validation Rules](./rules/validation.md)** — All validation types, script inversion, severity levels - **[Index Strategy](./rules/indexing.md)** — btree/gin/gist/fulltext, composite indexes, partial indexes +- **[Data Lifecycle & Retention](./rules/lifecycle.md)** — `lifecycle` classes (record/audit/telemetry/transient/event), retention/TTL/rotation/archive policies; ❗ append-only objects must declare one (distinct from lifecycle *hooks* below) - **[Lifecycle Hooks](./rules/hooks.md)** — Hook quick reference (→ see [references/data-hooks.md](./references/data-hooks.md) for the full 14-event guide) - **[Datasources & Federation](./rules/datasources.md)** — `defineDatasource`, external/federated objects (`remoteName`/`columnMap`), auto-connect gating, credentials; ❌ no `field.columnName` on external objects diff --git a/skills/objectstack-data/rules/lifecycle.md b/skills/objectstack-data/rules/lifecycle.md new file mode 100644 index 0000000000..aaeb87ee22 --- /dev/null +++ b/skills/objectstack-data/rules/lifecycle.md @@ -0,0 +1,129 @@ +# Data Lifecycle & Retention + +Guide for declaring how long an object's data lives and how its space is +reclaimed (ADR-0057). Not to be confused with **lifecycle hooks** +(`beforeInsert` / `afterUpdate` …) — those run business logic on data +operations; *this* page is about retention, rotation, and archival of the +rows themselves. + +## Why This Exists + +Every object without a `lifecycle` block is **immortal**: rows are never +deleted, and on SQLite the file never shrinks. That's correct for business +records — and a guaranteed disk leak for anything append-only. A scheduled +flow writing telemetry every 20 seconds once grew a dev database from 2 MB +to 260 MB with zero business data in it. + +**Rule: any append-only, high-write-rate object (event log, run history, +delivery outbox, ephemeral tokens) MUST declare a `lifecycle` block.** +The platform LifecycleService sweeps declared policies hourly, deletes +expired rows under a system context, and reclaims driver space. + +## Lifecycle Classes + +| Class | Contract | Typical use | +|:------|:---------|:------------| +| `record` | Business truth — permanent; policies FORBIDDEN | accounts, orders, invoices | +| `audit` | Compliance ledger — retain → archive → delete | audit trails | +| `telemetry` | High-frequency log — rotation / short retention | activity streams, run history | +| `transient` | Ephemeral state — TTL auto-expire | receipts, codes, sessions | +| `event` | Bus messages — very short TTL (hours) | scheduled fan-out | + +## Syntax + +```typescript +import { ObjectSchema, Field } from '@objectstack/spec/data'; + +// ✅ Telemetry stream: bounded by a rotation window +export const ApiCallLog = ObjectSchema.create({ + name: 'api_call_log', + lifecycle: { + class: 'telemetry', + retention: { maxAge: '14d' }, // reap past 14 days (created_at) + storage: { strategy: 'rotation', shards: 14, unit: 'day' }, // O(1) shard DROP on SQLite + }, + fields: { + endpoint: Field.text({}), + status: Field.number({}), + }, +}); + +// ✅ Ephemeral token: TTL on its own expiry field +export const ImportTicket = ObjectSchema.create({ + name: 'import_ticket', + lifecycle: { + class: 'transient', + ttl: { field: 'expires_at', expireAfter: '1d' }, // reap 1 day after expires_at + }, + fields: { + expires_at: Field.datetime({}), + }, +}); + +// ✅ Compliance ledger: hot window, then cold storage +export const ConsentLog = ObjectSchema.create({ + name: 'consent_log', + lifecycle: { + class: 'audit', + retention: { maxAge: '90d' }, + archive: { after: '90d', to: 'archive', keep: '7y' }, // must: after === maxAge + }, + fields: { + subject: Field.text({}), + }, +}); +``` + +Duration literals: `` + `h` / `d` / `w` / `y` — e.g. `'6h'`, `'14d'`, +`'12w'`, `'7y'`. + +## Validation Rules (rejected at parse time) + +```typescript +// ❌ Non-record class with no bounding policy — grows forever, exactly the bug +lifecycle: { class: 'telemetry' } + +// ❌ Policies on record-class — business truth is permanent +lifecycle: { class: 'record', retention: { maxAge: '30d' } } + +// ❌ Archive window detached from the hot window +lifecycle: { class: 'audit', retention: { maxAge: '90d' }, archive: { after: '30d', to: 'archive' } } + +// ❌ Free-form durations +lifecycle: { class: 'telemetry', retention: { maxAge: '2 weeks' } } // use '2w' +``` + +## Safety Semantics + +- **No `lifecycle` block = today's behavior.** Nothing is deleted. `record` + is the implicit default. +- **Archive-then-delete is atomic-ish and safe by default**: an object + declaring `archive` is never hot-deleted before the copy to the archive + datasource succeeded. If no datasource is registered under the `archive.to` + name, rows are simply retained — a compliance ledger cannot be destroyed + by declaring a lifecycle. +- **Rotation** physically time-shards the table on SQLite (writes hit the + current shard, reads go through a view under the object's name, expired + shards are DROPped). Other dialects enforce the same window with an + age-based reap. +- **Separation**: registering a datasource named `telemetry` moves every + `telemetry`/`event`/`audit` object onto it — telemetry bloat can't touch + the business DB. + +## Operations + +| Knob | Where | Effect | +|:-----|:------|:-------| +| `enabled` | `lifecycle` settings namespace | Runtime master switch | +| `retention_overrides` | settings, **tenant-scoped** | Per-object window overrides — a regulated tenant sets `'2y'` while dev keeps days | +| `quotas` / `quota_defaults` | settings | Row ceilings; breaches ALERT (never delete beyond declared policy) | +| `growth_alert_rows` | settings | Per-sweep growth spike alert | +| `OS_LIFECYCLE_DISABLED=1` | env | Disables sweeping entirely | + +## Decision Tree + +1. Users create/edit it and it represents business state? → `record` (omit the block). +2. Is it a compliance/audit trail? → `audit` + `retention`, add `archive` if cold storage exists. +3. Written by machines at high frequency, value decays in days/weeks? → `telemetry` + `retention` (add rotation `storage` for the hottest tables). +4. Meaningless after a deadline (tokens, receipts, read-state)? → `transient` + `ttl` on the natural expiry field. +5. Bus/fan-out messages? → `event` + a short `ttl` (hours).