Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions content/docs/data-modeling/drivers.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,19 @@ export OS_DATABASE_URL=file:./data/app.db
export OS_DATABASE_URL=:memory:
```

### Space Reclamation & Table Rotation (ADR-0057)

The SQL driver connects SQLite with `auto_vacuum=INCREMENTAL` and exposes
`reclaimSpace()` (`PRAGMA incremental_vacuum`), which the platform
LifecycleService calls after every sweep that deleted rows — so the database
file actually shrinks instead of pinning at its high-water mark. Objects
declaring `lifecycle.storage.strategy: 'rotation'` are physically
time-sharded on SQLite: writes land in the current shard, reads go through a
view under the table's name, and expired shards are reclaimed with a single
`DROP TABLE`. On PostgreSQL/MySQL both are no-ops — those engines manage
space with their own vacuum machinery, and rotation falls back to an
equivalent age-based reap.

## Memory Driver

The in-memory driver keeps records in plain in-process objects (queried via
Expand Down
67 changes: 67 additions & 0 deletions content/docs/data-modeling/objects.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,73 @@ versioning: {
}
```

#### Data Lifecycle (Retention & Rotation)

Declares how long the object's data lives and how its space is reclaimed
(ADR-0057). The platform **LifecycleService** (registered by
`ObjectQLPlugin`, default-on) sweeps declared policies hourly. Objects
without a `lifecycle` block keep permanent `record` semantics — nothing is
ever deleted.

```typescript
// High-frequency telemetry: rotation window + age reap
lifecycle: {
class: 'telemetry',
retention: { maxAge: '14d' },
storage: { strategy: 'rotation', shards: 14, unit: 'day' },
}

// Ephemeral rows: TTL on the natural expiry field
lifecycle: {
class: 'transient',
ttl: { field: 'expires_at', expireAfter: '1d' },
}

// Compliance ledger: hot window, then cold storage
lifecycle: {
class: 'audit',
retention: { maxAge: '90d' },
archive: { after: '90d', to: 'archive', keep: '7y' },
}
```

| Key | Description |
| :--- | :--- |
| `class` | Persistence contract (see table below); `record` is the implicit default |
| `retention.maxAge` | Age-based reap on `created_at` |
| `ttl` | Per-row expiry: `field` + `expireAfter` |
| `storage` | `{ strategy: 'rotation', shards, unit }` — time-shard + O(1) shard `DROP` (SQLite) |
| `archive` | Cold-store hand-off: `after` (must equal `retention.maxAge`) + `to` (datasource name) + optional `keep` |
| `reclaim` | Space reclamation after sweeps (default on for non-`record`) |

| Class | Contract | Typical use |
| :--- | :--- | :--- |
| `record` | Business truth — permanent, no policies allowed | accounts, invoices |
| `audit` | Compliance ledger — retain → archive → delete | audit logs |
| `telemetry` | High-frequency log — rotation / short retention | activity streams, job runs |
| `transient` | Ephemeral state — TTL auto-expire | receipts, device codes |
| `event` | Bus messages — very short TTL (hours) | scheduled fan-out |

Enforcement rules:

- A non-`record` class **must** declare at least one bounding policy
(`retention`, `ttl`, or rotation `storage`) — rejected at parse time
otherwise. Policies on `record` are rejected too.
- Duration literals are `<n>` + `h`/`d`/`w`/`y` (e.g. `'6h'`, `'14d'`, `'7y'`);
`archive.after` must equal `retention.maxAge`.
- An `archive`-declared object is **never** hot-deleted before the archive
copy succeeded. No datasource registered under the `archive.to` name ⇒
rows are retained (safe default), not dropped.
- Registering a datasource named `telemetry` routes every
`telemetry`/`event`/`audit` object to it — separate storage, opt-in purely
by the datasource's existence.

Operations knobs live in the `lifecycle` settings namespace: a runtime
`enabled` switch, tenant-scoped `retention_overrides` (a regulated tenant
sets years while dev keeps days), row `quotas` and `growth_alert_rows`
(observe-and-alert only). `OS_LIFECYCLE_DISABLED=1` disables sweeping
entirely.

### Record Name

Auto-generate unique record identifiers:
Expand Down
6 changes: 6 additions & 0 deletions content/docs/deployment/production-readiness.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,12 @@ the [HARDENING.md recipes](https://github.com/objectstack-ai/framework/blob/main
checklist in HARDENING.md).
- [ ] Cross-tenant negative tests in CI.
- [ ] Backup / restore drill documented and tested.
- [ ] Data-retention windows reviewed (ADR-0057): the platform's default
`lifecycle` declarations bound telemetry (activity 14d, job runs 30d,
notifications 90d, audit 90d-hot); extend per environment/tenant via
the `lifecycle.retention_overrides` setting **before** go-live if your
compliance regime needs longer, and register an `archive` datasource
if audit data must move to cold storage instead of being retained hot.

## What's NOT in the runtime (yet)

Expand Down
1 change: 1 addition & 0 deletions content/docs/kernel/services.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@ The core ecosystem defines several standard service contracts:
| `auth` | `IAuthService` | `plugin-auth` |
| `api-registry` | `ApiRegistry` | `@objectstack/core` |
| `cache` | `ICacheService` | Redis, Memcached, or in-memory |
| `lifecycle` | `LifecycleService` (`@objectstack/objectql`) | Registered by `ObjectQLPlugin` — enforces object `lifecycle` declarations (ADR-0057 retention/rotation/archival); call `sweep()` for an on-demand pass |

The logger is not a registered service — it is exposed directly as `ctx.logger`
(the `Logger` contract). Inter-plugin events also do not go through a service:
Expand Down
2 changes: 2 additions & 0 deletions skills/objectstack-data/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@ database table and exposes automatic CRUD APIs.
| `titleFormat` | — | **Retired (ADR-0079)** — a render-only template the server can't return or query. Use `nameField`; for a composite title, designate a `returnType: 'text'` formula field as `nameField` |
| `enable` | — | Capability flags (trackHistory, searchable, apiEnabled, etc.) |
| `fieldGroups` | — | Ordered list of logical field groups for forms/detail pages (see [Field Groups](#field-groups-mvp)) |
| `lifecycle` | `record` semantics (permanent) | Data retention/rotation/archival contract (ADR-0057). **Required for append-only, high-write-rate objects** — a `telemetry`/`transient`/`event`/`audit` class must declare a bounding policy or parsing fails (see [Data Lifecycle & Retention](./rules/lifecycle.md)) |

### Object Capabilities (`enable`)

Expand Down Expand Up @@ -205,6 +206,7 @@ For comprehensive documentation with incorrect/correct examples:
- **[Relationships](./rules/relationships.md)** — lookup vs master_detail, junction patterns, delete behaviors
- **[Validation Rules](./rules/validation.md)** — All validation types, script inversion, severity levels
- **[Index Strategy](./rules/indexing.md)** — btree/gin/gist/fulltext, composite indexes, partial indexes
- **[Data Lifecycle & Retention](./rules/lifecycle.md)** — `lifecycle` classes (record/audit/telemetry/transient/event), retention/TTL/rotation/archive policies; ❗ append-only objects must declare one (distinct from lifecycle *hooks* below)
- **[Lifecycle Hooks](./rules/hooks.md)** — Hook quick reference (→ see [references/data-hooks.md](./references/data-hooks.md) for the full 14-event guide)
- **[Datasources & Federation](./rules/datasources.md)** — `defineDatasource`, external/federated objects (`remoteName`/`columnMap`), auto-connect gating, credentials; ❌ no `field.columnName` on external objects

Expand Down
129 changes: 129 additions & 0 deletions skills/objectstack-data/rules/lifecycle.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
# Data Lifecycle & Retention

Guide for declaring how long an object's data lives and how its space is
reclaimed (ADR-0057). Not to be confused with **lifecycle hooks**
(`beforeInsert` / `afterUpdate` …) — those run business logic on data
operations; *this* page is about retention, rotation, and archival of the
rows themselves.

## Why This Exists

Every object without a `lifecycle` block is **immortal**: rows are never
deleted, and on SQLite the file never shrinks. That's correct for business
records — and a guaranteed disk leak for anything append-only. A scheduled
flow writing telemetry every 20 seconds once grew a dev database from 2 MB
to 260 MB with zero business data in it.

**Rule: any append-only, high-write-rate object (event log, run history,
delivery outbox, ephemeral tokens) MUST declare a `lifecycle` block.**
The platform LifecycleService sweeps declared policies hourly, deletes
expired rows under a system context, and reclaims driver space.

## Lifecycle Classes

| Class | Contract | Typical use |
|:------|:---------|:------------|
| `record` | Business truth — permanent; policies FORBIDDEN | accounts, orders, invoices |
| `audit` | Compliance ledger — retain → archive → delete | audit trails |
| `telemetry` | High-frequency log — rotation / short retention | activity streams, run history |
| `transient` | Ephemeral state — TTL auto-expire | receipts, codes, sessions |
| `event` | Bus messages — very short TTL (hours) | scheduled fan-out |

## Syntax

```typescript
import { ObjectSchema, Field } from '@objectstack/spec/data';

// ✅ Telemetry stream: bounded by a rotation window
export const ApiCallLog = ObjectSchema.create({
name: 'api_call_log',
lifecycle: {
class: 'telemetry',
retention: { maxAge: '14d' }, // reap past 14 days (created_at)
storage: { strategy: 'rotation', shards: 14, unit: 'day' }, // O(1) shard DROP on SQLite
},
fields: {
endpoint: Field.text({}),
status: Field.number({}),
},
});

// ✅ Ephemeral token: TTL on its own expiry field
export const ImportTicket = ObjectSchema.create({
name: 'import_ticket',
lifecycle: {
class: 'transient',
ttl: { field: 'expires_at', expireAfter: '1d' }, // reap 1 day after expires_at
},
fields: {
expires_at: Field.datetime({}),
},
});

// ✅ Compliance ledger: hot window, then cold storage
export const ConsentLog = ObjectSchema.create({
name: 'consent_log',
lifecycle: {
class: 'audit',
retention: { maxAge: '90d' },
archive: { after: '90d', to: 'archive', keep: '7y' }, // must: after === maxAge
},
fields: {
subject: Field.text({}),
},
});
```

Duration literals: `<n>` + `h` / `d` / `w` / `y` — e.g. `'6h'`, `'14d'`,
`'12w'`, `'7y'`.

## Validation Rules (rejected at parse time)

```typescript
// ❌ Non-record class with no bounding policy — grows forever, exactly the bug
lifecycle: { class: 'telemetry' }

// ❌ Policies on record-class — business truth is permanent
lifecycle: { class: 'record', retention: { maxAge: '30d' } }

// ❌ Archive window detached from the hot window
lifecycle: { class: 'audit', retention: { maxAge: '90d' }, archive: { after: '30d', to: 'archive' } }

// ❌ Free-form durations
lifecycle: { class: 'telemetry', retention: { maxAge: '2 weeks' } } // use '2w'
```

## Safety Semantics

- **No `lifecycle` block = today's behavior.** Nothing is deleted. `record`
is the implicit default.
- **Archive-then-delete is atomic-ish and safe by default**: an object
declaring `archive` is never hot-deleted before the copy to the archive
datasource succeeded. If no datasource is registered under the `archive.to`
name, rows are simply retained — a compliance ledger cannot be destroyed
by declaring a lifecycle.
- **Rotation** physically time-shards the table on SQLite (writes hit the
current shard, reads go through a view under the object's name, expired
shards are DROPped). Other dialects enforce the same window with an
age-based reap.
- **Separation**: registering a datasource named `telemetry` moves every
`telemetry`/`event`/`audit` object onto it — telemetry bloat can't touch
the business DB.

## Operations

| Knob | Where | Effect |
|:-----|:------|:-------|
| `enabled` | `lifecycle` settings namespace | Runtime master switch |
| `retention_overrides` | settings, **tenant-scoped** | Per-object window overrides — a regulated tenant sets `'2y'` while dev keeps days |
| `quotas` / `quota_defaults` | settings | Row ceilings; breaches ALERT (never delete beyond declared policy) |
| `growth_alert_rows` | settings | Per-sweep growth spike alert |
| `OS_LIFECYCLE_DISABLED=1` | env | Disables sweeping entirely |

## Decision Tree

1. Users create/edit it and it represents business state? → `record` (omit the block).
2. Is it a compliance/audit trail? → `audit` + `retention`, add `archive` if cold storage exists.
3. Written by machines at high frequency, value decays in days/weeks? → `telemetry` + `retention` (add rotation `storage` for the hottest tables).
4. Meaningless after a deadline (tokens, receipts, read-state)? → `transient` + `ttl` on the natural expiry field.
5. Bus/fan-out messages? → `event` + a short `ttl` (hours).