diff --git a/docs-site/src/content/docs/guides/web-dashboard.md b/docs-site/src/content/docs/guides/web-dashboard.md index 7e86901233..c8cdef3f1d 100644 --- a/docs-site/src/content/docs/guides/web-dashboard.md +++ b/docs-site/src/content/docs/guides/web-dashboard.md @@ -92,7 +92,7 @@ badge or the version value to read the full value. | **Subagents** | Feature up to five bare native or namespaced routed models in the `spawn_agent` override list. | | **Models** | Toggle native GPT and routed models, set provider allowlists and context caps, choose v1/base/v2, and configure the v2 thread limit. Configured providers stay visible as zero-model groups when discovery is off or returns no rows. | | **Logs** | Auto-refresh recent requests with tokens, requested effort and (when available) effective outbound effort, resolved model, provider, status, request id, duration, and error details. The detail view includes the exact reasoning wire field when the adapter emits one. Filter by opaque conversation/session id (when the client sends one) to total tokens and estimated list-price cost for the currently loaded Logs ring. | -| **Usage / Debug** | Inspect token-usage coverage and trends, or enable opt-in provider transport and usage-extraction diagnostics. | +| **Usage / Debug** | Inspect token-usage coverage and trends. The **Usage** page also exposes the opt-in size ceiling for `usage.jsonl` (`usageLedgerRetention.enabled` / `maxBytes`); the background scheduler compacts older complete rows when the ceiling is exceeded. Enable provider transport and usage-extraction diagnostics here as needed. | | **Storage** | Read-only CODEX_HOME disk breakdown (sessions, archives, DBs, attachments). Optional archived cleanup: preview the oldest N%, then quarantine to `CODEX_HOME/.trash` (default) or permanently delete behind an explicit checkbox. **Auto-cleanup policy** is opt-in and **default OFF** (`storageCleanupPolicy.enabled`); configure threshold/target/schedule/mode on the Storage page, or trigger **Run now**. Quarantined entries can be restored from the Storage page (JSONL + threads). Active sessions stay read-only. Cleanup and restore are refused while Codex holds the newest/active `state_*.sqlite` locked. | | **Stop** | Gracefully stop the proxy and installed background service, restore native Codex, and exit (`POST /api/stop`). On Windows with the Task Scheduler backend the dashboard refuses and asks you to run `ocx stop` instead: that wrapper can respawn the proxy after the task ends, and only a stop running outside this process can verify the restart window before restoring your client config. Nothing is changed when it refuses. | diff --git a/docs-site/src/content/docs/reference/cli/agents.md b/docs-site/src/content/docs/reference/cli/agents.md index ab96881d11..b8dd4ab8dc 100644 --- a/docs-site/src/content/docs/reference/cli/agents.md +++ b/docs-site/src/content/docs/reference/cli/agents.md @@ -162,6 +162,21 @@ separately, and requests with no matching price row are counted as ocx usage --range today --provider xai ``` +### `ocx storage usage-limit` + +Inspect or change the opt-in `usage.jsonl` size ceiling. The setting is also available in the +dashboard on the **Usage** page. + +```bash +ocx storage usage-limit show --json +ocx storage usage-limit set --enabled true --mib 512 --json +``` + +`set` sends only the fields supplied, so changing `--mib` preserves the saved enabled state. +The minimum ceiling is 1 MiB. A bare `usage-limit` invocation is read-only. The background +scheduler compacts complete JSONL rows after the ledger exceeds the configured ceiling. The +command drives `GET`/`PUT /api/storage/usage-ledger-retention` on the running proxy. + ### `ocx debug ` Read or change runtime debug overrides through the running proxy's management API. diff --git a/docs-site/src/content/docs/reference/configuration/server.md b/docs-site/src/content/docs/reference/configuration/server.md index d4cd1e5625..899e5373ba 100644 --- a/docs-site/src/content/docs/reference/configuration/server.md +++ b/docs-site/src/content/docs/reference/configuration/server.md @@ -23,6 +23,7 @@ runs helper features around provider requests. | `corsAllowOrigins?` | `string[]` | `[]` | Additional exact origins allowed by CORS. Loopback origins are always allowed. Authority-based browser extension origins such as `chrome-extension://` are supported; `*` is not a wildcard. Firefox and Safari regenerate the extension UUID (per install / per browser launch), so update the entry when the origin changes. | | `apiKeys?` | `OcxApiKey[]` | `[]` | Generated `ocx_…` credentials accepted by management and data-plane auth on non-loopback binds. Dashboard-managed. | | `storageCleanupPolicy?` | `StorageCleanupPolicy` | disabled | Opt-in archived-session cleanup policy. Never enabled implicitly. | +| `usageLedgerRetention?` | `UsageLedgerRetentionConfig` | disabled | Opt-in cap for the append-only `usage.jsonl` usage ledger. When enabled, the background scheduler compacts complete rows after the ledger exceeds `maxBytes`. | | `appOwnedMemoryBudgetMb?` | `number` | `256` | Cap in MiB for evictable app-owned logs, caches, blobs, and continuation payloads. Range 64–4096; not an RSS cap. | | `codexAutoStart?` | `boolean` | `true` | Let the Codex shim run `ocx ensure` before launching Codex. False makes ensure a no-op. | | `codexShimAutoRestore?` | `boolean` | `true` | Restore an installed shim after a completed external Codex update replaces it. Environment opt-out: `OPENCODEX_CODEX_SHIM_AUTO_RESTORE=0`. | @@ -314,6 +315,38 @@ either `target.reduceToBytes` or `target.removeOldestPercent`. `mode` defaults t Configure it on the Storage page or with `GET`/`PUT /api/storage/cleanup-policy`; trigger a manual run with `POST /api/storage/cleanup-policy/run`. +## Usage-ledger retention + +`usageLedgerRetention` is disabled by default. It is an opt-in size ceiling for +`$OPENCODEX_HOME/usage.jsonl`, the append-only usage ledger used by the Usage page and +`GET /api/usage`. Enabling it lets the proxy compact older rows in a background Worker once +the file exceeds the configured limit; normal request handling is not blocked by the scan. + +```json +{ + "usageLedgerRetention": { + "enabled": false + } +} +``` + +The effective default is **Unlimited** (`enabled: false`). An unconfigured installation remembers +**1 GiB** (`1073741824`) as `maxBytes`; that saved value becomes effective on first enable. Set +`maxBytes` explicitly only when selecting a different ceiling. Any explicit value must be a safe +integer of at least 1 MiB (`1048576`). A saved ceiling is retained when `enabled` is set to `false`, +so an operator can pause retention without losing the selected limit. Unknown keys and malformed +values fail closed and leave retention disabled. + +Compaction publishes a complete JSONL-row candidate only after the source revision and active-turn +checks still match. An unterminated crash tail is discarded; a single row larger than the ceiling +is dropped so the published ledger remains bounded. The derived request-history projection is +recreated after a successful publish. A policy change invalidates an in-flight candidate, and a +source append during scanning defers the commit for a later run. + +The dashboard exposes this control on the **Usage** page. For headless operation, use +`ocx storage usage-limit` or the `GET`/`PUT` management routes below. Setting the policy is +non-destructive; the background scheduler compacts older rows when the ceiling is exceeded. + ## Quota-reset notifications (`quotaResetNotify`) Off by default. When the section is absent, no detection runs, no timer starts, and no state diff --git a/docs-site/src/content/docs/reference/management-api.md b/docs-site/src/content/docs/reference/management-api.md index b7fdbf072f..a18b585e07 100644 --- a/docs-site/src/content/docs/reference/management-api.md +++ b/docs-site/src/content/docs/reference/management-api.md @@ -184,6 +184,8 @@ by the current window size. | `GET /api/claude/inbound-debug` | Read Claude inbound debug state and entries | — | | `GET /api/usage` | Stream the complete usage ledger into compact aggregates, then incrementally fold verified appends; summarize by preset or inclusive custom window and client surface, with a Codex `accounts` breakdown keyed by stable non-PII log labels | 400 invalid custom bounds; returns an `error: "read_failed"` summary if storage cannot be read | | `GET /api/storage` | Scan Codex storage usage by bucket | Returns an `error: "scan_failed"` payload on scan failure | +| `GET /api/storage/usage-ledger-retention` | Read the usage-ledger retention policy, current `usage.jsonl` size, over-limit state, and the last background job state | — | +| `PUT /api/storage/usage-ledger-retention` | Replace the retention fields supplied in `{ "enabled"?: boolean, "maxBytes"?: integer }`; omitted fields keep their saved values | 400 malformed body, unknown field, or `maxBytes` below 1 MiB; 500 `config_write_failed` | | `POST /api/storage/cleanup/preview` | Preview archived-session cleanup and return a binding digest | 400 `invalid_json` or `invalid_percent` | | `POST /api/storage/cleanup` | Quarantine or permanently remove the previewed archived set | 400 invalid input; 409 stale/busy/referenced state; 500 filesystem/database failure | | `GET /api/storage/trash` | List quarantined cleanup entries | 500 `trash_list_failed` | @@ -193,6 +195,13 @@ by the current window size. | `POST /api/storage/cleanup-policy/run` | Start a manual cleanup-policy run | 409 `already_running`; 500 `cleanup_failed` | | `GET /api/storage/cleanup-policy/test-stream` | Test-only policy stream hook | 404 `not_found` when unavailable | +The retention status response is shaped as `{ enabled, maxBytes, currentBytes, overLimit, job }`; +`job` reports the process-local background state (`idle` or `running`) and the last outcome when +one exists. `PUT` accepts only `enabled` and `maxBytes`, and merges the supplied fields with the +saved policy. It never starts a compaction by itself. The background scheduler queues a Worker +when the ledger exceeds the ceiling; the canonical ledger is replaced only after complete-row, +active-turn, and source-revision checks pass. + New xAI attempts in `usage.jsonl` include a request-time `credentialSource`: `grok-oauth` for the resolved Grok CLI OAuth transport, or `xai-api-key` for the public xAI API key transport. This fixed label contains no credential or account identifier. It belongs to diff --git a/gui/src/components/usage/UsageLedgerRetentionControl.tsx b/gui/src/components/usage/UsageLedgerRetentionControl.tsx new file mode 100644 index 0000000000..911474bcca --- /dev/null +++ b/gui/src/components/usage/UsageLedgerRetentionControl.tsx @@ -0,0 +1,243 @@ +import { useCallback, useEffect, useRef, useState } from "react"; +import { clampNumberDraft } from "../../clamp-draft"; +import { formatBytes } from "../../format-bytes"; +import { useI18n } from "../../i18n/shared"; +import { Switch } from "../../ui"; +import { NumberStepper } from "../NumberStepper"; + +const MIB = 1024 ** 2; +const MAX_MIB = Math.floor(Number.MAX_SAFE_INTEGER / MIB); + +interface RetentionStatus { + enabled: boolean; + maxBytes: number; + currentBytes?: number; +} + +function parseStatus(value: unknown): RetentionStatus { + if (!value || typeof value !== "object" || Array.isArray(value)) throw new Error("invalid_status"); + const candidate = value as Record; + if (typeof candidate.enabled !== "boolean" || typeof candidate.maxBytes !== "number" + || !Number.isFinite(candidate.maxBytes) || candidate.maxBytes <= 0) { + throw new Error("invalid_status"); + } + return { + enabled: candidate.enabled, + maxBytes: candidate.maxBytes, + currentBytes: typeof candidate.currentBytes === "number" && Number.isFinite(candidate.currentBytes) + ? candidate.currentBytes + : undefined, + }; +} + +function formatMaxMiBDraft(maxBytes: number): string { + return Number.isFinite(maxBytes) && maxBytes > 0 ? String(maxBytes / MIB) : ""; +} + +function parseMaxMiBDraft(raw: string): number | null { + const mib = Number(raw.trim()); + if (!Number.isSafeInteger(mib) || mib < 1 || mib > MAX_MIB) return null; + const bytes = mib * MIB; + return Number.isSafeInteger(bytes) ? bytes : null; +} + +/** + * Usage-page control for the opt-in usage-ledger byte ceiling. + * + * The dashboard keeps the switch as the primary action and exposes a MiB-aligned + * custom editor only while the policy is enabled. Toggling the switch always + * sends the exact server-reported byte value, so existing non-MiB-aligned values + * can never be rounded or silently rewritten. + */ +export default function UsageLedgerRetentionControl({ apiBase }: { apiBase: string }) { + const { locale, t } = useI18n(); + const mibLabel = formatBytes(MIB, locale).replace(/^[\d.,]+\s*/, ""); + const [status, setStatus] = useState(null); + const [customDraft, setCustomDraft] = useState(""); + const [editing, setEditing] = useState(false); + const [busy, setBusy] = useState(false); + const [error, setError] = useState(null); + const loadGeneration = useRef(0); + const inputRef = useRef(null); + + const load = useCallback(async (signal?: AbortSignal) => { + const generation = ++loadGeneration.current; + try { + const response = await fetch(`${apiBase}/api/storage/usage-ledger-retention`, { signal }); + if (!response.ok) throw new Error("load_failed"); + const next = parseStatus(await response.json()); + if (signal?.aborted || generation !== loadGeneration.current) return; + setError(null); + setStatus(next); + setCustomDraft(formatMaxMiBDraft(next.maxBytes)); + } catch (errorValue) { + // A successful PUT invalidates reads that started under the old policy. Stale reads + // must be silent whether they eventually succeed, fail HTTP, reject, or parse badly. + if (signal?.aborted || generation !== loadGeneration.current + || (errorValue as { name?: string })?.name === "AbortError") return; + throw errorValue; + } + }, [apiBase]); + + useEffect(() => { + const controller = new AbortController(); + const timeout = window.setTimeout(() => { + void load(controller.signal).catch(errorValue => { + if (!controller.signal.aborted && (errorValue as { name?: string })?.name !== "AbortError") { + setError(t("usage.retention.error")); + } + }); + }, 0); + return () => { + window.clearTimeout(timeout); + controller.abort(); + }; + }, [load, t]); + + const persist = useCallback(async (nextEnabled: boolean, maxBytes: number) => { + setBusy(true); + setError(null); + try { + const response = await fetch(`${apiBase}/api/storage/usage-ledger-retention`, { + method: "PUT", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ enabled: nextEnabled, maxBytes }), + }); + if (!response.ok) throw new Error("save_failed"); + const next = parseStatus(await response.json()); + // A GET may have started before this authoritative mutation completed (for example, + // after a locale change). Do not let that older snapshot repaint the saved state. + loadGeneration.current += 1; + setStatus(next); + setCustomDraft(formatMaxMiBDraft(next.maxBytes)); + setEditing(false); + } catch { + setError(t("usage.retention.error")); + } finally { + setBusy(false); + } + }, [apiBase, t]); + + const toggle = () => { + if (!status || busy) return; + void persist(!status.enabled, status.maxBytes); + }; + + const saveCustom = () => { + if (!status || busy) return; + // If the saved value is not MiB-aligned, leaving the field untouched must + // not force an unrelated rounding write; the switch path remains exact. + if (customDraft === formatMaxMiBDraft(status.maxBytes)) { + setEditing(false); + return; + } + const nextMaxBytes = parseMaxMiBDraft(customDraft); + if (nextMaxBytes === null) { + setError(t("usage.retention.error")); + return; + } + void persist(true, nextMaxBytes); + }; + + const resetCustom = () => { + if (!status) return; + setCustomDraft(formatMaxMiBDraft(status.maxBytes)); + setEditing(false); + setError(null); + }; + + return ( +
+
+
+

{t("usage.retention.title")}

+

{t("usage.retention.help")}

+
+ +
+ + {status?.enabled && ( +
+ +
+ )} + +

+ {t("usage.retention.current")}: {status?.currentBytes === undefined ? "—" : formatBytes(status.currentBytes, locale)} + {status && ( + <> + {" · "} + {!status.enabled && <>{t("usage.retention.unlimited")}{" · "}} + + {t("usage.retention.limit")}: {formatBytes(status.maxBytes, locale)} + + + )} +

+ {error &&

{error}

} +
+ ); +} diff --git a/gui/src/i18n/catalogs.ts b/gui/src/i18n/catalogs.ts index 5bb16d1320..461f97448c 100644 --- a/gui/src/i18n/catalogs.ts +++ b/gui/src/i18n/catalogs.ts @@ -1,4 +1,4 @@ -import { en, type TKey } from "./en"; +import { en, type TKey as BaseTKey } from "./en"; import { de } from "./de"; import { fr } from "./fr"; import { ko } from "./ko"; @@ -11,26 +11,30 @@ import { LAB_CATALOG_OVERRIDES, type LabLocale } from "./lab-translations"; /** React-free locale catalog registry for formatters and other shared helpers. */ export type Locale = LabLocale; +export type TKey = BaseTKey; -function withLabTranslations(locale: Locale, catalog: Record): Record { - return { ...catalog, ...LAB_CATALOG_OVERRIDES[locale] }; +/** Apply the centrally maintained Lab closed-surface translations to one base locale catalog. */ +function withCatalogOverlays(locale: Locale, catalog: Record): Record { + return { + ...catalog, + ...LAB_CATALOG_OVERRIDES[locale], + }; } /** - * CL-05 translations are overlaid centrally so the compatibility surface cannot regress to - * copied English values in a locale catalog. The locale parity test still validates the base - * catalogs; this overlay is deliberately limited to the closed `lab.*` namespace. + * Lab translations are overlaid centrally so the compatibility surface cannot regress to copied + * English values. Base locale parity remains compile-checked by the locale modules. */ export const DICTS: Record> = { - en: withLabTranslations("en", en), - de: withLabTranslations("de", de), - fr: withLabTranslations("fr", fr), - ko: withLabTranslations("ko", ko), - zh: withLabTranslations("zh", zh), - "zh-TW": withLabTranslations("zh-TW", zhTW), - ru: withLabTranslations("ru", ru), - ja: withLabTranslations("ja", ja), - tr: withLabTranslations("tr", tr), + en: withCatalogOverlays("en", en), + de: withCatalogOverlays("de", de), + fr: withCatalogOverlays("fr", fr), + ko: withCatalogOverlays("ko", ko), + zh: withCatalogOverlays("zh", zh), + "zh-TW": withCatalogOverlays("zh-TW", zhTW), + ru: withCatalogOverlays("ru", ru), + ja: withCatalogOverlays("ja", ja), + tr: withCatalogOverlays("tr", tr), }; /** Native language names shown by the language picker, kept inside i18n rather than UI metadata. */ @@ -38,8 +42,7 @@ export function localeDisplayName(locale: Locale): string { return DICTS[locale]["lang.nativeName"]; } +/** Read one localized string without requiring React context. */ export function catalogValue(locale: Locale, key: TKey): string { return DICTS[locale][key]; } - -export type { TKey }; diff --git a/gui/src/i18n/de.ts b/gui/src/i18n/de.ts index 68363152b5..20d39eccb0 100644 --- a/gui/src/i18n/de.ts +++ b/gui/src/i18n/de.ts @@ -869,6 +869,16 @@ export const de: Record = { "debug.noLines.usage": "Nutzungserfassung ist an, aber es wurde noch nichts erfasst. Sende einen Chat/eine Anfrage über Codex, dann erscheint es hier.", "debug.noLines.injection": "Injektions-Log ist an, aber es wurde noch nichts erfasst. Es erfasst Multi-Agent-Guidance-Injektion und Effort-Cap-Entscheidungen bei Collab- und Sub-Agent-Turns.", "usage.title": "Nutzung", + "usage.retention.title": "Größenlimit für Nutzungsverlauf", + "usage.retention.help": "Optional können die neuesten vollständigen Nutzungsdatensätze innerhalb eines Größenlimits behalten werden. Ältere Einträge werden automatisch entfernt, sobald der Verlauf das Limit überschreitet.", + "usage.retention.enabled": "Größe des Nutzungsverlaufs begrenzen", + "usage.retention.current": "Aktuelle Größe", + "usage.retention.limit": "Maximale Größe", + "usage.retention.increase": "Maximale Größe erhöhen", + "usage.retention.decrease": "Maximale Größe verringern", + "usage.retention.unlimited": "Unbegrenzt", + "usage.retention.error": "Das Größenlimit für den Nutzungsverlauf konnte nicht aktualisiert werden.", + "usage.retention.disabled": "Unbegrenzt — automatische Verlaufskomprimierung ist deaktiviert.", "usage.subtitle": "Lokale Token-Buchhaltung deines Proxys. Fehlende Nutzung wird nie als Null angezeigt.", "usage.loading": "Lade Nutzungsdaten…", "usage.empty": "Noch keine Nutzung erfasst. Sende eine Anfrage über den Proxy, um Aktivität hier zu sehen.", diff --git a/gui/src/i18n/en.ts b/gui/src/i18n/en.ts index 89fc5ec0d7..e40d0b1e19 100644 --- a/gui/src/i18n/en.ts +++ b/gui/src/i18n/en.ts @@ -922,6 +922,16 @@ export const en = { // usage page "usage.title": "Usage", + "usage.retention.title": "Usage history size limit", + "usage.retention.help": "Optionally keep the newest complete usage records within a size limit. Older rows are removed automatically when the ledger exceeds it.", + "usage.retention.enabled": "Limit usage history size", + "usage.retention.current": "Current size", + "usage.retention.limit": "Maximum size", + "usage.retention.increase": "Increase maximum size", + "usage.retention.decrease": "Decrease maximum size", + "usage.retention.unlimited": "Unlimited", + "usage.retention.error": "Could not update the usage history limit.", + "usage.retention.disabled": "Unlimited — automatic history compaction is off.", "usage.subtitle": "Local token accounting from your proxy. Missing usage is never shown as zero.", "usage.loading": "Loading usage data…", "usage.empty": "No usage recorded yet. Send a request through the proxy to see activity here.", diff --git a/gui/src/i18n/fr.ts b/gui/src/i18n/fr.ts index 4cf6818072..d6df1f0bf4 100644 --- a/gui/src/i18n/fr.ts +++ b/gui/src/i18n/fr.ts @@ -899,6 +899,16 @@ export const fr: Record = { "debug.noLines.usage": "L’extraction de l’utilisation est activée, mais rien n’a encore été capturé. Envoyez une conversation ou une requête par Codex pour qu’elle apparaisse ici.", "debug.noLines.injection": "Le journal des injections est activé, mais rien n’a encore été capturé. Il consigne l’injection des directives multi-agents et les décisions de plafonnement du niveau lors des tours collab et des sous-agents.", "usage.title": "Utilisation", + "usage.retention.title": "Limite de taille de l’historique d’utilisation", + "usage.retention.help": "Conservez facultativement les enregistrements d’utilisation complets les plus récents dans une limite de taille. Les lignes plus anciennes sont supprimées automatiquement lorsque l’historique la dépasse.", + "usage.retention.enabled": "Limiter la taille de l’historique d’utilisation", + "usage.retention.current": "Taille actuelle", + "usage.retention.limit": "Taille maximale", + "usage.retention.increase": "Augmenter la taille maximale", + "usage.retention.decrease": "Réduire la taille maximale", + "usage.retention.unlimited": "Illimitée", + "usage.retention.error": "Impossible de mettre à jour la limite de l’historique d’utilisation.", + "usage.retention.disabled": "Illimitée — la compression automatique de l’historique est désactivée.", "usage.subtitle": "Comptabilisation locale des jetons par votre proxy. Une utilisation manquante n’est jamais affichée comme nulle.", "usage.loading": "Chargement des données d’utilisation…", "usage.empty": "Aucune utilisation enregistrée pour le moment. Envoyez une requête par le proxy pour voir l’activité ici.", diff --git a/gui/src/i18n/ja.ts b/gui/src/i18n/ja.ts index c77edbe2c7..dfab24f2f9 100644 --- a/gui/src/i18n/ja.ts +++ b/gui/src/i18n/ja.ts @@ -835,6 +835,16 @@ export const ja: Record = { // usage page "usage.title": "使用量", + "usage.retention.title": "使用履歴のサイズ上限", + "usage.retention.help": "最新の完全な使用記録を、指定したサイズ以内に必要に応じて保持します。履歴が上限を超えると古い行が自動的に削除されます。", + "usage.retention.enabled": "使用履歴のサイズを制限", + "usage.retention.current": "現在のサイズ", + "usage.retention.limit": "最大サイズ", + "usage.retention.increase": "最大サイズを増やす", + "usage.retention.decrease": "最大サイズを減らす", + "usage.retention.unlimited": "無制限", + "usage.retention.error": "使用履歴のサイズ上限を更新できませんでした。", + "usage.retention.disabled": "無制限 — 使用履歴の自動圧縮はオフです。", "usage.subtitle": "プロキシからのローカルトークン会計です。欠損した使用量はゼロとして表示されることはありません。", "usage.loading": "使用量データを読み込み中…", "usage.empty": "まだ使用量が記録されていません。プロキシ経由でリクエストを送信するとここにアクティビティが表示されます。", diff --git a/gui/src/i18n/ko.ts b/gui/src/i18n/ko.ts index 34d5ceae87..51ca646077 100644 --- a/gui/src/i18n/ko.ts +++ b/gui/src/i18n/ko.ts @@ -903,6 +903,16 @@ export const ko: Record = { // usage page "usage.title": "사용량", + "usage.retention.title": "사용 기록 크기 제한", + "usage.retention.help": "최신의 완전한 사용 기록을 선택적으로 크기 제한 내에 보관합니다. 원장이 제한을 초과하면 오래된 행이 자동으로 삭제됩니다.", + "usage.retention.enabled": "사용 기록 크기 제한", + "usage.retention.current": "현재 크기", + "usage.retention.limit": "최대 크기", + "usage.retention.increase": "최대 크기 늘리기", + "usage.retention.decrease": "최대 크기 줄이기", + "usage.retention.unlimited": "제한 없음", + "usage.retention.error": "사용 기록 크기 제한을 업데이트할 수 없습니다.", + "usage.retention.disabled": "제한 없음 — 자동 사용 기록 압축이 꺼져 있습니다.", "usage.subtitle": "프록시의 로컬 토큰 집계입니다. 누락된 사용량은 0으로 표시하지 않습니다.", "usage.loading": "사용량 데이터를 불러오는 중…", "usage.empty": "아직 기록된 사용량이 없습니다. 프록시로 요청을 보내면 여기에 표시됩니다.", diff --git a/gui/src/i18n/ru.ts b/gui/src/i18n/ru.ts index 488f87d55b..37af871919 100644 --- a/gui/src/i18n/ru.ts +++ b/gui/src/i18n/ru.ts @@ -890,6 +890,16 @@ export const ru: Record = { // usage page "usage.title": "Использование", + "usage.retention.title": "Ограничение размера истории использования", + "usage.retention.help": "При желании сохраняйте самые новые полные записи использования в пределах заданного размера. Старые строки автоматически удаляются, когда история превышает лимит.", + "usage.retention.enabled": "Ограничить размер истории использования", + "usage.retention.current": "Текущий размер", + "usage.retention.limit": "Максимальный размер", + "usage.retention.increase": "Увеличить максимальный размер", + "usage.retention.decrease": "Уменьшить максимальный размер", + "usage.retention.unlimited": "Без ограничений", + "usage.retention.error": "Не удалось обновить ограничение размера истории использования.", + "usage.retention.disabled": "Без ограничений — автоматическое сжатие истории выключено.", "usage.subtitle": "Локальный учёт токенов вашего прокси. Отсутствующие данные никогда не показываются как ноль.", "usage.loading": "Загрузка данных об использовании…", "usage.empty": "Данных об использовании пока нет. Отправьте запрос через прокси, чтобы увидеть здесь активность.", diff --git a/gui/src/i18n/tr.ts b/gui/src/i18n/tr.ts index 2e46e2792f..eb4727d6b4 100644 --- a/gui/src/i18n/tr.ts +++ b/gui/src/i18n/tr.ts @@ -909,6 +909,16 @@ export const tr: Record = { // usage page "usage.title": "Kullanım", + "usage.retention.title": "Kullanım geçmişi boyut sınırı", + "usage.retention.help": "En yeni eksiksiz kullanım kayıtlarını isteğe bağlı olarak belirlenen boyut sınırı içinde tutar. Geçmiş sınırı aştığında eski satırlar otomatik olarak silinir.", + "usage.retention.enabled": "Kullanım geçmişi boyutunu sınırla", + "usage.retention.current": "Geçerli boyut", + "usage.retention.limit": "Maksimum boyut", + "usage.retention.increase": "Maksimum boyutu artır", + "usage.retention.decrease": "Maksimum boyutu azalt", + "usage.retention.unlimited": "Sınırsız", + "usage.retention.error": "Kullanım geçmişi boyut sınırı güncellenemedi.", + "usage.retention.disabled": "Sınırsız — otomatik geçmiş sıkıştırması kapalı.", "usage.subtitle": "Proxy'nizden yerel jeton muhasebesi.", "usage.loading": "Kullanım verileri yükleniyor…", "usage.empty": "Henüz kullanım kaydedilmedi.", diff --git a/gui/src/i18n/zh-TW.ts b/gui/src/i18n/zh-TW.ts index b9a26da41c..3141c295e1 100644 --- a/gui/src/i18n/zh-TW.ts +++ b/gui/src/i18n/zh-TW.ts @@ -715,6 +715,16 @@ export const zhTW: Record = { "debug.noLines.usage": "用量提取已開啟但尚未捕獲任何內容。請透過 Codex 傳送請求,隨後會顯示在此處。", "debug.noLines.injection": "注入日誌已開啟但尚未捕獲任何內容。它紀錄協作和子代理回合中的多代理指導注入與 effort-cap 決策。", "usage.title": "用量", + "usage.retention.title": "用量歷史大小限制", + "usage.retention.help": "可選擇將最新的完整用量紀錄保留在指定大小內。歷史超過上限後,較舊項目會自動刪除。", + "usage.retention.enabled": "限制用量歷史大小", + "usage.retention.current": "目前大小", + "usage.retention.limit": "最大大小", + "usage.retention.increase": "增大最大大小", + "usage.retention.decrease": "減小最大大小", + "usage.retention.unlimited": "無限制", + "usage.retention.error": "無法更新用量歷史大小限制。", + "usage.retention.disabled": "無限制 — 自動壓縮用量歷史已關閉。", "usage.subtitle": "代理本地的 Token 用量統計。缺失的用量不會顯示為零。", "usage.loading": "正在載入用量資料…", "usage.empty": "尚無用量紀錄。透過代理傳送請求後將在此顯示。", diff --git a/gui/src/i18n/zh.ts b/gui/src/i18n/zh.ts index 46866680b0..b95575e7f9 100644 --- a/gui/src/i18n/zh.ts +++ b/gui/src/i18n/zh.ts @@ -884,6 +884,16 @@ export const zh: Record = { // usage page "usage.title": "用量", + "usage.retention.title": "用量历史大小限制", + "usage.retention.help": "可选择将最新的完整用量记录保留在指定大小以内。历史超过上限后,较旧条目会自动删除。", + "usage.retention.enabled": "限制用量历史大小", + "usage.retention.current": "当前大小", + "usage.retention.limit": "最大大小", + "usage.retention.increase": "增大最大大小", + "usage.retention.decrease": "减小最大大小", + "usage.retention.unlimited": "无限制", + "usage.retention.error": "无法更新用量历史大小限制。", + "usage.retention.disabled": "无限制 — 自动压缩用量历史已关闭。", "usage.subtitle": "代理本地的 Token 用量统计。缺失的用量不会显示为零。", "usage.loading": "正在加载用量数据…", "usage.empty": "尚无用量记录。通过代理发送请求后将在此显示。", diff --git a/gui/src/pages/Usage.tsx b/gui/src/pages/Usage.tsx index 96f0f1db0c..9562181b86 100644 --- a/gui/src/pages/Usage.tsx +++ b/gui/src/pages/Usage.tsx @@ -12,6 +12,7 @@ import { DataSurfaceSkeleton } from "../components/data-surface"; import { SectionTabs } from "../components/section-tabs"; import { sectionAnchorId } from "../section-anchors"; import { parseUsageTimeRange, type UsageRangeError, type UsageTimeWindow } from "../usage-time-range"; +import UsageLedgerRetentionControl from "../components/usage/UsageLedgerRetentionControl"; type Range = "all" | "30d" | "7d"; type UsageSurface = "all" | "codex" | "claude" | "grok"; @@ -1022,6 +1023,7 @@ export default function Usage({ apiBase, connected = false, apiKeyId }: { apiBas /> )} + ); } diff --git a/gui/src/styles-usage-workspace.css b/gui/src/styles-usage-workspace.css index aa9e3bb45c..047279fe4c 100644 --- a/gui/src/styles-usage-workspace.css +++ b/gui/src/styles-usage-workspace.css @@ -214,6 +214,69 @@ gap: 6px; } +/* Retention belongs to Usage, but stays a compact setting row rather than a second dashboard card. */ +.usage-retention-control { + display: flex; + flex-direction: column; + gap: var(--space-2); + margin: 0 0 var(--space-4); + min-width: 0; +} + +.usage-retention-heading { + display: flex; + align-items: center; + justify-content: space-between; + gap: var(--space-3); + min-width: 0; +} + +.usage-retention-heading .h-section { + margin: 0; + font-size: var(--text-body); +} + +.usage-retention-heading p { + margin: var(--space-1) 0 0; + max-width: 68ch; +} + +.usage-retention-editor { + display: flex; + align-items: flex-end; + flex-wrap: wrap; + gap: var(--space-3); +} + +.usage-retention-editor .usage-retention-limit { + display: flex; + flex-direction: column; + align-items: flex-start; + gap: var(--space-1); + margin: 0; +} + +.usage-retention-editor .usage-retention-limit .field-label { + white-space: nowrap; +} + +.usage-retention-editor .codex-auto-switch-input { + width: 104px; +} + +.usage-retention-current { + margin: 0; + white-space: nowrap; +} + +/* Match the Models context-cap cluster: keep the remembered value visible when off, + but visually demote it so Unlimited remains the active state. */ +.usage-retention-limit.is-disabled { + opacity: 0.55; +} + @media (max-width: 640px) { .usage-source-row { align-items: flex-start; flex-direction: column; } + .usage-retention-heading { align-items: flex-start; } + .usage-retention-current { white-space: normal; } } diff --git a/gui/tests/i18n-locales.test.ts b/gui/tests/i18n-locales.test.ts index 178c645024..0eb4d589b5 100644 --- a/gui/tests/i18n-locales.test.ts +++ b/gui/tests/i18n-locales.test.ts @@ -6,8 +6,19 @@ import { detectInitial, } from "../src/i18n/shared"; import { en } from "../src/i18n/en"; +import { de } from "../src/i18n/de"; +import { fr } from "../src/i18n/fr"; +import { ko } from "../src/i18n/ko"; +import { zh } from "../src/i18n/zh"; +import { zhTW } from "../src/i18n/zh-TW"; +import { ru } from "../src/i18n/ru"; +import { ja } from "../src/i18n/ja"; +import { tr } from "../src/i18n/tr"; +import { LAB_CATALOG_OVERRIDES } from "../src/i18n/lab-translations"; import { formatUptime } from "../src/formatUptime"; +const BASE_DICTS = { en, de, fr, ko, zh, "zh-TW": zhTW, ru, ja, tr }; + describe("i18n locale contracts", () => { test("LOCALES and DICTS contain exactly the same locales", () => { const localeCodes = LOCALES.map(locale => locale.code).sort(); @@ -31,15 +42,58 @@ describe("i18n locale contracts", () => { } }); - test("every locale catalog has exactly the English key set", () => { + test("every base locale catalog has exactly the English key set", () => { const expectedKeys = Object.keys(en).sort(); for (const { code } of LOCALES) { - const actualKeys = Object.keys(DICTS[code]).sort(); + const actualKeys = Object.keys(BASE_DICTS[code]).sort(); expect(actualKeys).toEqual(expectedKeys); } }); + test("lab catalog overlay preserves its key set in every locale", () => { + const overlays = [["lab", LAB_CATALOG_OVERRIDES]] as const; + + for (const [name, catalog] of overlays) { + const expectedKeys = Object.keys(catalog.en).sort(); + + for (const { code } of LOCALES) { + expect(Object.keys(catalog[code]).sort(), `${name}.${code}`).toEqual(expectedKeys); + + const prefix = "lab."; + const composedKeys = Object.keys(DICTS[code]) + .filter(key => key.startsWith(prefix) && !key.startsWith("lab.production.")) + .sort(); + expect(composedKeys, `DICTS.${code}.${name}`).toEqual(expectedKeys); + } + } + }); + + test("usage retention strings are ordinary base catalog keys", () => { + const expectedKeys = [ + "usage.retention.title", + "usage.retention.help", + "usage.retention.enabled", + "usage.retention.current", + "usage.retention.limit", + "usage.retention.increase", + "usage.retention.decrease", + "usage.retention.unlimited", + "usage.retention.error", + "usage.retention.disabled", + ].sort(); + + expect(Object.keys(en).filter(key => key.startsWith("storage.usageRetention.")).sort()).toEqual([]); + expect(Object.keys(en).filter(key => key.startsWith("usage.retention.")).sort()).toEqual(expectedKeys); + + for (const { code } of LOCALES) { + expect( + Object.keys(BASE_DICTS[code]).filter(key => key.startsWith("usage.retention.")).sort(), + code, + ).toEqual(expectedKeys); + } + }); + test("every locale preserves interpolation placeholders exactly", () => { const placeholderRe = /\{([a-zA-Z0-9_]+)\}/g; const mismatches: string[] = []; diff --git a/gui/tests/usage-custom-range.test.tsx b/gui/tests/usage-custom-range.test.tsx index ea4e98e51c..a975c67e54 100644 --- a/gui/tests/usage-custom-range.test.tsx +++ b/gui/tests/usage-custom-range.test.tsx @@ -35,9 +35,24 @@ beforeEach(() => { // The page also has a held memory cache: each test gets a distinct report identity. apiBase = `http://usage-custom-${++sequence}`; requests = []; - globalThis.fetch = ((input: RequestInfo | URL) => new Promise(resolve => { - requests.push({ url: String(input), resolve }); - })) as typeof fetch; + globalThis.fetch = ((input: RequestInfo | URL) => { + const url = String(input); + // Usage now mounts its compact retention control alongside the report. Keep that + // independent status read out of the report request gates so the range assertions + // continue to describe only `/api/usage` generation ordering. + if (url.includes("/api/storage/usage-ledger-retention")) { + return Promise.resolve(Response.json({ + enabled: false, + maxBytes: 128 * 1024 * 1024, + currentBytes: 0, + overLimit: false, + job: { status: "idle" }, + })); + } + return new Promise(resolve => { + requests.push({ url, resolve }); + }); + }) as typeof fetch; }); afterEach(async () => { diff --git a/gui/tests/usage-retention-control.test.ts b/gui/tests/usage-retention-control.test.ts new file mode 100644 index 0000000000..cf95a54bde --- /dev/null +++ b/gui/tests/usage-retention-control.test.ts @@ -0,0 +1,293 @@ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { Window } from "happy-dom"; +import { act, createElement } from "react"; +import { createRoot, type Root } from "react-dom/client"; +import UsageLedgerRetentionControl from "../src/components/usage/UsageLedgerRetentionControl"; +import { LanguageProvider } from "../src/i18n"; +import { useI18n } from "../src/i18n/shared"; + +const globals = ["document", "window", "navigator", "localStorage", "fetch", "IS_REACT_ACT_ENVIRONMENT"] as const; +type GlobalName = (typeof globals)[number]; + +let previous: Record; +let testWindow: Window; +let root: Root | null = null; +let host: HTMLElement; + +function restoreProperty(target: object, key: PropertyKey, descriptor: PropertyDescriptor | undefined): void { + if (descriptor) Object.defineProperty(target, key, descriptor); + else Reflect.deleteProperty(target, key); +} + +beforeEach(() => { + previous = Object.fromEntries( + globals.map(key => [key, Object.getOwnPropertyDescriptor(globalThis, key)]), + ) as typeof previous; + testWindow = new Window({ url: "http://localhost/" }); + Object.defineProperties(globalThis, { + document: { configurable: true, value: testWindow.document }, + window: { configurable: true, value: testWindow }, + navigator: { configurable: true, value: testWindow.navigator }, + localStorage: { configurable: true, value: testWindow.localStorage }, + IS_REACT_ACT_ENVIRONMENT: { configurable: true, value: true }, + }); + host = testWindow.document.createElement("div") as never as HTMLElement; + testWindow.document.body.appendChild(host as never); +}); + +afterEach(async () => { + if (root) await act(async () => root?.unmount()); + root = null; + for (const key of globals) restoreProperty(globalThis, key, previous[key]); + await testWindow.happyDOM?.close?.(); +}); + +async function settleTimers(): Promise { + await act(async () => { + await new Promise(resolve => testWindow.setTimeout(resolve, 0)); + await Promise.resolve(); + }); +} + +async function mount(apiBase: string): Promise { + await act(async () => { + root = createRoot(host); + root.render(createElement( + LanguageProvider, + null, + createElement(UsageLedgerRetentionControl, { apiBase }), + )); + }); + await settleTimers(); +} + +function LocaleHarness({ apiBase }: { apiBase: string }) { + const { setLocale } = useI18n(); + return createElement( + "div", + null, + createElement("button", { type: "button", id: "locale-switch", onClick: () => setLocale("de") }, "locale"), + createElement(UsageLedgerRetentionControl, { apiBase }), + ); +} + +test("retention control stays on Usage and out of Storage", async () => { + const page = await Bun.file(new URL("../src/pages/Usage.tsx", import.meta.url)).text(); + const storageWorkspace = await Bun.file(new URL("../src/components/storage-workspace/StorageWorkspace.tsx", import.meta.url)).text(); + + expect(page).toContain("UsageLedgerRetentionControl"); + expect(storageWorkspace).not.toContain("UsageLedgerRetentionPanel"); +}); + +test("renders one switch and toggles without rewriting the saved byte ceiling", async () => { + const apiBase = "http://usage-retention-test"; + const maxBytes = 512 * 1024 * 1024 + 17; + const writes: Array<{ enabled: boolean; maxBytes: number }> = []; + let enabled = false; + + globalThis.fetch = (async (input: RequestInfo | URL, init?: RequestInit) => { + const url = String(input); + if (url !== `${apiBase}/api/storage/usage-ledger-retention`) throw new Error(`unexpected fetch: ${url}`); + if ((init?.method ?? "GET") === "PUT") { + const body = JSON.parse(String(init?.body)) as { enabled: boolean; maxBytes: number }; + writes.push(body); + enabled = body.enabled; + return Response.json({ enabled, maxBytes, currentBytes: 1234 }); + } + return Response.json({ enabled, maxBytes, currentBytes: 1234 }); + }) as typeof fetch; + + await mount(apiBase); + + const switches = host.querySelectorAll("button.switch"); + expect(switches.length).toBe(1); + expect(host.querySelector('[aria-haspopup="listbox"]')).toBeNull(); + expect(switches[0].disabled).toBe(false); + expect(switches[0].getAttribute("aria-pressed")).toBe("false"); + expect(host.querySelector(".usage-retention-state")?.textContent).toBe("Unlimited"); + expect(host.querySelector(".usage-retention-limit")?.classList.contains("is-disabled")).toBe(true); + + await act(async () => { + switches[0].click(); + await Promise.resolve(); + }); + expect(writes[0]).toEqual({ enabled: true, maxBytes }); + expect(switches[0].getAttribute("aria-pressed")).toBe("true"); + expect(host.querySelector(".usage-retention-state")).toBeNull(); + expect(host.querySelector(".usage-retention-limit")?.classList.contains("is-disabled")).toBe(false); + + await act(async () => { + switches[0].click(); + await Promise.resolve(); + }); + expect(writes[1]).toEqual({ enabled: false, maxBytes }); + expect(switches[0].getAttribute("aria-pressed")).toBe("false"); + expect(host.querySelector(".usage-retention-state")?.textContent).toBe("Unlimited"); + expect(host.querySelector(".usage-retention-limit")?.classList.contains("is-disabled")).toBe(true); +}); + +test("a stale GET cannot repaint policy after a successful toggle", async () => { + const apiBase = "http://usage-retention-stale"; + const maxBytes = 1024 * 1024 * 1024; + let getCount = 0; + let resolveStaleGet: ((response: Response) => void) | undefined; + + globalThis.fetch = ((input: RequestInfo | URL, init?: RequestInit) => { + const url = String(input); + if (url !== `${apiBase}/api/storage/usage-ledger-retention`) throw new Error(`unexpected fetch: ${url}`); + if ((init?.method ?? "GET") === "PUT") { + return Promise.resolve(Response.json({ enabled: true, maxBytes, currentBytes: 1234 })); + } + getCount += 1; + if (getCount === 1) return Promise.resolve(Response.json({ enabled: false, maxBytes, currentBytes: 1234 })); + return new Promise(resolve => { resolveStaleGet = resolve; }); + }) as typeof fetch; + + await act(async () => { + root = createRoot(host); + root.render(createElement(LanguageProvider, null, createElement(LocaleHarness, { apiBase }))); + }); + await settleTimers(); + + const localeSwitch = host.querySelector("#locale-switch"); + if (!localeSwitch) throw new Error("locale switch missing"); + await act(async () => { localeSwitch.click(); }); + await settleTimers(); + expect(getCount).toBe(2); + + const toggle = host.querySelector("button.switch"); + if (!toggle) throw new Error("retention switch missing"); + await act(async () => { + toggle.click(); + await Promise.resolve(); + }); + expect(toggle.getAttribute("aria-pressed")).toBe("true"); + + if (!resolveStaleGet) throw new Error("stale GET was not started"); + await act(async () => { + resolveStaleGet(Response.json({ enabled: false, maxBytes, currentBytes: 1234 })); + await Promise.resolve(); + }); + expect(toggle.getAttribute("aria-pressed")).toBe("true"); +}); + +test("a stale failed GET is silent after a successful toggle", async () => { + const apiBase = "http://usage-retention-stale-failure"; + const maxBytes = 1024 * 1024 * 1024; + let getCount = 0; + let resolveStaleGet: ((response: Response) => void) | undefined; + + globalThis.fetch = ((input: RequestInfo | URL, init?: RequestInit) => { + const url = String(input); + if (url !== `${apiBase}/api/storage/usage-ledger-retention`) throw new Error(`unexpected fetch: ${url}`); + if ((init?.method ?? "GET") === "PUT") { + return Promise.resolve(Response.json({ enabled: true, maxBytes, currentBytes: 1234 })); + } + getCount += 1; + if (getCount === 1) return Promise.resolve(Response.json({ enabled: false, maxBytes, currentBytes: 1234 })); + return new Promise(resolve => { resolveStaleGet = resolve; }); + }) as typeof fetch; + + await act(async () => { + root = createRoot(host); + root.render(createElement(LanguageProvider, null, createElement(LocaleHarness, { apiBase }))); + }); + await settleTimers(); + + const localeSwitch = host.querySelector("#locale-switch"); + if (!localeSwitch) throw new Error("locale switch missing"); + await act(async () => { localeSwitch.click(); }); + await settleTimers(); + expect(getCount).toBe(2); + + const toggle = host.querySelector("button.switch"); + if (!toggle) throw new Error("retention switch missing"); + await act(async () => { + toggle.click(); + await Promise.resolve(); + }); + expect(toggle.getAttribute("aria-pressed")).toBe("true"); + + if (!resolveStaleGet) throw new Error("stale GET was not started"); + await act(async () => { + resolveStaleGet(new Response("", { status: 500 })); + await Promise.resolve(); + }); + expect(toggle.getAttribute("aria-pressed")).toBe("true"); + expect(host.querySelector('[role="alert"]')).toBeNull(); +}); + +test("shows a custom MiB editor only when enabled and saves the edited ceiling", async () => { + const apiBase = "http://usage-retention-custom"; + const initialMaxBytes = 768 * 1024 * 1024; + const writes: Array<{ enabled: boolean; maxBytes: number }> = []; + let maxBytes = initialMaxBytes; + + globalThis.fetch = (async (input: RequestInfo | URL, init?: RequestInit) => { + const url = String(input); + if (url !== `${apiBase}/api/storage/usage-ledger-retention`) throw new Error(`unexpected fetch: ${url}`); + if ((init?.method ?? "GET") === "PUT") { + const body = JSON.parse(String(init?.body)) as { enabled: boolean; maxBytes: number }; + writes.push(body); + maxBytes = body.maxBytes; + return Response.json({ enabled: body.enabled, maxBytes, currentBytes: 1234 }); + } + return Response.json({ enabled: true, maxBytes, currentBytes: 1234 }); + }) as typeof fetch; + + await mount(apiBase); + + const input = host.querySelector('input[type="number"]'); + if (!input) throw new Error("custom retention input missing"); + expect(input.value).toBe("768"); + expect(input.min).toBe("1"); + expect(host.querySelector('[aria-haspopup="listbox"]')).toBeNull(); + + const increment = input.parentElement?.querySelector(".ocx-stepper__btn"); + if (!increment) throw new Error("retention stepper missing"); + await act(async () => { increment.click(); }); + expect(testWindow.document.activeElement).toBe(input); + expect(input.value).toBe("769"); + expect(writes).toEqual([]); + + const outside = testWindow.document.createElement("button") as never as HTMLButtonElement; + outside.type = "button"; + host.appendChild(outside as never); + await act(async () => { + outside.focus(); + await Promise.resolve(); + }); + expect(writes).toEqual([{ enabled: true, maxBytes: 769 * 1024 * 1024 }]); + + const toggle = host.querySelector("button.switch"); + if (!toggle) throw new Error("retention switch missing"); + await act(async () => { + toggle.click(); + await Promise.resolve(); + }); + expect(host.querySelector('input[type="number"]')).toBeNull(); +}); + +test("failed toggle keeps the last server state and surfaces an error", async () => { + const apiBase = "http://usage-retention-failure"; + const maxBytes = 256 * 1024 * 1024; + + globalThis.fetch = (async (input: RequestInfo | URL, init?: RequestInit) => { + const url = String(input); + if (url !== `${apiBase}/api/storage/usage-ledger-retention`) throw new Error(`unexpected fetch: ${url}`); + if ((init?.method ?? "GET") === "PUT") return new Response("", { status: 500 }); + return Response.json({ enabled: false, maxBytes, currentBytes: 0 }); + }) as typeof fetch; + + await mount(apiBase); + const toggle = host.querySelector("button.switch"); + if (!toggle) throw new Error("retention switch missing"); + + await act(async () => { + toggle.click(); + await Promise.resolve(); + }); + + expect(toggle.getAttribute("aria-pressed")).toBe("false"); + expect(host.querySelector('[role="alert"]')?.textContent?.length).toBeGreaterThan(0); +}); diff --git a/scripts/test-layout/layout.json b/scripts/test-layout/layout.json index fad27db650..4392e773e3 100644 --- a/scripts/test-layout/layout.json +++ b/scripts/test-layout/layout.json @@ -1164,6 +1164,7 @@ "settings-oauth-open-browser.test.ts": "config", "settings-startup-health-seam.test.ts": "config", "settings-stream-mode.test.ts": "config", + "settings-usage-ledger-retention.test.ts": "config", "shutdown-drain.test.ts": "service", "shutdown-launcher.test.ts": "service", "sidebar-routes.test.ts": "server", diff --git a/skills/ocx/references/01_management_surface.md b/skills/ocx/references/01_management_surface.md index 7fcf629e50..fed84aaabc 100644 --- a/skills/ocx/references/01_management_surface.md +++ b/skills/ocx/references/01_management_surface.md @@ -624,6 +624,27 @@ JSON mode: `payload`. - `policy set` never enables implicitly: omitting `--enabled` keeps the stored value. - `policy run` forces a run regardless of schedule, so it needs `--yes`. +### `ocx storage usage-limit` + +Show or change the usage-history size limit. + +| Method | Route | +|---|---| +| GET | `/api/storage/usage-ledger-retention` | +| PUT | `/api/storage/usage-ledger-retention` | + +| Flag | Value | Meaning | +|---|---|---| +| `--enabled` | string | true or false. | +| `--mib` | number | Maximum usage-ledger size in MiB; minimum 1. | +| `--json` | boolean | Emit the policy or status as JSON. | + +JSON mode: `payload`. + +- The limit is opt-in; a bare invocation only reads status. +- Changing the MiB value without `--enabled` preserves the saved enabled state. +- Oversized ledgers are compacted by the automatic scheduler after the limit is enabled. + ### `ocx system codex-restart` Restart the Codex app-server. @@ -728,6 +749,6 @@ JSON mode: `payload`. ## Counts -- declared capabilities: 39 -- of those, state-changing: 18 +- declared capabilities: 40 +- of those, state-changing: 19 - head-resolved invocations: 2 diff --git a/skills/ocx/references/02_json_shapes.md b/skills/ocx/references/02_json_shapes.md index e93be59748..784fc88de4 100644 --- a/skills/ocx/references/02_json_shapes.md +++ b/skills/ocx/references/02_json_shapes.md @@ -115,6 +115,23 @@ The value returned is the **applied** one after server normalization, not what y `digest` binds a run to this preview; the mutating call must carry it and the server rejects a stale one with 409. The CLI handles that for you — it always previews first. +## `ocx storage usage-limit --json` + +The status response is the management payload: + +```json +{"enabled":false,"maxBytes":1073741824,"currentBytes":67108864,"overLimit":false,"job":{"status":"idle"}} +``` + +The default policy is Unlimited (`enabled:false`). On an unconfigured installation, `maxBytes` +remembers 1 GiB (`1073741824`) as the ceiling used on first enable; later API/CLI changes preserve +the saved ceiling while retention is disabled. + +`set` returns the same fields with `ok: true`; it merges only the fields supplied by +`--enabled` and `--mib`. Oversized ledgers are compacted by the automatic scheduler after the +limit is enabled; there is no manual run response. The `job` field reports the scheduler's +process-local status and latest outcome. + ## Error shape A management error prints up to three lines and returns a non-zero code: diff --git a/skills/ocx/references/03_recipes.md b/skills/ocx/references/03_recipes.md index 14c16cdf53..9cf66fd89f 100644 --- a/skills/ocx/references/03_recipes.md +++ b/skills/ocx/references/03_recipes.md @@ -226,6 +226,16 @@ ocx storage trash restore --yes --json The preview runs in both paths because the mutating route requires the `digest` the preview returns and rejects a stale one with 409. So the two invocations agree about what is being authorized. +For the append-only usage ledger, inspect the saved ceiling before changing it: + +```bash +ocx storage usage-limit show --json +ocx storage usage-limit set --enabled true --mib 512 --json +``` + +Changing the ceiling is non-destructive. The scheduler compacts complete `usage.jsonl` rows in the +background once `maxBytes` is exceeded; there is no manual compaction command. + ## 9. Read Muse Code usage, and know why it can be old `meta-muse` reports usage differently from every other provider, and the difference changes what diff --git a/src/cli/capabilities.ts b/src/cli/capabilities.ts index d9aa8d0402..7ead74725e 100644 --- a/src/cli/capabilities.ts +++ b/src/cli/capabilities.ts @@ -439,6 +439,26 @@ export const CAPABILITIES: readonly Capability[] = [ "`policy run` forces a run regardless of schedule, so it needs `--yes`.", ], }, + { + command: ["storage", "usage-limit"], + summary: "Show or change the usage-history size limit.", + routes: [ + { method: "GET", path: "/api/storage/usage-ledger-retention" }, + { method: "PUT", path: "/api/storage/usage-ledger-retention" }, + ], + flags: [ + { name: "--enabled", value: "string", summary: "true or false." }, + { name: "--mib", value: "number", summary: "Maximum usage-ledger size in MiB; minimum 1." }, + { name: "--json", value: "boolean", summary: "Emit the policy or status as JSON." }, + ], + mutates: true, + json: "payload", + details: [ + "The limit is opt-in; a bare invocation only reads status.", + "Changing the MiB value without `--enabled` preserves the saved enabled state.", + "Oversized ledgers are compacted by the automatic scheduler after the limit is enabled.", + ], + }, { command: ["inspect", "config"], summary: "The effective merged configuration the proxy is running.", diff --git a/src/cli/help.ts b/src/cli/help.ts index 764029b63e..a6b899aa62 100644 --- a/src/cli/help.ts +++ b/src/cli/help.ts @@ -77,7 +77,7 @@ Usage: ocx logs [filters] Alias of ocx observe logs ocx usage [--range ] [--provider ] [--model ] Token and estimated-cost report (alias of ocx observe usage) - ocx storage Storage report, cleanup, trash, and the cleanup policy + ocx storage Storage report, cleanup, trash, cleanup policy, and usage retention ocx memory [--json] Alias of ocx observe memory ocx api-key Alias of ocx access key ocx access External API keys and endpoint information diff --git a/src/cli/registry.ts b/src/cli/registry.ts index f219043254..ade079bbfd 100644 --- a/src/cli/registry.ts +++ b/src/cli/registry.ts @@ -307,11 +307,12 @@ export const CLI_COMMANDS: CliCommandEntry[] = [ }, { name: "storage", - usage: "ocx storage ...", - summary: "Storage report, archived-session cleanup, trash restore, and the cleanup policy.", + usage: "ocx storage ...", + summary: "Storage report, archived-session cleanup, trash restore, cleanup policy, and usage-ledger retention.", details: [ "A bare `ocx storage` prints the report, as it did when this was an alias of `observe storage`.", "`cleanup` previews by default and only deletes under --yes; `trash restore` and `policy run` also require --yes.", + "`usage-limit` shows or changes the opt-in usage-history ceiling; oversized ledgers are compacted by the automatic scheduler.", ], }, { name: "memory", usage: "ocx memory [--json]", summary: "Alias of ocx observe memory." }, diff --git a/src/cli/storage.ts b/src/cli/storage.ts index ed13aa6710..2b80e76191 100644 --- a/src/cli/storage.ts +++ b/src/cli/storage.ts @@ -1,8 +1,8 @@ /** - * `ocx storage` — the archived-session cleanup, trash, and cleanup-policy surface (wp7). + * `ocx storage` — archived-session cleanup, trash, cleanup-policy, and usage-ledger controls. * * Every route here existed with no CLI caller, so reclaiming disk space was dashboard-only. - * Three of them delete or move operator data, and the rules for those are deliberate: + * Destructive operations keep the original delegation boundary: * * 1. **Default to preview.** `ocx storage cleanup --percent N` runs the preview route and prints * what WOULD be freed, then exits 0 having mutated nothing. @@ -11,9 +11,8 @@ * 3. **`--json` on the preview emits the candidate list**, so an agent can decide from data * rather than from a sentence. * - * This is the opposite of the GitHub star POST, which no flag can authorize: cleanup spends the - * operator's DATA, which they can delegate, while starring spends their IDENTITY, which they - * cannot delegate to an agent. + * Usage-limit policy writes are non-destructive; oversized ledgers are compacted by the + * automatic scheduler after the limit is enabled. */ import { CliUsageError, @@ -28,6 +27,8 @@ import { type RuntimeApiDeps, } from "./runtime-api"; +const MIB = 1024 * 1024; + const USAGE = `Usage: ocx storage report [--json] ocx storage cleanup --percent <0-100> [--mode ] [--yes] [--json] @@ -37,8 +38,10 @@ const USAGE = `Usage: ocx storage policy set [--enabled ] [--percent <0-100>] [--mode ] [--schedule ] [--json] ocx storage policy run [--yes] [--json] + ocx storage usage-limit [show] [--json] + ocx storage usage-limit set [--enabled ] [--mib ] [--json] -Cleanup and restore MUTATE operator data and require --yes. +Cleanup, restore, and policy run MUTATE operator data and require --yes where noted. Without --yes, cleanup prints the preview and changes nothing.`; /** The digest binds a run to the preview it was authorized against. */ @@ -50,11 +53,13 @@ interface CleanupPreview { candidates?: { relPath?: string; bytes?: number }[]; } +/** Format a byte count for CLI summaries without changing the API representation. */ function mib(bytes: number | undefined): string { if (typeof bytes !== "number" || !Number.isFinite(bytes)) return "unknown size"; - return `${(bytes / 1024 / 1024).toFixed(1)} MiB`; + return `${(bytes / MIB).toFixed(1)} MiB`; } +/** Render the non-mutating archive-cleanup preview used before any confirmed deletion. */ function previewLines(preview: CleanupPreview): string[] { const lines = [ `Would remove ${preview.count ?? 0} archived session file(s), freeing ${mib(preview.bytes)}.`, @@ -68,6 +73,7 @@ function previewLines(preview: CleanupPreview): string[] { return lines; } +/** Preview or explicitly execute archived-session cleanup. */ async function cleanup(argv: string[], deps: RuntimeApiDeps): Promise { const args = [...argv]; const wantsJson = takeFlag(args, "--json"); @@ -110,6 +116,7 @@ async function cleanup(argv: string[], deps: RuntimeApiDeps): Promise { printData(result, wantsJson, summaryLines(result)); } +/** List quarantine entries or explicitly restore one. */ async function trash(argv: string[], deps: RuntimeApiDeps): Promise { const action = argv[0] && !argv[0].startsWith("-") ? argv[0] : "list"; const rest = argv[0] && !argv[0].startsWith("-") ? argv.slice(1) : argv; @@ -146,6 +153,7 @@ async function trash(argv: string[], deps: RuntimeApiDeps): Promise { printData(result, wantsJson, summaryLines(result)); } +/** Show, edit, or explicitly run archived-session cleanup policy. */ async function policy(argv: string[], deps: RuntimeApiDeps): Promise { const action = argv[0] && !argv[0].startsWith("-") ? argv[0] : "show"; const rest = argv[0] && !argv[0].startsWith("-") ? argv.slice(1) : argv; @@ -215,6 +223,53 @@ async function policy(argv: string[], deps: RuntimeApiDeps): Promise { printData(result, wantsJson, summaryLines(result)); } +/** Show or edit the usage-history ceiling; enforcement is performed by the scheduler. */ +async function usageLimit(argv: string[], deps: RuntimeApiDeps): Promise { + const action = argv[0] && !argv[0].startsWith("-") ? argv[0] : "show"; + const rest = argv[0] && !argv[0].startsWith("-") ? argv.slice(1) : argv; + + if (action === "show") { + const args = [...rest]; + const wantsJson = takeFlag(args, "--json"); + rejectArgs(args, USAGE); + const result = await runtimeRequest("/api/storage/usage-ledger-retention", {}, deps); + printData(result, wantsJson, summaryLines(result)); + return; + } + + if (action === "set") { + const args = [...rest]; + const wantsJson = takeFlag(args, "--json"); + const enabled = takeOption(args, "--enabled"); + const maxMiB = takeIntegerOption(args, "--mib", { min: 1 }); + rejectArgs(args, USAGE); + + if (enabled !== undefined && enabled !== "true" && enabled !== "false") { + throw new CliUsageError("--enabled must be true or false", USAGE); + } + if (maxMiB !== undefined && !Number.isSafeInteger(maxMiB * MIB)) { + throw new CliUsageError("--mib is too large", USAGE); + } + const body: Record = {}; + if (enabled !== undefined) body.enabled = enabled === "true"; + if (maxMiB !== undefined) body.maxBytes = maxMiB * MIB; + if (Object.keys(body).length === 0) { + throw new CliUsageError("usage-limit set needs at least one of --enabled or --mib", USAGE); + } + + const result = await runtimeRequest("/api/storage/usage-ledger-retention", { + method: "PUT", + headers: { "content-type": "application/json" }, + body: JSON.stringify(body), + }, deps); + printData(result, wantsJson, summaryLines(result)); + return; + } + + throw new CliUsageError(`unknown usage-limit action ${action}`, USAGE); +} + +/** Dispatch `ocx storage` while preserving explicit confirmation boundaries for mutations. */ export async function handleStorageCommand(argv: string[], deps: RuntimeApiDeps = {}): Promise { const hasSub = argv[0] !== undefined && !argv[0].startsWith("-"); const sub = hasSub ? argv[0]! : "report"; @@ -236,6 +291,7 @@ export async function handleStorageCommand(argv: string[], deps: RuntimeApiDeps else if (sub === "cleanup") await cleanup(rest, deps); else if (sub === "trash") await trash(rest, deps); else if (sub === "policy") await policy(rest, deps); + else if (sub === "usage-limit") await usageLimit(rest, deps); else throw new CliUsageError(`unknown storage command ${sub}`, USAGE); }); } diff --git a/src/config.ts b/src/config.ts index 66162c6a6a..41850a22e6 100644 --- a/src/config.ts +++ b/src/config.ts @@ -1298,6 +1298,12 @@ const configSchema = z.object({ enabled: z.boolean().optional(), leadTimeMinutes: z.number().int().min(1).max(60).optional(), }).optional().catch(undefined), + // Opt-in usage.jsonl byte ceiling. Reject unknown nested keys and degrade the + // whole optional section so a misspelled policy can never enable retention. + usageLedgerRetention: z.object({ + enabled: z.boolean().optional(), + maxBytes: z.number().int().min(1024 * 1024).optional(), + }).strict().optional().catch(undefined), // Model ids excluded from the Grok Build managed block (dashboard switches). grokExcludedModels: z.array(z.string()).optional(), // Invalid values degrade to undefined ("auto") instead of failing the whole diff --git a/src/lib/windows-atomic-replace.ts b/src/lib/windows-atomic-replace.ts index 0f3ba94552..bca35f8a14 100644 --- a/src/lib/windows-atomic-replace.ts +++ b/src/lib/windows-atomic-replace.ts @@ -34,6 +34,7 @@ export type ReplacePublisher = | "lab-automation" | "lab-ledger" | "storage-cleanup" + | "usage-retention" | "tray"; /** The Windows error codes this module treats as a momentary hold. */ diff --git a/src/routing/history/discard-index.ts b/src/routing/history/discard-index.ts new file mode 100644 index 0000000000..18fe594777 --- /dev/null +++ b/src/routing/history/discard-index.ts @@ -0,0 +1,49 @@ +import { unlinkSync } from "node:fs"; +import { getConfigDir } from "../../config"; +import { closeRequestHistoryIndex } from "./indexer"; +import { historyIndexPath } from "./schema"; + +const DELETE_RETRY_DELAYS_MS = [25, 50] as const; + +/** Return true only for Windows-style transient sharing violations worth retrying briefly. */ +function isTransientDeleteError(error: unknown): boolean { + if (process.platform !== "win32") return false; + const code = (error as NodeJS.ErrnoException | undefined)?.code; + return code === "EBUSY" || code === "EPERM" || code === "EACCES"; +} + +/** Remove one derived-index file, treating absence as success and retrying short Windows holds. */ +function unlinkDerivedFile(path: string): boolean { + for (let attempt = 0; ; attempt += 1) { + try { + unlinkSync(path); + return true; + } catch (error) { + if ((error as NodeJS.ErrnoException | undefined)?.code === "ENOENT") return true; + if (!isTransientDeleteError(error) || attempt >= DELETE_RETRY_DELAYS_MS.length) return false; + Bun.sleepSync(DELETE_RETRY_DELAYS_MS[attempt]!); + } + } +} + +/** + * Close and best-effort delete the disposable request-history projection and WAL sidecars. + * + * Retention replaces the canonical `usage.jsonl` with a new filesystem identity. The indexer + * would detect that identity change on its next query and rebuild automatically, but deleting + * the old projection here reclaims its disk immediately even when no later history query occurs. + * Failure is non-fatal: the next index open still validates source identity and recreates it. + * + * Sidecars are removed before the main database. If either sidecar remains locked, leave the + * main file in place too; the indexer can later discard the complete stale set rather than + * opening a fresh main database beside an old same-name WAL/SHM file. + * + * `configDir` is injectable so isolated retention tests never touch the process' real config home. + */ +export function discardRequestHistoryProjection(configDir = getConfigDir()): boolean { + closeRequestHistoryIndex(); + const path = historyIndexPath(configDir); + if (!unlinkDerivedFile(`${path}-wal`)) return false; + if (!unlinkDerivedFile(`${path}-shm`)) return false; + return unlinkDerivedFile(path); +} diff --git a/src/server/background-lifecycle.ts b/src/server/background-lifecycle.ts index 17a7a7fe57..13c3fa7259 100644 --- a/src/server/background-lifecycle.ts +++ b/src/server/background-lifecycle.ts @@ -10,6 +10,12 @@ import { startStorageCleanupScheduler, stopStorageCleanupScheduler, } from "../storage/policy-scheduler"; +import { abortUsageLedgerRetentionJobAsync } from "../usage/ledger-retention-job"; +import { + scheduleUsageLedgerRetentionStartupRun, + startUsageLedgerRetentionScheduler, + stopUsageLedgerRetentionScheduler, +} from "../usage/ledger-retention-scheduler"; import { startQuotaResetPoller, stopQuotaResetPoller } from "../quota/reset-poller"; import { cancelQueuedStorageWorkerSpawns, @@ -47,11 +53,13 @@ const owners: LeaseOwner[] = []; let processLoops: ProcessLoops | null = null; let cleanupInProgress = false; +/** Route cleanup-policy state updates to the newest live server owner, or detach the sink. */ function setLivePolicyOwner(applyPolicy: PolicyApply | null): void { setStorageCleanupPolicyLiveSink(applyPolicy); setStorageCleanupPolicyJobLiveApply(applyPolicy); } +/** Start the process-wide watchdogs, sweepers, schedulers, and optional quota background hooks. */ function startProcessLoops(applyPolicy: PolicyApply): ProcessLoops { let memoryWatchdog: MemoryWatchdog | null = null; let stateStoreSweeper: ReturnType | null = null; @@ -60,6 +68,7 @@ function startProcessLoops(applyPolicy: PolicyApply): ProcessLoops { stateStoreSweeper = startStateStoreSweeper(); setLivePolicyOwner(applyPolicy); startStorageCleanupScheduler(); + startUsageLedgerRetentionScheduler(); // Opt-in: the tick itself is a no-op unless config.quotaResetNotify is enabled with a // sink, and the interval is unref'd, so a default install pays one dormant timer. startQuotaResetPoller(); @@ -84,31 +93,37 @@ function startProcessLoops(applyPolicy: PolicyApply): ProcessLoops { memoryWatchdog?.stop(); stateStoreSweeper?.stop(); stopStorageCleanupScheduler(); + stopUsageLedgerRetentionScheduler(); stopQuotaResetPoller(); setLivePolicyOwner(null); throw error; } } +/** Stop process-wide timer loops and detach the current live-policy sink. */ function stopProcessLoops(): void { const loops = processLoops; processLoops = null; loops?.memoryWatchdog.stop(); loops?.stateStoreSweeper.stop(); stopStorageCleanupScheduler(); + stopUsageLedgerRetentionScheduler(); stopQuotaResetPoller(); setLivePolicyOwner(null); } +/** Cancel both storage Worker controllers, then join every shared storage Worker before exit. */ async function stopStoragePolicyWorker(): Promise { cancelQueuedStorageWorkerSpawns(); - const abortResult = await Promise.allSettled([abortStorageCleanupPolicyJobAsync()]); - if (abortResult[0]?.status === "rejected") { + const abortResult = await Promise.allSettled([ + abortStorageCleanupPolicyJobAsync(), + abortUsageLedgerRetentionJobAsync(), + ]); + for (const result of abortResult) { + if (result.status !== "rejected") continue; console.warn( - "[storage] policy worker abort during server stop failed:", - abortResult[0].reason instanceof Error - ? abortResult[0].reason.message - : abortResult[0].reason, + "[storage] worker abort during server stop failed:", + result.reason instanceof Error ? result.reason.message : result.reason, ); } try { @@ -121,6 +136,7 @@ async function stopStoragePolicyWorker(): Promise { } } +/** Remove one lifecycle owner by token and report whether it was still active. */ function removeOwner(owner: LeaseOwner): boolean { const index = owners.findIndex(candidate => candidate.token === owner.token); if (index === -1) return false; @@ -128,6 +144,7 @@ function removeOwner(owner: LeaseOwner): boolean { return true; } +/** Release one owner synchronously and classify whether shared process resources remain. */ function releaseOwnerSynchronously(owner: LeaseOwner): "inactive" | "shared" | "last" { if (!removeOwner(owner)) return "inactive"; owner.resources.release(); @@ -177,6 +194,7 @@ export function acquireServerBackgroundLifecycle( scheduleStartupRun() { if (owners.some(candidate => candidate.token === owner.token)) { scheduleStorageCleanupStartupRun(); + scheduleUsageLedgerRetentionStartupRun(); } }, release() { diff --git a/src/server/management/route-registry.ts b/src/server/management/route-registry.ts index d8f81de5ec..1864750eab 100644 --- a/src/server/management/route-registry.ts +++ b/src/server/management/route-registry.ts @@ -313,10 +313,12 @@ export const MANAGEMENT_ROUTES: readonly ManagementRoute[] = [ { method: "POST", path: "/api/github/star", module: "server/management/sidebar-routes", mutates: true, exempt: { reason: "session-only", why: "User-consent boundary in AGENTS_INSTALL.md: starring spends the user's identity. Must never gain a CLI verb." } }, // server/management/storage-log-guard-routes { method: "GET", path: "/api/storage/codex-logs", module: "server/management/storage-log-guard-routes", mutates: false }, + { method: "GET", path: "/api/storage/usage-ledger-retention", module: "server/management/storage-log-guard-routes", mutates: false }, { method: "POST", path: "/api/storage/codex-logs/compact", module: "server/management/storage-log-guard-routes", mutates: true }, { method: "POST", path: "/api/storage/codex-logs/protect", module: "server/management/storage-log-guard-routes", mutates: true }, { method: "POST", path: "/api/storage/codex-logs/repair", module: "server/management/storage-log-guard-routes", mutates: true }, { method: "POST", path: "/api/storage/codex-logs/unprotect", module: "server/management/storage-log-guard-routes", mutates: true }, + { method: "PUT", path: "/api/storage/usage-ledger-retention", module: "server/management/storage-log-guard-routes", mutates: true }, // server/management/system-routes { method: "GET", path: "/api/system/health", module: "server/management/system-routes", mutates: false }, { method: "GET", path: "/api/system/memory", module: "server/management/system-routes", mutates: false }, diff --git a/src/server/management/storage-log-guard-routes.ts b/src/server/management/storage-log-guard-routes.ts index 324e746bd7..cfa6de7c0a 100644 --- a/src/server/management/storage-log-guard-routes.ts +++ b/src/server/management/storage-log-guard-routes.ts @@ -12,6 +12,16 @@ import { type CodexLogGuardStatus, } from "../../codex/log-guard/protection"; import { scanStorage } from "../../storage/scanner"; +import { + applyUsageLedgerRetentionToLiveConfig, + getUsageLedgerRetentionStatus, + parseUsageLedgerRetentionInput, + writeUsageLedgerRetentionToConfig, +} from "../../usage/ledger-retention-config"; +import { + getUsageLedgerRetentionJobState, + invalidateUsageLedgerRetentionRun, +} from "../../usage/ledger-retention-job"; import { jsonResponse } from "../auth-cors"; import { managementBodyTooLargeResponse, @@ -21,10 +31,12 @@ import type { ManagementContext } from "./context"; const INSPECTION_FAILED_MESSAGE = "Codex log inspection failed"; +/** Report whether the Log Guard schema cannot be inspected on this install. */ function inspectionUnavailable(report: CodexLogGuardStatus): boolean { return report.schema.state === "unavailable"; } +/** Map a Log Guard mutation result to its management HTTP status. */ function mutationStatus(result: CodexLogGuardMutationResult): number { if (result.ok) return 200; switch (result.error) { @@ -42,6 +54,7 @@ function mutationStatus(result: CodexLogGuardMutationResult): number { } } +/** Map a Log Guard compaction result to its management HTTP status. */ function compactStatus(result: CodexLogGuardCompactionResult): number { if (result.ok) return 200; switch (result.error) { @@ -59,6 +72,7 @@ function compactStatus(result: CodexLogGuardCompactionResult): number { } } +/** Serialize a Log Guard mutation result through the shared CORS-aware JSON helper. */ function mutationResponse( result: CodexLogGuardMutationResult, ctx: ManagementContext, @@ -68,6 +82,7 @@ function mutationResponse( : jsonResponse({ error: result.error }, mutationStatus(result), ctx.req, ctx.config); } +/** Serialize a Log Guard compaction result through the shared CORS-aware JSON helper. */ function compactResponse( result: CodexLogGuardCompactionResult, ctx: ManagementContext, @@ -85,6 +100,7 @@ function compactResponse( ); } +/** Parse the explicit Log Guard protection mode from a bounded management body. */ async function readProtectMode(ctx: ManagementContext): Promise<"compat" | "quiet" | Response> { let body: unknown; try { @@ -104,11 +120,48 @@ async function readProtectMode(ctx: ManagementContext): Promise<"compat" | "quie return mode; } -/** Codex Log Guard diagnostics plus explicit protection and maintenance mutations. */ +/** Storage diagnostics plus explicit protection, retention, and maintenance mutations. */ export async function handleStorageLogGuardRoutes(ctx: ManagementContext): Promise { const { req, url, config, deps } = ctx; const protectionDeps = deps.codexLogGuardProtectionDeps; + if (url.pathname === "/api/storage/usage-ledger-retention" && req.method === "GET") { + return jsonResponse({ + ...getUsageLedgerRetentionStatus(config), + job: getUsageLedgerRetentionJobState(), + }, 200, req, config); + } + + if (url.pathname === "/api/storage/usage-ledger-retention" && req.method === "PUT") { + let body: unknown; + try { + body = await readManagementJsonBody(req); + } catch (error) { + const tooLarge = managementBodyTooLargeResponse(error, req, config); + if (tooLarge) return tooLarge; + return jsonResponse({ error: "invalid_json" }, 400, req, config); + } + const previous = getUsageLedgerRetentionStatus(config); + const parsed = parseUsageLedgerRetentionInput(body, previous); + if (!parsed.ok) return jsonResponse({ error: parsed.error }, 400, req, config); + try { + const saved = writeUsageLedgerRetentionToConfig(parsed.policy); + applyUsageLedgerRetentionToLiveConfig(config, saved); + // Every policy change invalidates the snapshot captured by an older Worker. + // The old preparation may finish, but its generation can no longer commit. + invalidateUsageLedgerRetentionRun(); + // PUT changes policy only. Automatic enforcement belongs to the scheduler; + // there is no public manual trigger for destructive compaction. + return jsonResponse({ + ok: true, + ...getUsageLedgerRetentionStatus(config), + job: getUsageLedgerRetentionJobState(), + }, 200, req, config); + } catch { + return jsonResponse({ error: "config_write_failed" }, 500, req, config); + } + } + if (url.pathname === "/api/storage/codex-logs") { if (req.method !== "GET") return null; try { diff --git a/src/types.ts b/src/types.ts index d4b937d040..95229bb837 100644 --- a/src/types.ts +++ b/src/types.ts @@ -61,6 +61,7 @@ export type { OcxClaudeDesktopAssignment, OcxClaudeDesktopProfile, StorageCleanupPolicy, + UsageLedgerRetentionConfig, OcxCustomModel, OcxApiKeyEntry, OcxClientIntegrationsConfig, diff --git a/src/types/config.ts b/src/types/config.ts index 499ca59eb1..7468fd7f59 100644 --- a/src/types/config.ts +++ b/src/types/config.ts @@ -180,6 +180,19 @@ export interface StorageCleanupPolicy { nextRun?: number; } +/** + * Opt-in byte ceiling for the canonical `usage.jsonl` ledger. + * Persisted under `OcxConfig.usageLedgerRetention`; the feature is disabled by default. + * When enabled, older complete JSONL rows are dropped permanently so the file stays within + * `maxBytes`. The derived routing-history SQLite projection is disposable and rebuilt later. + */ +export interface UsageLedgerRetentionConfig { + /** When false/unset, the ledger is never rewritten. Default false. */ + enabled?: boolean; + /** Keep the newest complete JSONL rows within this many bytes. Floor 1 MiB. */ + maxBytes?: number; +} + /** 사용자가 대시보드에서 직접 추가한 커스텀 모델 정의. */ export interface OcxCustomModel { /** 고유 ID (crypto.randomUUID()) */ @@ -728,6 +741,8 @@ export interface OcxConfig { * See `src/storage/policy.ts`. */ storageCleanupPolicy?: StorageCleanupPolicy; + /** Opt-in cap for `usage.jsonl` and its disposable SQLite projection. Default OFF. */ + usageLedgerRetention?: UsageLedgerRetentionConfig; /** Generated API keys for external access to the proxy's /v1/responses endpoint. */ apiKeys?: OcxApiKeyEntry[]; /** Auto-start/sync the proxy from the Codex shim before launching Codex. Default true. */ diff --git a/src/usage/ledger-retention-config.ts b/src/usage/ledger-retention-config.ts new file mode 100644 index 0000000000..eee6f582d9 --- /dev/null +++ b/src/usage/ledger-retention-config.ts @@ -0,0 +1,101 @@ +import { statSync } from "node:fs"; +import { loadConfig, saveConfigPreservingClaudeCode } from "../config"; +import type { OcxConfig } from "../types"; +import { usageLogPath } from "./log"; +import { + DEFAULT_USAGE_LEDGER_MAX_BYTES, + MIN_USAGE_LEDGER_MAX_BYTES, + normalizeUsageLedgerRetention, + type UsageLedgerRetention, +} from "./ledger-retention"; + +export type UsageLedgerRetentionStatus = UsageLedgerRetention & { + currentBytes: number; + overLimit: boolean; +}; + +/** Read the opt-in policy from config. Unknown/malformed persisted keys fail closed. */ +export function readUsageLedgerRetentionFromConfig(config?: OcxConfig): UsageLedgerRetention { + const source = config ?? loadConfig(); + return normalizeUsageLedgerRetention(source.usageLedgerRetention); +} + +/** Strict live-write parser. Destructive settings reject unknown keys instead of ignoring typos. */ +export function parseUsageLedgerRetentionInput( + raw: unknown, + previous: UsageLedgerRetention = { enabled: false, maxBytes: DEFAULT_USAGE_LEDGER_MAX_BYTES }, +): { ok: true; policy: UsageLedgerRetention } | { ok: false; error: string } { + if (!raw || typeof raw !== "object" || Array.isArray(raw)) { + return { ok: false, error: "body must be a JSON object" }; + } + const row = raw as Record; + const allowed = new Set(["enabled", "maxBytes"]); + const unknownKey = Object.keys(row).find(key => !allowed.has(key)); + if (unknownKey) return { ok: false, error: `unknown field: ${unknownKey}` }; + + if (row.enabled !== undefined && typeof row.enabled !== "boolean") { + return { ok: false, error: "enabled must be a boolean" }; + } + if (row.maxBytes !== undefined) { + if ( + typeof row.maxBytes !== "number" + || !Number.isSafeInteger(row.maxBytes) + || row.maxBytes < MIN_USAGE_LEDGER_MAX_BYTES + ) { + return { + ok: false, + error: `maxBytes must be a safe integer >= ${MIN_USAGE_LEDGER_MAX_BYTES}`, + }; + } + } + + return { + ok: true, + policy: { + enabled: row.enabled === undefined ? previous.enabled : row.enabled, + maxBytes: row.maxBytes === undefined ? previous.maxBytes : row.maxBytes, + }, + }; +} + +/** Persist a complete normalized policy. The feature is never enabled implicitly. */ +export function writeUsageLedgerRetentionToConfig(policy: UsageLedgerRetention): UsageLedgerRetention { + const normalized = normalizeUsageLedgerRetention({ + enabled: policy.enabled, + maxBytes: policy.maxBytes, + }); + const config = loadConfig(); + config.usageLedgerRetention = { + enabled: normalized.enabled, + maxBytes: normalized.maxBytes, + }; + saveConfigPreservingClaudeCode(config); + return normalized; +} + +/** Mirror a persisted policy into the live server config after a management PUT. */ +export function applyUsageLedgerRetentionToLiveConfig( + config: OcxConfig, + policy: UsageLedgerRetention, +): void { + config.usageLedgerRetention = { + enabled: policy.enabled, + maxBytes: policy.maxBytes, + }; +} + +/** Bounded status projection for API/UI; missing ledger is reported as zero bytes. */ +export function getUsageLedgerRetentionStatus(config?: OcxConfig): UsageLedgerRetentionStatus { + const policy = readUsageLedgerRetentionFromConfig(config); + let currentBytes = 0; + try { + currentBytes = statSync(usageLogPath()).size; + } catch { + currentBytes = 0; + } + return { + ...policy, + currentBytes, + overLimit: policy.enabled && currentBytes > policy.maxBytes, + }; +} diff --git a/src/usage/ledger-retention-job.ts b/src/usage/ledger-retention-job.ts new file mode 100644 index 0000000000..f395464848 --- /dev/null +++ b/src/usage/ledger-retention-job.ts @@ -0,0 +1,383 @@ +import { chmodSync, statSync, unlinkSync } from "node:fs"; +import { dirname } from "node:path"; +import { renameAtomicFile } from "../lib/windows-atomic-replace"; +import { discardRequestHistoryProjection } from "../routing/history/discard-index"; +import { closeRequestHistoryIndex } from "../routing/history/indexer"; +import { getActiveTurnCount } from "../server/lifecycle"; +import { + StorageWorkerAdmissionBusyError, + terminateStorageWorker, + tryReserveStorageWorker, + withStorageWorkerSpawnGate, +} from "../storage/worker-lifecycle"; +import { usageLogPath } from "./log"; +import { + usageLedgerRevisionFromStat, + usageLedgerRevisionMatches, + type PreparedUsageLedgerCompaction, + type UsageLedgerCompactionPreparation, +} from "./ledger-retention"; +import { readUsageLedgerRetentionFromConfig } from "./ledger-retention-config"; + +export type UsageLedgerRetentionDeferredReason = "active_turns" | "source_changed"; + +export interface UsageLedgerRetentionJobOutcome { + ok: boolean; + skipped?: "disabled" | "missing" | "within_limit"; + deferred?: UsageLedgerRetentionDeferredReason; + error?: "worker_busy" | "worker_failed" | "commit_failed"; + beforeBytes?: number; + afterBytes?: number; + droppedBytes?: number; +} + +export interface UsageLedgerRetentionJobState { + status: "idle" | "running"; + startedAt?: number; + finishedAt?: number; + lastError?: string; + lastOutcome?: UsageLedgerRetentionJobOutcome; +} + +export interface UsageLedgerRetentionCommitDeps { + activeTurnCount?: () => number; + closeHistoryIndex?: () => void; + discardHistoryProjection?: (configDir: string) => boolean; + stat?: typeof statSync; + rename?: (source: string, destination: string) => void; + chmod?: typeof chmodSync; + unlink?: typeof unlinkSync; +} + +let state: UsageLedgerRetentionJobState = { status: "idle" }; +let inflight: Promise | null = null; +let activeWorker: Worker | null = null; +let cancelActiveRun: (() => void) | null = null; +let runGeneration = 0; +let lastWarningAt = 0; +const WARNING_INTERVAL_MS = 60_000; +const WORKER_TIMEOUT_MS = 10 * 60 * 1000; + +/** Remove a Worker candidate without surfacing path-bearing filesystem errors. */ +function discardCandidate(path: string, unlink: typeof unlinkSync = unlinkSync): void { + try { unlink(path); } catch { /* already absent / best effort */ } +} + +/** Emit at most one fixed, path-free retention warning per minute. */ +function warnRetentionFailure(): void { + const now = Date.now(); + if (now - lastWarningAt < WARNING_INTERVAL_MS) return; + lastWarningAt = now; + console.warn("[usage] usage ledger retention failed; it will be retried later"); +} + +/** + * Commit a Worker-prepared candidate only while no data-plane turn is active and + * only if the canonical ledger is byte-for-byte the same filesystem revision the + * Worker inspected. This function is intentionally synchronous: after the idle + * and revision checks, no request callback can interleave before the rename. + */ +export function commitPreparedUsageLedgerCompaction( + prepared: PreparedUsageLedgerCompaction, + deps: UsageLedgerRetentionCommitDeps = {}, +): UsageLedgerRetentionJobOutcome { + const activeTurnCount = deps.activeTurnCount ?? getActiveTurnCount; + const closeHistoryIndex = deps.closeHistoryIndex ?? closeRequestHistoryIndex; + const discardHistoryProjection = deps.discardHistoryProjection ?? discardRequestHistoryProjection; + const stat = deps.stat ?? statSync; + // Keep the final publication synchronous. The shared helper retries the short + // Windows sharing-violation window with sleepSync, so no request callback can + // interleave after the revision check and publish a newer append underneath us. + const rename = deps.rename ?? ((source: string, destination: string) => { + renameAtomicFile(source, destination, undefined, "usage-retention"); + }); + const chmod = deps.chmod ?? chmodSync; + const unlink = deps.unlink ?? unlinkSync; + + if (activeTurnCount() !== 0) { + discardCandidate(prepared.tempPath, unlink); + return { + ok: true, + deferred: "active_turns", + beforeBytes: prepared.beforeBytes, + afterBytes: prepared.beforeBytes, + droppedBytes: 0, + }; + } + + let currentRevision; + try { + currentRevision = usageLedgerRevisionFromStat(stat(prepared.path)); + } catch { + discardCandidate(prepared.tempPath, unlink); + return { + ok: true, + deferred: "source_changed", + beforeBytes: prepared.beforeBytes, + afterBytes: prepared.beforeBytes, + droppedBytes: 0, + }; + } + + if (!usageLedgerRevisionMatches(prepared.sourceRevision, currentRevision)) { + discardCandidate(prepared.tempPath, unlink); + return { + ok: true, + deferred: "source_changed", + beforeBytes: prepared.beforeBytes, + afterBytes: currentRevision.size, + droppedBytes: 0, + }; + } + + try { + // The index is a disposable projection of usage.jsonl. Drop its live handle + // before replacing the canonical source so Windows cannot hold the source-adjacent + // projection open during publication. + closeHistoryIndex(); + rename(prepared.tempPath, prepared.path); + try { chmod(prepared.path, 0o600); } catch { /* platform may ignore chmod */ } + + // Publication succeeded. Reclaim the now-stale derived SQLite projection immediately + // instead of waiting for a later history query to notice the source identity change. + // This cleanup must never reverse a successful canonical-ledger commit. + try { + const discarded = discardHistoryProjection(dirname(prepared.path)); + if (!discarded) { + console.warn("[usage] request-history projection cleanup was incomplete; a later history access will rebuild it"); + } + } catch { + console.warn("[usage] request-history projection cleanup failed; a later history access will rebuild it"); + } + + return { + ok: true, + beforeBytes: prepared.beforeBytes, + afterBytes: prepared.afterBytes, + droppedBytes: prepared.droppedBytes, + }; + } catch { + discardCandidate(prepared.tempPath, unlink); + return { + ok: false, + error: "commit_failed", + beforeBytes: prepared.beforeBytes, + afterBytes: prepared.beforeBytes, + droppedBytes: 0, + }; + } +} + +/** Return a detached snapshot of the process-local retention controller state. */ +export function getUsageLedgerRetentionJobState(): UsageLedgerRetentionJobState { + return { + ...state, + ...(state.lastOutcome ? { lastOutcome: { ...state.lastOutcome } } : {}), + }; +} + +/** Allocate the candidate name in the parent before the Worker can create it. */ +function retentionCandidatePath(path: string): string { + return `${path}.retention-${process.pid}-${crypto.randomUUID()}.tmp`; +} + +/** Run the expensive scan/copy phase in the shared, admission-controlled Worker lane. */ +function runInWorker(path: string, maxBytes: number): Promise { + const reservation = tryReserveStorageWorker(); + if (!reservation) return Promise.reject(new StorageWorkerAdmissionBusyError()); + const tempPath = retentionCandidatePath(path); + + return withStorageWorkerSpawnGate(() => new Promise((resolve, reject) => { + const requestId = crypto.randomUUID(); + let settled = false; + let worker: Worker; + try { + worker = new Worker(new URL("./ledger-retention-worker.ts", import.meta.url).href); + reservation.bind(worker); + } catch (error) { + reservation.release(); + reject(error); + return; + } + activeWorker = worker; + + const finish = (fn: () => void, cleanupCandidate = false) => { + if (settled) return; + settled = true; + cancelActiveRun = null; + clearTimeout(timer); + if (activeWorker === worker) activeWorker = null; + const afterTerminate = () => { + if (cleanupCandidate) discardCandidate(tempPath); + fn(); + }; + void terminateStorageWorker(worker).then(afterTerminate, afterTerminate); + }; + + const timer = setTimeout(() => { + finish(() => reject(new Error("usage_ledger_retention_worker_timeout")), true); + }, WORKER_TIMEOUT_MS); + + cancelActiveRun = () => { + finish(() => reject(new Error("aborted")), true); + }; + + worker.onmessage = (event: MessageEvent) => { + const data = event.data; + if (!data || typeof data !== "object" || Array.isArray(data)) return; + const message = data as Record; + if (message.requestId !== requestId) return; + if (message.type === "done" && message.result && typeof message.result === "object") { + finish(() => resolve(message.result as UsageLedgerCompactionPreparation)); + return; + } + if (message.type === "error") { + finish(() => reject(new Error("usage_ledger_retention_worker_failed")), true); + } + }; + worker.onerror = () => { + finish(() => reject(new Error("usage_ledger_retention_worker_failed")), true); + }; + + worker.postMessage({ + type: "run", + requestId, + path, + tempPath, + maxBytes, + env: { + ...(process.env.OPENCODEX_HOME ? { OPENCODEX_HOME: process.env.OPENCODEX_HOME } : {}), + }, + }); + })).catch(error => { + reservation.release(); + discardCandidate(tempPath); + throw error; + }); +} + +/** Execute one policy snapshot and discard its candidate if that snapshot becomes stale. */ +async function executeJob(generation: number): Promise { + const policy = readUsageLedgerRetentionFromConfig(); + if (!policy.enabled) { + if (generation === runGeneration) { + state = { + status: "idle", + startedAt: state.startedAt, + finishedAt: Date.now(), + lastOutcome: { ok: true, skipped: "disabled" }, + }; + } + return; + } + + try { + const prepared = await runInWorker(usageLogPath(), policy.maxBytes); + if (generation !== runGeneration) { + if (prepared.changed) discardCandidate(prepared.tempPath); + return; + } + const outcome: UsageLedgerRetentionJobOutcome = prepared.changed + ? commitPreparedUsageLedgerCompaction(prepared) + : { + ok: true, + skipped: prepared.reason, + beforeBytes: prepared.beforeBytes, + afterBytes: prepared.afterBytes, + droppedBytes: 0, + }; + if (!outcome.ok) warnRetentionFailure(); + state = { + status: "idle", + startedAt: state.startedAt, + finishedAt: Date.now(), + ...(outcome.ok ? {} : { lastError: outcome.error }), + lastOutcome: outcome, + }; + } catch (error) { + if (generation !== runGeneration) return; + const workerBusy = error instanceof StorageWorkerAdmissionBusyError; + const outcome: UsageLedgerRetentionJobOutcome = { + ok: false, + error: workerBusy ? "worker_busy" : "worker_failed", + }; + warnRetentionFailure(); + state = { + status: "idle", + startedAt: state.startedAt, + finishedAt: Date.now(), + lastError: outcome.error, + lastOutcome: outcome, + }; + } +} + +/** + * Invalidate the policy snapshot owned by any current run. + * + * Policy PUTs call this after persisting/applying the new settings. The old Worker + * may finish its read-only preparation, but its generation can no longer commit. + */ +export function invalidateUsageLedgerRetentionRun(): void { + runGeneration += 1; +} + +/** Start one asynchronous retention evaluation. */ +export function requestUsageLedgerRetentionRun(): + | { accepted: true; state: UsageLedgerRetentionJobState } + | { accepted: false; error: "already_running"; state: UsageLedgerRetentionJobState } { + if (inflight || state.status === "running") { + return { accepted: false, error: "already_running", state: getUsageLedgerRetentionJobState() }; + } + const generation = ++runGeneration; + state = { + status: "running", + startedAt: Date.now(), + ...(state.lastOutcome ? { lastOutcome: state.lastOutcome } : {}), + }; + const job = executeJob(generation); + inflight = job; + void job.finally(() => { + if (inflight !== job) return; + inflight = null; + if (generation !== runGeneration && state.status === "running") { + state = { + status: "idle", + startedAt: state.startedAt, + finishedAt: Date.now(), + ...(state.lastOutcome ? { lastOutcome: state.lastOutcome } : {}), + }; + } + }); + return { accepted: true, state: getUsageLedgerRetentionJobState() }; +} + +/** Join an active retention Worker during final server teardown. */ +export async function abortUsageLedgerRetentionJobAsync(): Promise { + runGeneration += 1; + const worker = activeWorker; + const job = inflight; + const cancel = cancelActiveRun; + cancelActiveRun = null; + cancel?.(); + if (worker) await terminateStorageWorker(worker); + if (job) await job.catch(() => undefined); + activeWorker = null; + inflight = null; + if (state.status === "running") { + state = { + status: "idle", + startedAt: state.startedAt, + finishedAt: Date.now(), + lastError: "aborted", + lastOutcome: { ok: false, error: "worker_failed" }, + }; + } +} + +/** Test reset for the module-local controller state. */ +export async function resetUsageLedgerRetentionJobForTests(): Promise { + await abortUsageLedgerRetentionJobAsync(); + state = { status: "idle" }; + lastWarningAt = 0; +} diff --git a/src/usage/ledger-retention-scheduler.ts b/src/usage/ledger-retention-scheduler.ts new file mode 100644 index 0000000000..6a6a825952 --- /dev/null +++ b/src/usage/ledger-retention-scheduler.ts @@ -0,0 +1,46 @@ +import { getUsageLedgerRetentionStatus } from "./ledger-retention-config"; +import { requestUsageLedgerRetentionRun } from "./ledger-retention-job"; + +const DEFAULT_INTERVAL_MS = 60_000; +let timer: ReturnType | null = null; +let startupTimer: ReturnType | null = null; + +/** Request one background run only when the current persisted policy is enabled and over limit. */ +function requestIfOverLimit(): void { + try { + const status = getUsageLedgerRetentionStatus(); + if (!status.enabled || !status.overLimit) return; + requestUsageLedgerRetentionRun(); + } catch { + // A later tick retries; scheduler failures never block the proxy. + } +} + +/** Poll only metadata on the main thread; file scanning/copying stays in the Worker job. */ +export function startUsageLedgerRetentionScheduler(intervalMs = DEFAULT_INTERVAL_MS): void { + if (timer) return; + timer = setInterval(requestIfOverLimit, intervalMs); + timer.unref?.(); +} + +/** Evaluate once after listeners bind so oversized ledgers are handled after startup. */ +export function scheduleUsageLedgerRetentionStartupRun(): void { + if (startupTimer) return; + startupTimer = setTimeout(() => { + startupTimer = null; + requestIfOverLimit(); + }, 0); + startupTimer.unref?.(); +} + +/** Stop both periodic and pending startup evaluations without touching an active Worker. */ +export function stopUsageLedgerRetentionScheduler(): void { + if (timer) { + clearInterval(timer); + timer = null; + } + if (startupTimer) { + clearTimeout(startupTimer); + startupTimer = null; + } +} diff --git a/src/usage/ledger-retention-worker.ts b/src/usage/ledger-retention-worker.ts new file mode 100644 index 0000000000..59a01d061a --- /dev/null +++ b/src/usage/ledger-retention-worker.ts @@ -0,0 +1,41 @@ +import { prepareUsageLedgerCompaction } from "./ledger-retention"; + +interface RunMessage { + type: "run"; + requestId: string; + path: string; + tempPath: string; + maxBytes: number; + env?: { OPENCODEX_HOME?: string }; +} + +/** Validate the fixed-shape message accepted by the retention Worker. */ +function isRunMessage(data: unknown): data is RunMessage { + if (!data || typeof data !== "object" || Array.isArray(data)) return false; + const row = data as Record; + return row.type === "run" + && typeof row.requestId === "string" + && typeof row.path === "string" + && typeof row.tempPath === "string" + && typeof row.maxBytes === "number"; +} + +declare const self: Worker; + +/** Prepare one candidate and return only fixed, path-free failures to the parent. */ +self.onmessage = (event: MessageEvent) => { + if (!isRunMessage(event.data)) return; + const { requestId, path, tempPath, maxBytes, env } = event.data; + try { + if (env?.OPENCODEX_HOME) process.env.OPENCODEX_HOME = env.OPENCODEX_HOME; + const result = prepareUsageLedgerCompaction(path, maxBytes, tempPath); + self.postMessage({ type: "done", requestId, result }); + } catch { + // Keep worker errors fixed and path-free: OPENCODEX_HOME may contain user information. + self.postMessage({ type: "error", requestId, message: "usage_ledger_retention_failed" }); + } finally { + try { + (self as unknown as { close?: () => void }).close?.(); + } catch { /* already closing */ } + } +}; diff --git a/src/usage/ledger-retention.ts b/src/usage/ledger-retention.ts new file mode 100644 index 0000000000..289b39d1f9 --- /dev/null +++ b/src/usage/ledger-retention.ts @@ -0,0 +1,249 @@ +import { + chmodSync, + closeSync, + existsSync, + fstatSync, + fsyncSync, + openSync, + readSync, + unlinkSync, + writeSync, +} from "node:fs"; + +export const DEFAULT_USAGE_LEDGER_MAX_BYTES = 1024 * 1024 * 1024; +export const MIN_USAGE_LEDGER_MAX_BYTES = 1024 * 1024; +const SCAN_CHUNK_BYTES = 1024 * 1024; + +/** Fully normalized policy used by the mutation path. */ +export interface UsageLedgerRetention { + enabled: boolean; + maxBytes: number; +} + +export interface UsageLedgerRevision { + dev: number; + ino: number; + size: number; + mtimeMs: number; + ctimeMs: number; +} + +export interface PreparedUsageLedgerCompaction { + changed: true; + path: string; + tempPath: string; + beforeBytes: number; + afterBytes: number; + droppedBytes: number; + sourceRevision: UsageLedgerRevision; +} + +export interface SkippedUsageLedgerCompaction { + changed: false; + path: string; + beforeBytes: number; + afterBytes: number; + droppedBytes: 0; + reason: "missing" | "within_limit"; +} + +export type UsageLedgerCompactionPreparation = + | PreparedUsageLedgerCompaction + | SkippedUsageLedgerCompaction; + +/** + * Normalize the destructive retention policy fail-closed. + * + * Unknown keys disable the feature rather than being silently stripped: a typo + * such as `maxByets` must never turn an intended large limit into the default. + * Invalid/unsafe byte values likewise disable the feature. A valid maxBytes is + * retained while disabled so toggling the feature off does not erase user choice. + */ +export function normalizeUsageLedgerRetention(raw: unknown): UsageLedgerRetention { + const disabled = { enabled: false, maxBytes: DEFAULT_USAGE_LEDGER_MAX_BYTES } as const; + if (raw === undefined || raw === null) return disabled; + if (typeof raw !== "object" || Array.isArray(raw)) return disabled; + + const row = raw as Record; + const allowed = new Set(["enabled", "maxBytes"]); + if (Object.keys(row).some(key => !allowed.has(key))) return disabled; + if (row.enabled !== undefined && typeof row.enabled !== "boolean") return disabled; + + const maxBytes = row.maxBytes ?? DEFAULT_USAGE_LEDGER_MAX_BYTES; + if ( + typeof maxBytes !== "number" + || !Number.isSafeInteger(maxBytes) + || maxBytes < MIN_USAGE_LEDGER_MAX_BYTES + ) { + return disabled; + } + return { enabled: row.enabled === true, maxBytes }; +} + +/** Snapshot the identity fields used to prove the source did not change. */ +export function usageLedgerRevisionFromStat(stat: { + dev: number | bigint; + ino: number | bigint; + size: number | bigint; + mtimeMs: number; + ctimeMs: number; +}): UsageLedgerRevision { + return { + dev: Number(stat.dev), + ino: Number(stat.ino), + size: Number(stat.size), + mtimeMs: Number(stat.mtimeMs), + ctimeMs: Number(stat.ctimeMs), + }; +} + +/** Exact revision comparison used immediately before the atomic replace. */ +export function usageLedgerRevisionMatches( + left: UsageLedgerRevision, + right: UsageLedgerRevision, +): boolean { + return left.dev === right.dev + && left.ino === right.ino + && left.size === right.size + && left.mtimeMs === right.mtimeMs + && left.ctimeMs === right.ctimeMs; +} + +/** Find the final complete-line delimiter before `endExclusive`. */ +function findLastNewline(fd: number, endExclusive: number): number { + const buffer = Buffer.allocUnsafe(SCAN_CHUNK_BYTES); + let end = endExclusive; + while (end > 0) { + const start = Math.max(0, end - buffer.length); + const length = end - start; + const read = readSync(fd, buffer, 0, length, start); + for (let index = read - 1; index >= 0; index -= 1) { + if (buffer[index] === 0x0a) return start + index; + } + end = start; + } + return -1; +} + +/** Find the next complete-line delimiter at or after `startInclusive`. */ +function findFirstNewline(fd: number, startInclusive: number, endExclusive: number): number { + const buffer = Buffer.allocUnsafe(SCAN_CHUNK_BYTES); + let start = startInclusive; + while (start < endExclusive) { + const length = Math.min(buffer.length, endExclusive - start); + const read = readSync(fd, buffer, 0, length, start); + if (read <= 0) return -1; + for (let index = 0; index < read; index += 1) { + if (buffer[index] === 0x0a) return start + index; + } + start += read; + } + return -1; +} + +/** Copy an exact byte range while tolerating short reads/writes. */ +function copyRange(sourceFd: number, targetFd: number, start: number, endExclusive: number): number { + const buffer = Buffer.allocUnsafe(SCAN_CHUNK_BYTES); + let offset = start; + let written = 0; + while (offset < endExclusive) { + const wanted = Math.min(buffer.length, endExclusive - offset); + const read = readSync(sourceFd, buffer, 0, wanted, offset); + if (read <= 0) break; + let cursor = 0; + while (cursor < read) { + cursor += writeSync(targetFd, buffer, cursor, read - cursor); + } + offset += read; + written += read; + } + return written; +} + +/** + * Build a compacted candidate without mutating the live ledger. + * + * The candidate contains only complete JSONL rows. The start scan has no fixed + * probe ceiling, so a single row larger than the copy chunk cannot leak a + * partial prefix. The backward scan drops an unterminated crash tail. If one + * complete row itself exceeds maxBytes it is dropped, preserving the hard cap. + * + * `candidatePath` lets the parent process own the temporary path before a Worker + * starts. That ownership is required so timeout/shutdown can remove a candidate + * even when the Worker produced it but its completion message was never claimed. + */ +export function prepareUsageLedgerCompaction( + path: string, + maxBytes: number, + candidatePath?: string, +): UsageLedgerCompactionPreparation { + if (!Number.isSafeInteger(maxBytes) || maxBytes < MIN_USAGE_LEDGER_MAX_BYTES) { + throw new RangeError(`maxBytes must be a safe integer >= ${MIN_USAGE_LEDGER_MAX_BYTES}`); + } + if (!existsSync(path)) { + return { changed: false, path, beforeBytes: 0, afterBytes: 0, droppedBytes: 0, reason: "missing" }; + } + + const sourceFd = openSync(path, "r"); + let tempPath: string | null = null; + try { + const sourceStat = fstatSync(sourceFd); + const sourceRevision = usageLedgerRevisionFromStat(sourceStat); + const beforeBytes = sourceRevision.size; + if (beforeBytes <= maxBytes) { + return { + changed: false, + path, + beforeBytes, + afterBytes: beforeBytes, + droppedBytes: 0, + reason: "within_limit", + }; + } + + const lastNewline = findLastNewline(sourceFd, beforeBytes); + const completeEnd = lastNewline < 0 ? 0 : lastNewline + 1; + const desiredStart = Math.max(0, completeEnd - maxBytes); + let retainedStart = 0; + if (desiredStart > 0) { + const previousByte = Buffer.allocUnsafe(1); + const startsAtRowBoundary = + readSync(sourceFd, previousByte, 0, 1, desiredStart - 1) === 1 + && previousByte[0] === 0x0a; + if (startsAtRowBoundary) { + retainedStart = desiredStart; + } else { + const newline = findFirstNewline(sourceFd, desiredStart, completeEnd); + retainedStart = newline < 0 ? completeEnd : newline + 1; + } + } + + tempPath = candidatePath ?? `${path}.retention-${process.pid}-${crypto.randomUUID()}.tmp`; + const targetFd = openSync(tempPath, "wx", 0o600); + let afterBytes = 0; + try { + afterBytes = copyRange(sourceFd, targetFd, retainedStart, completeEnd); + fsyncSync(targetFd); + } finally { + closeSync(targetFd); + } + try { chmodSync(tempPath, 0o600); } catch { /* best-effort on platforms that ignore chmod */ } + + const result: PreparedUsageLedgerCompaction = { + changed: true, + path, + tempPath, + beforeBytes, + afterBytes, + droppedBytes: beforeBytes - afterBytes, + sourceRevision, + }; + tempPath = null; + return result; + } finally { + closeSync(sourceFd); + if (tempPath) { + try { unlinkSync(tempPath); } catch { /* best-effort cleanup */ } + } + } +} diff --git a/tests/cli/cli-storage-inspect.test.ts b/tests/cli/cli-storage-inspect.test.ts index 336cc7fc90..9a32ca10ff 100644 --- a/tests/cli/cli-storage-inspect.test.ts +++ b/tests/cli/cli-storage-inspect.test.ts @@ -291,3 +291,77 @@ describe("ocx integration native", () => { } }); }); + +const RETENTION_STATUS = { + enabled: false, + maxBytes: 128 * 1024 * 1024, + currentBytes: 64 * 1024 * 1024, + overLimit: false, + job: { status: "idle" }, +}; + +describe("ocx storage usage-limit", () => { + test("show reads the usage-ledger retention status", async () => { + const { calls, deps } = harness(() => ({ json: RETENTION_STATUS })); + const cap = capture(); + try { + expect(await handleStorageCommand(["usage-limit", "show"], deps)).toBe(0); + } finally { + cap.restore(); + } + expect(calls).toEqual([{ method: "GET", path: "/api/storage/usage-ledger-retention", body: undefined }]); + }); + + test("set sends only the fields explicitly given", async () => { + const { calls, deps } = harness(() => ({ json: { ok: true, ...RETENTION_STATUS } })); + const cap = capture(); + try { + expect(await handleStorageCommand(["usage-limit", "set", "--mib", "1024"], deps)).toBe(0); + } finally { + cap.restore(); + } + expect(calls[0]).toMatchObject({ + method: "PUT", + path: "/api/storage/usage-ledger-retention", + body: { maxBytes: 1024 * 1024 * 1024 }, + }); + expect(calls[0]?.body).not.toHaveProperty("enabled"); + }); + + test("set can explicitly enable without changing the saved ceiling", async () => { + const { calls, deps } = harness(() => ({ json: { ok: true, ...RETENTION_STATUS, enabled: true } })); + const cap = capture(); + try { + expect(await handleStorageCommand(["usage-limit", "set", "--enabled", "true"], deps)).toBe(0); + } finally { + cap.restore(); + } + expect(calls[0]?.body).toEqual({ enabled: true }); + }); + + test("set with no fields is rejected locally", async () => { + const { calls, deps } = harness(() => ({ json: RETENTION_STATUS })); + const cap = capture(); + let code: number; + try { + code = await handleStorageCommand(["usage-limit", "set"], deps); + } finally { + cap.restore(); + } + expect(code).not.toBe(0); + expect(calls).toHaveLength(0); + }); + + test("manual run is no longer exposed", async () => { + const { calls, deps } = harness(() => ({ json: { ok: true, started: true } })); + const cap = capture(); + let code: number; + try { + code = await handleStorageCommand(["usage-limit", "run"], deps); + } finally { + cap.restore(); + } + expect(code).not.toBe(0); + expect(calls).toHaveLength(0); + }); +}); diff --git a/tests/config/settings-usage-ledger-retention.test.ts b/tests/config/settings-usage-ledger-retention.test.ts new file mode 100644 index 0000000000..87a12796fa --- /dev/null +++ b/tests/config/settings-usage-ledger-retention.test.ts @@ -0,0 +1,66 @@ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { + getConfigPath, + getDefaultConfig, + loadConfig, + saveConfig, + validateConfigCandidate, +} from "../../src/config"; + +let testHome = ""; +const previousOpenCodexHome = process.env.OPENCODEX_HOME; + +beforeEach(() => { + testHome = mkdtempSync(join(tmpdir(), "ocx-usage-ledger-config-")); + process.env.OPENCODEX_HOME = testHome; +}); + +afterEach(() => { + if (previousOpenCodexHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousOpenCodexHome; + rmSync(testHome, { recursive: true, force: true }); +}); + +test("usageLedgerRetention is accepted as a first-class config section", () => { + const candidate = { + ...getDefaultConfig(), + usageLedgerRetention: { enabled: true, maxBytes: 8 * 1024 * 1024 }, + }; + + const result = validateConfigCandidate(candidate); + + expect(result.ok).toBe(true); + if (result.ok) { + expect(result.config.usageLedgerRetention).toEqual(candidate.usageLedgerRetention); + } +}); + +test("a malformed usageLedgerRetention section degrades without dropping providers", () => { + saveConfig({ + ...getDefaultConfig(), + usageLedgerRetention: { enabled: true, maxBytes: 8 * 1024 * 1024 }, + }); + const raw = JSON.parse(readFileSync(getConfigPath(), "utf8")) as Record; + raw.usageLedgerRetention = { enabled: true, maxByets: 8 * 1024 * 1024 }; + writeFileSync(getConfigPath(), JSON.stringify(raw, null, 2), "utf8"); + + const loaded = loadConfig(); + + expect(loaded.usageLedgerRetention).toBeUndefined(); + expect(loaded.providers.openai).toBeDefined(); +}); + +test("partial usageLedgerRetention config remains valid for hand-edited files", () => { + saveConfig({ + ...getDefaultConfig(), + usageLedgerRetention: { enabled: true, maxBytes: 8 * 1024 * 1024 }, + }); + const raw = JSON.parse(readFileSync(getConfigPath(), "utf8")) as Record; + raw.usageLedgerRetention = { enabled: true }; + writeFileSync(getConfigPath(), JSON.stringify(raw, null, 2), "utf8"); + + expect(loadConfig().usageLedgerRetention).toEqual({ enabled: true }); +}); diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index e8e712c839..f0343b0ad6 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -999,6 +999,7 @@ "settings-oauth-open-browser.test.ts": "config", "settings-startup-health-seam.test.ts": "config", "settings-stream-mode.test.ts": "config", + "settings-usage-ledger-retention.test.ts": "config", "shutdown-drain.test.ts": "service", "shutdown-launcher.test.ts": "service", "sidebar-routes.test.ts": "server", diff --git a/tests/server/system-routes.test.ts b/tests/server/system-routes.test.ts index 45907fb6ab..7b499c57a2 100644 --- a/tests/server/system-routes.test.ts +++ b/tests/server/system-routes.test.ts @@ -118,6 +118,7 @@ describe("windows replace retry counters", () => { "lab-automation", "lab-ledger", "storage-cleanup", + "usage-retention", "tray", ]; for (const publisher of publishers) renameAtomicFile("a", "b", flakyIo(1), publisher); @@ -130,6 +131,7 @@ describe("windows replace retry counters", () => { "prompt-journal:EBUSY", "storage-cleanup:EBUSY", "tray:EBUSY", + "usage-retention:EBUSY", ]); // @ts-expect-error a path is not a ReplacePublisher renameAtomicFile("a", "b", flakyIo(0), "C:\\Users\\someone\\.opencodex"); diff --git a/tests/storage/api-storage.test.ts b/tests/storage/api-storage.test.ts index b5ca548e3d..561bd2c041 100644 --- a/tests/storage/api-storage.test.ts +++ b/tests/storage/api-storage.test.ts @@ -148,3 +148,28 @@ describe("GET /api/storage", () => { } }); }); + +describe("usage ledger retention management route", () => { + test("keeps GET/PUT policy management while removing the manual run endpoint", async () => { + const server = startServer(0); + try { + const status = await fetch(new URL("/api/storage/usage-ledger-retention", server.url)); + expect(status.status).toBe(200); + expect(await status.json()).toMatchObject({ enabled: false, maxBytes: expect.any(Number), currentBytes: expect.any(Number) }); + + const updated = await fetch(new URL("/api/storage/usage-ledger-retention", server.url), { + method: "PUT", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ enabled: true, maxBytes: 8 * 1024 * 1024 }), + }); + expect(updated.status).toBe(200); + expect(await updated.json()).toMatchObject({ enabled: true, maxBytes: 8 * 1024 * 1024 }); + + const removed = await fetch(new URL("/api/storage/usage-ledger-retention/run", server.url), { method: "POST" }); + expect(removed.status).toBe(404); + expect(await removed.json()).toMatchObject({ error: { type: "not_found", code: "not_found" } }); + } finally { + await server.stop(true); + } + }); +}); diff --git a/tests/usage/usage-ledger-scanner.test.ts b/tests/usage/usage-ledger-scanner.test.ts index b5bf34adc8..a2999f342a 100644 --- a/tests/usage/usage-ledger-scanner.test.ts +++ b/tests/usage/usage-ledger-scanner.test.ts @@ -1,6 +1,6 @@ import { afterEach, beforeEach, describe, expect, test } from "bun:test"; import { createHash } from "node:crypto"; -import { appendFileSync, mkdtempSync, renameSync, rmSync, writeFileSync } from "node:fs"; +import { appendFileSync, existsSync, mkdtempSync, readdirSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { @@ -10,6 +10,24 @@ import { UsageLedgerRebuildRequiredError, } from "../../src/usage/ledger-scanner"; import { usageLogIdentityKey, usageLogPath, type PersistedUsageEntry } from "../../src/usage/log"; +import { discardRequestHistoryProjection } from "../../src/routing/history/discard-index"; +import { historyIndexPath } from "../../src/routing/history/schema"; +import { + DEFAULT_USAGE_LEDGER_MAX_BYTES, + MIN_USAGE_LEDGER_MAX_BYTES, + normalizeUsageLedgerRetention, + prepareUsageLedgerCompaction, + usageLedgerRevisionMatches, +} from "../../src/usage/ledger-retention"; +import { parseUsageLedgerRetentionInput } from "../../src/usage/ledger-retention-config"; +import { + commitPreparedUsageLedgerCompaction, + getUsageLedgerRetentionJobState, + invalidateUsageLedgerRetentionRun, + requestUsageLedgerRetentionRun, + resetUsageLedgerRetentionJobForTests, +} from "../../src/usage/ledger-retention-job"; +import { getConfigPath, getDefaultConfig, saveConfig } from "../../src/config"; let testDir = ""; let previousHome: string | undefined; @@ -496,3 +514,313 @@ describe("usage ledger cooperative scanner", () => { })).rejects.toBe(sentinel); }); }); + +const homes: string[] = []; + +/** Allocate one isolated filesystem home and remember it for teardown. */ +function home(): string { + const dir = mkdtempSync(join(tmpdir(), "ocx-ledger-retention-")); + homes.push(dir); + return dir; +} + +/** Build one JSONL row whose encoded byte length is exactly `totalBytes`. */ +function jsonlRowOfSize(requestId: string, totalBytes: number, fill = "x"): string { + const empty = `${JSON.stringify({ requestId, filler: "" })}\n`; + const overhead = Buffer.byteLength(empty); + if (totalBytes < overhead) throw new Error("row target is smaller than JSONL overhead"); + const row = `${JSON.stringify({ requestId, filler: fill.repeat(totalBytes - overhead) })}\n`; + if (Buffer.byteLength(row) !== totalBytes) throw new Error("row byte sizing drifted"); + return row; +} + +afterEach(async () => { + await resetUsageLedgerRetentionJobForTests(); + for (const dir of homes.splice(0)) rmSync(dir, { recursive: true, force: true }); +}); + +async function waitForRetentionIdle(timeoutMs = 10_000): Promise { + const deadline = Date.now() + timeoutMs; + while (Date.now() < deadline) { + if (getUsageLedgerRetentionJobState().status === "idle") return; + await Bun.sleep(10); + } + throw new Error("timed out waiting for usage ledger retention job"); +} + +describe("usage ledger retention v2", () => { + test("missing or unknown persisted config keys stay Unlimited", () => { + expect(normalizeUsageLedgerRetention(undefined).enabled).toBe(false); + expect(normalizeUsageLedgerRetention({ enabled: true, maxByets: 8 * 1024 * 1024 })).toEqual({ + enabled: false, + maxBytes: DEFAULT_USAGE_LEDGER_MAX_BYTES, + }); + }); + + test("live writes reject unknown config keys instead of silently stripping them", () => { + const parsed = parseUsageLedgerRetentionInput( + { enabled: true, maxByets: 8 * 1024 * 1024 }, + { enabled: false, maxBytes: DEFAULT_USAGE_LEDGER_MAX_BYTES }, + ); + expect(parsed.ok).toBe(false); + if (parsed.ok) throw new Error("expected strict parser failure"); + expect(parsed.error).toContain("maxByets"); + }); + + test("partial live writes preserve the previous enabled state", () => { + const maxBytes = 8 * 1024 * 1024; + expect(parseUsageLedgerRetentionInput( + { maxBytes }, + { enabled: true, maxBytes: DEFAULT_USAGE_LEDGER_MAX_BYTES }, + )).toEqual({ ok: true, policy: { enabled: true, maxBytes } }); + }); + + test("unsafe or below-floor byte limits disable destructive retention", () => { + for (const maxBytes of [Number.MAX_SAFE_INTEGER + 1, MIN_USAGE_LEDGER_MAX_BYTES - 1, MIN_USAGE_LEDGER_MAX_BYTES + 0.5]) { + expect(normalizeUsageLedgerRetention({ enabled: true, maxBytes }).enabled).toBe(false); + } + }); + + test("normalizes an explicitly enabled safe byte limit", () => { + const maxBytes = 8 * 1024 * 1024; + expect(normalizeUsageLedgerRetention({ enabled: true, maxBytes })).toEqual({ enabled: true, maxBytes }); + }); + + test("drops an oversized single row instead of retaining a partial JSONL fragment", () => { + const dir = home(); + const path = join(dir, "usage.jsonl"); + const huge = `${JSON.stringify({ requestId: "huge", payload: "x".repeat(MIN_USAGE_LEDGER_MAX_BYTES + 1024) })}\n`; + writeFileSync(path, huge); + + const prepared = prepareUsageLedgerCompaction(path, MIN_USAGE_LEDGER_MAX_BYTES); + expect(prepared.changed).toBe(true); + if (!prepared.changed) throw new Error("expected compaction"); + expect(prepared.afterBytes).toBe(0); + expect(readFileSync(prepared.tempPath, "utf8")).toBe(""); + }); + + test("drops an unterminated crash tail while retaining a complete row at the ceiling", () => { + const dir = home(); + const path = join(dir, "usage.jsonl"); + const complete = jsonlRowOfSize("complete", MIN_USAGE_LEDGER_MAX_BYTES); + const partial = JSON.stringify({ requestId: "partial", filler: "y".repeat(1024) }); + writeFileSync(path, complete + partial); + + const prepared = prepareUsageLedgerCompaction(path, MIN_USAGE_LEDGER_MAX_BYTES); + expect(prepared.changed).toBe(true); + if (!prepared.changed) throw new Error("expected compaction"); + const retained = readFileSync(prepared.tempPath, "utf8"); + expect(retained).toBe(complete); + expect(retained.endsWith("\n")).toBe(true); + expect(retained).not.toContain("partial"); + }); + + test("retains the row when the byte ceiling lands exactly on its start boundary", () => { + const dir = home(); + const path = join(dir, "usage.jsonl"); + const old = `${JSON.stringify({ requestId: "old" })}\n`; + const newest = jsonlRowOfSize("new", MIN_USAGE_LEDGER_MAX_BYTES, "b"); + writeFileSync(path, old + newest); + + const prepared = prepareUsageLedgerCompaction(path, MIN_USAGE_LEDGER_MAX_BYTES); + expect(prepared.changed).toBe(true); + if (!prepared.changed) throw new Error("expected compaction"); + expect(prepared.afterBytes).toBe(MIN_USAGE_LEDGER_MAX_BYTES); + expect(readFileSync(prepared.tempPath, "utf8")).toBe(newest); + }); + + test("never starts the candidate in the middle of a long row", () => { + const dir = home(); + const path = join(dir, "usage.jsonl"); + const first = `${JSON.stringify({ requestId: "old", filler: "a".repeat(MIN_USAGE_LEDGER_MAX_BYTES + 128) })}\n`; + const second = `${JSON.stringify({ requestId: "new", filler: "b".repeat(64 * 1024) })}\n`; + writeFileSync(path, first + second); + + const prepared = prepareUsageLedgerCompaction(path, MIN_USAGE_LEDGER_MAX_BYTES); + expect(prepared.changed).toBe(true); + if (!prepared.changed) throw new Error("expected compaction"); + const retained = readFileSync(prepared.tempPath, "utf8"); + expect(retained).toBe(second); + expect(() => JSON.parse(retained.trim())).not.toThrow(); + }); + + test("uses a parent-owned candidate path when one is supplied", () => { + const dir = home(); + const path = join(dir, "usage.jsonl"); + const tempPath = join(dir, "owned-retention.tmp"); + writeFileSync(path, jsonlRowOfSize("old", MIN_USAGE_LEDGER_MAX_BYTES) + `${JSON.stringify({ requestId: "new" })}\n`); + + const prepared = prepareUsageLedgerCompaction(path, MIN_USAGE_LEDGER_MAX_BYTES, tempPath); + expect(prepared.changed).toBe(true); + if (!prepared.changed) throw new Error("expected compaction"); + expect(prepared.tempPath).toBe(tempPath); + expect(existsSync(tempPath)).toBe(true); + }); + + test("discards the derived request-history database and sidecars from an isolated config home", () => { + const dir = home(); + const path = historyIndexPath(dir); + writeFileSync(path, "main"); + writeFileSync(`${path}-wal`, "wal"); + writeFileSync(`${path}-shm`, "shm"); + + expect(discardRequestHistoryProjection(dir)).toBe(true); + expect(existsSync(path)).toBe(false); + expect(existsSync(`${path}-wal`)).toBe(false); + expect(existsSync(`${path}-shm`)).toBe(false); + }); + + test("revision comparator detects a source mutation before commit", () => { + const revision = { dev: 1, ino: 2, size: 3, mtimeMs: 4, ctimeMs: 5 }; + expect(usageLedgerRevisionMatches(revision, revision)).toBe(true); + expect(usageLedgerRevisionMatches(revision, { ...revision, size: 4 })).toBe(false); + }); + + test("defers commit while a request turn is active and discards the candidate", () => { + const dir = home(); + const path = join(dir, "usage.jsonl"); + const old = `${JSON.stringify({ requestId: "old", filler: "a".repeat(MIN_USAGE_LEDGER_MAX_BYTES) })}\n`; + const latest = `${JSON.stringify({ requestId: "new" })}\n`; + writeFileSync(path, old + latest); + const prepared = prepareUsageLedgerCompaction(path, MIN_USAGE_LEDGER_MAX_BYTES); + if (!prepared.changed) throw new Error("expected compaction"); + + const result = commitPreparedUsageLedgerCompaction(prepared, { activeTurnCount: () => 1 }); + expect(result.deferred).toBe("active_turns"); + expect(existsSync(prepared.tempPath)).toBe(false); + expect(readFileSync(path, "utf8")).toBe(old + latest); + }); + + test("does not overwrite an append that landed after Worker preparation", () => { + const dir = home(); + const path = join(dir, "usage.jsonl"); + const old = `${JSON.stringify({ requestId: "old", filler: "a".repeat(MIN_USAGE_LEDGER_MAX_BYTES) })}\n`; + const latest = `${JSON.stringify({ requestId: "new" })}\n`; + writeFileSync(path, old + latest); + const prepared = prepareUsageLedgerCompaction(path, MIN_USAGE_LEDGER_MAX_BYTES); + if (!prepared.changed) throw new Error("expected compaction"); + + const appended = `${JSON.stringify({ requestId: "after-prepare" })}\n`; + appendFileSync(path, appended); + const result = commitPreparedUsageLedgerCompaction(prepared, { activeTurnCount: () => 0 }); + expect(result.deferred).toBe("source_changed"); + expect(existsSync(prepared.tempPath)).toBe(false); + expect(readFileSync(path, "utf8")).toBe(old + latest + appended); + }); + + test("closes the derived history index before replace and discards it only after publication", () => { + const dir = home(); + const path = join(dir, "usage.jsonl"); + const old = `${JSON.stringify({ requestId: "old", filler: "a".repeat(MIN_USAGE_LEDGER_MAX_BYTES) })}\n`; + const latest = `${JSON.stringify({ requestId: "new" })}\n`; + writeFileSync(path, old + latest); + const prepared = prepareUsageLedgerCompaction(path, MIN_USAGE_LEDGER_MAX_BYTES); + if (!prepared.changed) throw new Error("expected compaction"); + const expected = readFileSync(prepared.tempPath, "utf8"); + let closed = false; + let replaced = false; + let discarded = false; + + const result = commitPreparedUsageLedgerCompaction(prepared, { + activeTurnCount: () => 0, + closeHistoryIndex: () => { closed = true; }, + rename: (from, to) => { + expect(closed).toBe(true); + renameSync(from, to); + replaced = true; + }, + discardHistoryProjection: configDir => { + expect(replaced).toBe(true); + expect(configDir).toBe(dir); + discarded = true; + return true; + }, + }); + expect(result.ok).toBe(true); + expect(result.droppedBytes).toBeGreaterThan(0); + expect(discarded).toBe(true); + expect(readFileSync(path, "utf8")).toBe(expected); + }); + + test("derived projection cleanup failure does not reverse a successful canonical commit", () => { + const dir = home(); + const path = join(dir, "usage.jsonl"); + const old = `${JSON.stringify({ requestId: "old", filler: "a".repeat(MIN_USAGE_LEDGER_MAX_BYTES) })}\n`; + const latest = `${JSON.stringify({ requestId: "new" })}\n`; + writeFileSync(path, old + latest); + const prepared = prepareUsageLedgerCompaction(path, MIN_USAGE_LEDGER_MAX_BYTES); + if (!prepared.changed) throw new Error("expected compaction"); + const expected = readFileSync(prepared.tempPath, "utf8"); + const warn = console.warn; + console.warn = () => undefined; + try { + const result = commitPreparedUsageLedgerCompaction(prepared, { + activeTurnCount: () => 0, + rename: renameSync, + discardHistoryProjection: () => { throw new Error("projection busy"); }, + }); + expect(result.ok).toBe(true); + expect(readFileSync(path, "utf8")).toBe(expected); + } finally { + console.warn = warn; + } + }); + + test("does not discard the derived projection when canonical publication fails", () => { + const dir = home(); + const path = join(dir, "usage.jsonl"); + const old = `${JSON.stringify({ requestId: "old", filler: "a".repeat(MIN_USAGE_LEDGER_MAX_BYTES) })}\n`; + const latest = `${JSON.stringify({ requestId: "new" })}\n`; + const original = old + latest; + writeFileSync(path, original); + const prepared = prepareUsageLedgerCompaction(path, MIN_USAGE_LEDGER_MAX_BYTES); + if (!prepared.changed) throw new Error("expected compaction"); + let discarded = false; + + const result = commitPreparedUsageLedgerCompaction(prepared, { + activeTurnCount: () => 0, + closeHistoryIndex: () => undefined, + rename: () => { throw new Error("rename failed"); }, + discardHistoryProjection: () => { + discarded = true; + return true; + }, + }); + expect(result.ok).toBe(false); + expect(result.error).toBe("commit_failed"); + expect(discarded).toBe(false); + expect(existsSync(prepared.tempPath)).toBe(false); + expect(readFileSync(path, "utf8")).toBe(original); + }); + + test("invalidating a policy generation prevents a prepared Worker candidate from publishing", async () => { + const dir = home(); + const previousRetentionHome = process.env.OPENCODEX_HOME; + process.env.OPENCODEX_HOME = dir; + try { + const maxBytes = MIN_USAGE_LEDGER_MAX_BYTES; + const config = { + ...getDefaultConfig(), + usageLedgerRetention: { enabled: true, maxBytes }, + }; + saveConfig(config); + + const path = join(dir, "usage.jsonl"); + const original = jsonlRowOfSize("old", maxBytes) + jsonlRowOfSize("new", 256); + writeFileSync(path, original); + + const started = requestUsageLedgerRetentionRun(); + expect(started.accepted).toBe(true); + invalidateUsageLedgerRetentionRun(); + await waitForRetentionIdle(); + + expect(readFileSync(path, "utf8")).toBe(original); + expect(getUsageLedgerRetentionJobState().lastOutcome).toBeUndefined(); + expect(readdirSync(dir).filter(name => name.includes(".retention-")).length).toBe(0); + } finally { + if (previousRetentionHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousRetentionHome; + expect(getConfigPath()).not.toBe(join(dir, "config.json")); + } + }); +});