Skip to content

Post-deploy verification: usage metering + data erasure #517

Description

@kcarnold

Why this is an issue and not a PR checklist

These checks can only run against a deployed environment, so they outlive the PR
that introduces them. Close this once they've actually been run against prod.

Context: feat/llm-usage-metering (stacked on privacy) authenticates the OpenAI
proxy, meters every model request into llm_usage, and routes both deletion paths
through one eraseLoggedData().

1. Schema + the auth.db -> app.db rename

The shared DB was renamed (db.ts renames a pre-existing auth.db, -wal/-shm
included, on first open). On the prod volume, after rollout:

  • app.db exists in DATA_DIR, auth.db is gone.
  • Existing users still sign in (i.e. the rename carried the data, it didn't
    start a fresh DB).
  • PRAGMA user_version; is >= 1 and llm_usage exists:
    sqlite3 $DATA_DIR/app.db 'pragma user_version; .tables'

2. Metering attributes spend to the right key

  • A signed-in request writes a row keyed to the Better Auth user id, on the main
    key.
  • A sessionless request (demo mode / pre-sign-in editor) returns 200 and is
    metered to the demo bucket — i.e. OPENAI_DEMO_API_KEY is really reaching the
    container. If this 401s, demo mode is broken in prod.
  • Spend shows up: GET /api/usage_summary?secret=$LOG_SECRET returns per-user
    rows with non-null costUsd, and unpricedModels is empty (if it isn't, the
    served model is missing from backend/src/pricing.ts).
  • Sanity-check the totals against the OpenAI dashboard for the same window —
    main project vs the Thoughtful-demo project. This is the whole point of the
    table; if it doesn't reconcile, something is being missed.

3. deletePosthogPerson actually works (carried over from #503)

This is the one promise in the privacy policy currently backed by a code comment
saying "verify the endpoint shape against your PostHog version before relying on it".
Both deletion paths now depend on it, and the tests mock it, so nothing verifies it.

  • POSTHOG_PERSONAL_API_KEY and POSTHOG_PROJECT_ID are set in prod (without
    them it logs a warning and no-ops — deletion silently doesn't happen).
  • With a throwaway account: capture some analytics, then
    DELETE /api/me/activity, and confirm the person and their events are gone
    from the PostHog project.
  • Confirm the management-API endpoint shape in backend/src/posthog.ts matches
    our PostHog version (a 4xx is reported to captureException, not thrown — so a
    wrong URL fails quietly).

4. Deletion semantics end to end

  • DELETE /api/me/activity (withdrawal): study logs + PostHog person gone, the
    account still works, and new usage rows keep accruing for it.
  • Account deletion (Better Auth delete-user): all of the above, plus the
    user's llm_usage rows move to the deleted bucket with their token counts
    intact — the spend total for the window must not drop.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions