diff --git a/.github/workflows/openwiki-update.yml b/.github/workflows/openwiki-update.yml deleted file mode 100644 index 1fca26f..0000000 --- a/.github/workflows/openwiki-update.yml +++ /dev/null @@ -1,70 +0,0 @@ -name: OpenWiki Update - -on: - # Refresh on a schedule rather than after every merge. A per-merge trigger raced the - # merges themselves and produced a documentation pull request per change; one nightly - # run batches them. This also matches the "scheduled OpenWiki workflow" that AGENTS.md - # and CLAUDE.md already describe. - schedule: - - cron: "0 9 * * *" - workflow_dispatch: - -permissions: - contents: write - pull-requests: write - -jobs: - update: - runs-on: ubuntu-latest - steps: - - name: Check out repository - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 - with: - persist-credentials: true - # Full history lets OpenWiki diff HEAD against the commit it last documented. - fetch-depth: 0 - - - name: Set up Node.js - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4 - with: - node-version: "22" - - - name: Install OpenWiki - # Keep OpenWiki's LangChain peers on the last known-compatible release set. - run: npm install --global openwiki@0.3.2 langchain@1.5.6 @langchain/core@1.2.6 mermaid@11.16.0 jsdom@29.1.1 - - - name: Verify OpenWiki startup - run: openwiki --help - - - name: Run OpenWiki - run: openwiki code --update --print - env: - OPENWIKI_PROVIDER: openrouter - OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }} - # Pin a specific model rather than the `openrouter/free` router: the router picks a - # different model per request, which breaks OpenWiki's multi-turn agent loop partway - # through. - # - # This is a paid model, unlike the free one it replaces, so a refresh now depends on - # the account balance. That is the deliberate trade: the free tier kept failing this - # workflow outright, and documentation that never regenerates is worth less than the - # fractions of a cent a run costs. - OPENWIKI_MODEL_ID: google/gemini-3.7-flash - - - name: Create OpenWiki update pull request - uses: peter-evans/create-pull-request@22a9089034f40e5a961c8808d113e2c98fb63676 # v7 - with: - add-paths: | - openwiki - AGENTS.md - CLAUDE.md - .github/workflows/openwiki-update.yml - branch: openwiki/update - commit-message: "docs: update OpenWiki" - title: "docs: update OpenWiki" - body: | - Automated OpenWiki documentation update. - - This PR refreshes StoryRail's generated repository documentation after changes landed on `main`. - - Review generated documentation for accuracy before merging. diff --git a/AGENTS.md b/AGENTS.md index bb65005..6756670 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,7 +4,7 @@ StoryRail is an open-source, agent-first editorial control plane for solo publishers and small editorial teams. It turns raw sources into researched, reviewed, publishable stories through a visible, operator-supervised agentic newsroom workflow. StoryRail manages editorial state and will publish through APIs and replaceable adapters; it is not a page-building CMS. -Keep `Source`, `Story`, and `Article` distinct. `Story` is the central editorial object. PostgreSQL is authoritative for persisted editorial state; agent memory must never become the database. Obscura is a planned optional extraction adapter, not a foundational dependency. OpenWiki is automatically refreshed by GitHub Actions after qualifying `main`-branch changes; implementation agents must not run it as part of ordinary change work. +Keep `Source`, `Story`, and `Article` distinct. `Story` is the central editorial object. PostgreSQL is authoritative for persisted editorial state; agent memory must never become the database. Obscura is a planned optional extraction adapter, not a foundational dependency. OpenWiki is optional documentation tooling that maintainers run manually when useful; implementation agents must not run it as part of ordinary change work. ## Change workflow @@ -47,6 +47,16 @@ This repository has a generated `openwiki/` evidence index. It is optional just- - Treat source code and tests as authoritative. A brief's unknowns and review items are verification gaps, not automatic requirements. - Prefer the narrowest quiet validation that proves the changed behavior. Preserve complete failure output. -The scheduled OpenWiki GitHub Actions workflow refreshes the repository wiki. Do not hand-edit generated OpenWiki pages unless explicitly asked; prefer updating source code/docs and letting OpenWiki regenerate. +OpenWiki is not run by CI. A maintainer can refresh the generated index locally with `openwiki code --update --print` when the OpenWiki CLI is installed and configured. The repository has no package script or setup command for OpenWiki. Do not hand-edit generated OpenWiki pages unless explicitly asked; update source code and ordinary documentation instead. + + + +# This is NOT the Next.js you know + +This version has breaking changes — APIs, conventions, and file structure may all differ from your training data. Read the relevant guide in `node_modules/next/dist/docs/` (resolved from this file's directory; in monorepos the `next` package may not be visible from the repo root) before writing any code. Heed deprecation notices. + +This block is written and re-added by `next dev` — verify at `node_modules/next/dist/server/lib/generate-agent-files.js`. Removing it from a diff only re-creates the uncommitted change; committing it with your work keeps the tree clean. + + diff --git a/CLAUDE.md b/CLAUDE.md index a981a34..3d4ed42 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -7,6 +7,6 @@ This repository has a generated `openwiki/` evidence index. It is optional just- - Treat source code and tests as authoritative. A brief's unknowns and review items are verification gaps, not automatic requirements. - Prefer the narrowest quiet validation that proves the changed behavior. Preserve complete failure output. -The scheduled OpenWiki GitHub Actions workflow refreshes the repository wiki. Do not hand-edit generated OpenWiki pages unless explicitly asked; prefer updating source code/docs and letting OpenWiki regenerate. +OpenWiki is optional documentation tooling and is not run by CI. A maintainer may refresh the generated index locally with `openwiki code --update --print` when the CLI is installed and configured. Do not hand-edit generated OpenWiki pages unless explicitly asked; prefer updating source code and ordinary documentation. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index fc921db..d72172a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -37,8 +37,12 @@ The single application package uses pnpm scripts: - `pnpm dev` starts the local Next.js development server. - `pnpm build` creates the production build. - `pnpm start` serves an existing production build. +- `pnpm migrate` applies pending PostgreSQL schema migrations. +- `pnpm migrate:status` reports migration status without applying migrations. +- `pnpm migrate:adopt` records existing migrations as applied without executing them. - `pnpm test` runs the focused Vitest suite once. -- `pnpm test:postgres` runs only the PostgreSQL persistence integration suite for Source evidence, Stories, and Story-Source attachments and requires `STORYRAIL_TEST_DATABASE_URL`. +- `pnpm test:postgres` runs the PostgreSQL persistence and schema migration integration suites and requires `STORYRAIL_TEST_DATABASE_URL`. +- `pnpm test:e2e` runs Playwright browser acceptance tests and requires `STORYRAIL_TEST_DATABASE_URL` for a disposable `storyrail_test` database. - `pnpm test:watch` runs Vitest in watch mode. - `pnpm typecheck` generates Next.js route and framework types, then checks strict TypeScript types without emitting files. - `pnpm lint` runs ESLint. @@ -49,20 +53,23 @@ Agents may write or update the code, tests, and configuration behind these comma ## Server runtime configuration -The server-only Source-evidence runtime requires these production variable names: +The server runtime reads its configuration from the variable names documented in `.env.example`: - `STORYRAIL_DATABASE_URL` -- `FIRECRAWL_API_KEY` +- `STORYRAIL_OPENROUTER_BASE_URL` (optional) +- `STORYRAIL_CREDENTIAL_KEY` (for encrypted site credentials) +- `STORYRAIL_SITE_ID` (optional; selects a Site when an installation has more than one) +- `STORYRAIL_OPERATOR_ID` (required for actions attributed to an operator) -The separate server-only Story runtime requires only `STORYRAIL_DATABASE_URL`; Story creation, Source attachment, and Story inspection do not require Firecrawl. +Provider credentials such as Firecrawl and OpenRouter keys are managed as site credentials in the application, rather than through `FIRECRAWL_API_KEY` environment configuration. `.env.example` documents names only. Never commit credentials, connection strings, or working example values. Unit tests inject external services and require no production database or provider access. PostgreSQL migrations must be applied before a composed runtime can persist editorial state; application runtime does not run them automatically. -`.env.example` documents names only. Never commit credentials, connection strings, or working example values. Runtime unit tests inject Pool, fetch, UUID, and clock substitutes, so they require no real PostgreSQL or Firecrawl access. PostgreSQL migrations must be applied externally before a composed runtime can persist Source evidence or Story state; application runtime does not execute them. Ordinary validation must never make Firecrawl, other provider, or production database requests. +Ordinary validation must not make requests to production databases or external providers. ## PostgreSQL integration tests -Source-evidence, Story, and Story-Source attachment persistence integration tests run against PostgreSQL 18.4 itself. They do not use mocks, testcontainers, an embedded database, or a simulated PostgreSQL implementation. +PostgreSQL integration tests run against PostgreSQL 18.4 itself. They do not use mocks, testcontainers, an embedded database, or a simulated PostgreSQL implementation. -Provide the test-only connection through `STORYRAIL_TEST_DATABASE_URL`. Never use a production `DATABASE_URL`. The configured database name must be exactly `storyrail_test`; the suite connects and verifies that name before any destructive setup. It never creates or drops a database, but it does drop and recreate the `storyrail` schema, applies migrations 0012, 0017, and 0018 in order, and truncates the two evidence tables, Stories table, and Story-Source attachment table between cases. Use a disposable local test database with no data outside this test workflow that depends on the `storyrail` schema. +Provide the test-only connection through `STORYRAIL_TEST_DATABASE_URL`. Never use a production database URL. The configured database name must be exactly `storyrail_test`. These tests are destructive: persistence tests drop and recreate the `storyrail` schema, while migration-runner tests create and drop a separate `storyrail_migration_runner_test` database. Use a disposable PostgreSQL instance and a test role with the permissions those operations require; do not point either suite at data you need to keep. When `STORYRAIL_TEST_DATABASE_URL` is absent, `pnpm test` skips the PostgreSQL suite while continuing to run every non-PostgreSQL test. The dedicated command fails before Vitest when the variable is absent. Run the integration suite explicitly with a test URL whose database component is `storyrail_test`: @@ -71,7 +78,9 @@ STORYRAIL_TEST_DATABASE_URL='postgresql://postgres:postgres@127.0.0.1:5432/story pnpm test:postgres ``` -Credentials, host, and port may differ locally. Do not commit connection strings or credentials. The integration suite owns and closes only the Pool it creates; the server-only application runtime separately owns only the Pool created for its own composed instance. +Credentials, host, and port may differ locally. Do not commit connection strings or credentials. + +Browser acceptance tests use the same `STORYRAIL_TEST_DATABASE_URL` and require Chromium. They reset the test database's `storyrail` schema and migration ledger, then apply the complete migration history before starting the application. Install the browser once with `pnpm exec playwright install chromium`, then run `pnpm test:e2e`. Use only a disposable `storyrail_test` database. ## Continuous integration @@ -84,13 +93,15 @@ pnpm lint pnpm typecheck pnpm test STORYRAIL_TEST_DATABASE_URL='postgresql://postgres:postgres@127.0.0.1:5432/storyrail_test' pnpm test:postgres +pnpm exec playwright install --with-deps chromium +STORYRAIL_TEST_DATABASE_URL='postgresql://postgres:postgres@127.0.0.1:5432/storyrail_test' pnpm test:e2e pnpm build git diff --check ``` -GitHub Actions runs these commands independently for pull requests targeting `main`. CI also runs for -pushes to `main` and can be started manually. A green CI run confirms the automated checks passed for -that revision; it does not replace Chris's explicit approval to merge. +GitHub Actions runs these checks for pull requests targeting `main`, pushes to `main`, and manual +workflow dispatch. OpenWiki is not part of CI and is run manually when a maintainer wants to refresh +the generated documentation index. pnpm dependency lifecycle scripts fail closed until reviewed. Record each approval in the committed `pnpm-workspace.yaml` `allowBuilds` map with an exact package version; never approve an unversioned package or all dependency builds. A version change requires a new script review before installation can proceed. diff --git a/README.md b/README.md index ae36277..ee6ff26 100644 --- a/README.md +++ b/README.md @@ -1,159 +1,64 @@ # StoryRail [![CI](https://github.com/BlackSwampAI/StoryRail/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/BlackSwampAI/StoryRail/actions/workflows/ci.yml) +[![License: AGPL-3.0-only](https://img.shields.io/badge/License-AGPL--3.0--only-blue.svg)](LICENSE) +[![Node.js 24](https://img.shields.io/badge/Node.js-24-339933?logo=node.js&logoColor=white)](https://nodejs.org/) +[![pnpm 11.20.0](https://img.shields.io/badge/pnpm-11.20.0-F69220?logo=pnpm&logoColor=white)](https://pnpm.io/) +[![Pre-alpha](https://img.shields.io/badge/status-pre--alpha-orange.svg)](#project-status) -> An agentic editorial system where evidence, claims, model actions, review decisions, and publication provenance are inspectable and mechanically constrained instead of merely prompted. +StoryRail is an agent-first editorial control plane for solo publishers and small editorial teams. It turns source material into researched, reviewed stories, with editorial state and evidence preserved in PostgreSQL. It is a newsroom workflow, not a page-building CMS. -StoryRail helps solo publishers and small editorial teams preserve source evidence, decide what deserves coverage, organize Stories, and run bounded editorial agents whose work can be checked rather than trusted. It is a headless editorial system, not a page-building CMS. +**StoryRail is pre-alpha software. It has no authentication and is not ready for public or production deployment.** -Plenty of tools can chain a researcher, a writer, and an editor together. What is unusual here is that the chain is held to its evidence: a reader can open any claim and see the passage that supports it, a Writer that cites something the evidence does not contain is refused before anything is written, a Director cannot approve work without quoting the passage it judged, and every external tool call becomes part of the durable record. +## Editorial objects -**Status: pre-alpha and under active development. StoryRail is not production-ready.** +StoryRail keeps three objects distinct: -## What is StoryRail? +- **Source** is incoming material and its extraction history. Sources retain provenance and can be attached to coverage without losing their original records. +- **Story** is the central editorial object: a decision to pursue and organize coverage. A Story can group many Sources and track work through review and publication. +- **Article** is a versioned work product belonging to a Story. Its ordered blocks distinguish factual claims from context and headings. Claims carry citations to a Source, a specific evidence record, and the supporting passage. -StoryRail separates evidence acquisition, editorial decisions, writing, review, and publication into explicit stages with durable state and human supervision. +An Article citation is checked against preserved evidence before it can be recorded. The newsroom can open a claim to inspect its supporting passage and follow the Source. PostgreSQL is authoritative for persisted editorial state; agent memory is not. -The constraints are mechanical rather than prompted. An Article is an ordered list of blocks, and a block that asserts a fact must carry the Source, the evidence record, and the passage it relied on. That passage is checked against the evidence before anything is persisted, in the domain and again in a database constraint, so an unsupported claim is as impossible to record as an invalid state transition. +## Workflow -Its central distinction is **Source ≠ Story**: +```text +Submit URL → preserve and extract evidence → triage into a Story + → research and assign → draft cited Article → review + → approve → publish → optionally deliver to a destination +``` -- A **Source** is incoming evidence, such as a submitted URL and its extraction history. -- A **Story** is an editorial decision to pursue and organize coverage. -- An **Article** is a durable, versioned editorial work product separate from its Story. Supervised Writer execution creates the first Article and immutable Revision 1. +Operators can work each step directly or authorize Autopilot to run the existing workflows in sequence. Policy runs and tool calls are durable records, with reconciliation workflows for work interrupted by a process failure. Agents have bounded tool access, and external tool results are treated as untrusted input. -One Story may draw on many Sources. A Source may be skipped or attached to an existing Story; submitting a URL never creates a Story automatically. +StoryRail keeps **editorial publication** separate from **delivery**. Approval and publication record the Story's editorial state. A separate delivery workflow can send an Article from a published Story to a configured destination, including WordPress as Gutenberg blocks, while preserving delivery outcomes and reconciliation decisions. Destinations are replaceable adapters. -## Why StoryRail? +## Screenshots -Traditional CMS products begin with an article editor. StoryRail begins earlier: +These screenshots show the actual newsroom and claim reader with fictional demo data. See the [capture notes](docs/screenshots/README.md). -```text -source evidence → editorial decision → Story → assignment → writing → review → publication -``` +![StoryRail newsroom](docs/screenshots/newsroom.png) -Small teams need to preserve provenance, make automation inspectable, and keep editorial and publishing decisions under operator control. Every stage can be driven by hand; automation is an operator-authorised policy over the same durable workflows, never a separate path around them. - -## Current workflow - -```mermaid -flowchart LR - URL[Submitted URL] --> FC[Firecrawl extraction] - FC --> RAW[Immutable raw evidence] - RAW -->|successful extraction| PREP[Automatic Prepared Evidence] - RAW -->|failed extraction| INBOX[Source Inbox] - PREP --> EVIDENCE_REVIEW[Evidence review] - EVIDENCE_REVIEW --> INBOX - INBOX --> DECIDE{Editorial decision} - DECIDE -->|New Story| NEW[Persisted Story + Source] - DECIDE -->|Existing Story| EXISTING[Attach Source to Story] - DECIDE -->|Skip| SKIP[Durable skip decision] - NEW --> RESEARCH[Optional Researcher: retrieve and attach more Sources] - EXISTING --> RESEARCH - RESEARCH --> PROPOSE[Optional supervised Assignment Editor suggestion] - PROPOSE --> ASSIGNMENT_REVIEW[Operator review and editing] - ASSIGNMENT_REVIEW --> ASSIGN[Durable Assignment + intake-to-assigned transition] - ASSIGN --> WRITE[Supervised Writer + Article Revision 1] - WRITE --> GROUND{Every claim supported by its cited evidence?} - GROUND -->|No| CORRECT[One citation correction, then refusal] - GROUND -->|Yes| SUBMIT[Article sent to review] - SUBMIT --> DIRECTOR[Director review, quoting what it judged] - DIRECTOR --> OPERATOR{Review decision} - OPERATOR -->|Approve| APPROVED[Approved] - OPERATOR -->|Request changes| CHANGES[Changes requested] - CHANGES --> REVISE[Supervised Writer + next immutable revision] - REVISE --> GROUND - APPROVED --> PUBLISH[Operator publication + terminal transition] -``` +![Article claim provenance](docs/screenshots/article-provenance.png) + +## Highlights + +- Durable Source intake, immutable extraction attempts, evidence preparation, and triage into a new Story, an existing Story, or a skipped item. +- Research, operator-reviewed assignments, bounded agent profiles, and durable run records. +- Cited Article revisions with evidence-grounding checks and inspectable claim provenance. +- Operator-controlled review decisions, publication, and separate destination delivery. +- Durable Autopilot policy runs, recorded tool calls, and reconciliation for interrupted work. +- Per-Site settings and encrypted connector credentials, allowing multiple newsroom Sites to be configured independently. +- PostgreSQL migrations and persistence adapters that keep editorial state explicit and auditable. + +## Get started + +### Requirements + +- Node.js `24.18.0` (the `.nvmrc` version; supported range is `>=24.15.0 <25`) +- pnpm `11.20.0` through Corepack +- PostgreSQL -Every step above can be taken by hand. **Autopilot** runs the same sequence unattended from an Intake Story, optionally researching first, writing each durable record through the same workflows with the operator as the actor. - -Prepared Evidence is model-derived, cleaned evidence. It never replaces the immutable raw extraction; both histories remain available for audit and survive triage and reload. - -Durable Profiles configure the Assignment Editor, Writer, and Director/editor-in-chief roles. After the supervised Writer creates an immutable revision, the operator explicitly submits it to review, runs the Director, and records an approval or request-changes decision. A request-changes decision lets the Writer create the next immutable revision from the exact historical evidence, Director review, and authoritative operator reason. The Director is advisory, and the existing two-cycle Story limit bounds Articles at Revision 3. - -## What works today - -- URL Source preservation with conservative canonicalization and exact-duplicate detection. -- Firecrawl v2 Markdown extraction using its automatic proxy strategy. -- Rejection of obvious challenge/interstitial responses as failed extraction attempts. -- Immutable, append-ordered raw extraction history, including durable failures and retries. -- A PostgreSQL-backed Source Inbox with durable **New Story**, **Existing Story**, and **Skip** triage. -- Persistent Story queues, Story creation, Source-to-Story attachments, and Story inspection. -- Automatic Prepared Evidence generation after successful new extraction, plus explicit append-only preparation retries through a provider-neutral model boundary, LangChain, and OpenRouter. -- Immutable successful and failed preparation history alongside the original raw evidence. -- Reconstruction of raw and prepared evidence from PostgreSQL after triage or browser reload. -- Immutable, PostgreSQL-backed Agent Profiles with built-in Assignment Editor, General Writer, and Director configurations plus custom Writer creation and optional provider-neutral model selection. -- Durable manual Assignments with Writer selection from immutable Agent Profiles and a server-derived snapshot of every attached Source identity. -- The first persisted Story transition, `intake` to `assigned`, committed atomically with its Assignment and durable transition receipt/activity. -- Supervised Assignment Editor proposal generation through the provider-neutral structured-model boundary, with no browsing or tools. -- Durable, append-ordered AgentRun history that records the exact Story, evidence references, Writer candidates, model, prompt version, requester, outcome, and proposal or safe failure. -- Supervised Writer execution with Assignment-selected identity, Profile model override or the newsroom's configured Writer model, durable Article Revision 1, and an `assigned` to `in_progress` transition. -- Supervised review submission, a durable Director AgentRun against the exact evidence IDs recorded by the Writer run, and an operator-owned ReviewDecision that atomically moves the Story to Approved or Changes Requested. -- Explicit operator rejection from Intake, Assigned, In Progress, In Review, or Changes Requested, with a required reason and an atomic terminal Story transition receipt. -- Supervised Writer revisions after Request Changes, with immutable Revision 2/3 history, exact evidence reuse, operator-owned revision direction, and an atomic return to In Progress. -- Operator review and editing of suggestions in the existing Assignment form; the manual Assignment remains the authoritative state-mutation boundary and remains attributed to the operator. -- Operator publication of an Approved Story as a durable terminal transition with a required reason. -- **Cited Articles.** A Revision is an ordered list of blocks. A `claim` block must carry at least one citation naming the Source, the evidence record, and the passage relied on; `context` blocks are the Writer's own prose and carry none. Both rules hold in the domain and in database constraints. -- **Mechanical grounding.** Every quote is checked against the evidence the Assignment actually carries before anything is persisted. Typography, re-wrapping, and over-escaping are forgiven; paraphrase, invention, and quoting across a paragraph break are not. A refused draft records which citations failed and what they claimed to quote. -- **One citation correction.** A Writer whose citations do not hold is handed the specific findings against it and may correct them once, checked again by the same rule. A corrected draft is recorded as corrected, never as clean. -- **Grounding measurement.** Every Revision reports how much of its prose is attributed to evidence and how much occurs verbatim in that evidence, derived on demand so it applies to Revisions written before citations existed. -- **A provenance reader.** Any claim in the Article opens to show the passage it rests on, the Source as a followable link, and whether the record read was prepared or raw. Uncited prose is labelled as the Writer's own framing. -- **A Director that must point at what it read.** Six checks including claim support, each required to quote the passage of the Article it judges, verified against that Article before the review is recorded. -- **A working record.** Runs, transitions, and review decisions interleaved in the order they happened, with the model, duration, and — for a refusal — the exact passages that could not be supported. -- **Bounded tool access.** Tools declare themselves through an open registry with JSON Schema, so an operator's own tools can be added. Two ceilings bound an exchange, calls are recorded durably, and tool output is handed to models as untrusted data. -- **A Researcher.** Reads the evidence already attached, follows what it points at, retrieves those pages, and attaches what is worth citing. Only a page it actually retrieved can be attached, and retrieved material becomes a Source with its own immutable extraction. -- **A newsroom that remembers.** The Researcher can search what StoryRail has already published, by subject, and is shown the earlier reporting with the Sources behind it. Prior reporting is deliberately not evidence: it carries no evidence record, so a citation naming it is refused by the same grounding check as any other unsupported citation. -- **Autopilot.** An operator-authorised policy that runs an Intake Story to publication unattended, optionally researching first. Every durable record is still written by the same workflows with the operator as the actor, and every reason says the decision was made under autopilot. - -## Where StoryRail is going - -The editorial path from Source to publication is implemented end to end and can run unattended. The work ahead is durability, curation, and delivery rather than new stages: - -- **Durable automation and reconciliation.** Autopilot sequencing is still in-memory. Nothing durable records that a Story is under a policy run, so a process that dies between steps leaves a run marked `running` forever. Tool calls are recorded after the external call rather than before it. Both need the treatment AgentRuns already got. -- **A correction-scope invariant.** The citation correction turn is told not to rewrite unrelated blocks; nothing yet enforces it. -- **Richer automation provenance.** An autopilot decision is attributed to the operator who authorised the run, with a reason that says so. A distinct execution mode would let an audit separate "Chris authorised this" from "the system executed this particular decision." -- **A knowledge corpus.** House style as instructions, and reference knowledge as citable evidence, kept deliberately distinct so a Writer can never cite the style guide as support for a news claim. -- **Publishing destinations.** Publication is a durable editorial transition today; where a published Story is delivered remains a separate, replaceable concern. - -## Core concepts - -- **Source** — preserved incoming evidence. A URL Source retains the exact submitted URL and a conservative canonical URL. -- **Raw Extraction** — an immutable Firecrawl success or failure record. Retries append new records rather than overwrite history. -- **Prepared Evidence** — an immutable model-derived attempt to clean a successful raw extraction. It is optional and never authoritative over raw evidence. -- **Source Inbox** — the queue of preserved Sources awaiting a final editorial decision. -- **Triage Decision** — an attributable, durable choice to create a Story, attach to an existing Story, or skip coverage. -- **Story** — the central editorial object that groups evidence and will carry work through the editorial lifecycle. -- **Agent Profile** — an immutable configuration snapshot for a bounded editorial persona and optional provider-neutral model selection; profiles do not execute agents. -- **Assignment** — an immutable operator-created brief that selects a Writer Profile, records angle/brief/optional constraints and provenance, and snapshots attached Source identities. -- **Assignment Proposal** — a supervised Assignment Editor suggestion that prefills the manual Assignment form but cannot create an Assignment or transition a Story. -- **AgentRun** — one immutable execution record containing bounded input references, configuration, timing, and a structured success or failure outcome. -- **Writer and Article** — supervised Writer execution creates immutable Article revisions from the Assignment and exact durable evidence; it cannot browse, use tools, or send work to review. -- **Director** — an independently supervised advisory review role whose recommendation cannot mutate Article or Story state, and which must quote the passage each of its checks judged. -- **Article Block** — one ordered piece of a Revision, labelled `claim`, `context`, or `heading`. The label decides whether attribution is required or forbidden. -- **Citation** — the Source, the evidence record, and the verbatim passage a claim rests on, stored so support can be checked rather than trusted. -- **Grounding** — the mechanical check that every cited passage appears in the evidence it names, owned by the Source it names. It runs before anything durable is written. -- **Researcher** — a bounded role that retrieves further evidence for a Story and attaches what is worth citing, recording every tool call it makes. -- **Tool Call** — a durable record of what an agent reached for and what came back. It is an audit fact, not a copy of the material, which becomes evidence with its own record. -- **Autopilot** — an operator-authorised policy that runs the existing workflows in sequence without further clicks. It decides only when each step runs; it writes nothing itself. - -## Architecture - -StoryRail is a Next.js 16 / React 19 application backed by PostgreSQL. Its domain and application layers define provider-neutral editorial rules and ports; external systems are attached through replaceable adapters. Firecrawl is the current extraction adapter, while LangChain and OpenRouter provide the current structured-model path for Prepared Evidence. - -PostgreSQL is authoritative for editorial state. Source extractions, evidence preparations, attachments, triage decisions, Assignments, and Story transition receipts preserve auditable facts rather than relying on agent memory or overwriting history. - -See the [OpenWiki architecture overview](openwiki/architecture/overview.md) for deeper code-grounded documentation, or browse the [technical documentation index](openwiki/index.md). - -## Getting started - -### Prerequisites - -- Node.js 24 (`package.json` requires `>=24.15.0 <25`; `.nvmrc` pins 24.18.0) -- pnpm 11.20.0 through Corepack -- PostgreSQL with an application database you control -- A Firecrawl API key for URL extraction -- An OpenRouter API key and model name only if you want to prepare evidence +Some workflows need provider credentials: Firecrawl for URL extraction, and credentials for the model or delivery provider being used. Local workflows such as browsing persisted newsroom state do not need all providers configured. ### Install and run @@ -161,48 +66,36 @@ See the [OpenWiki architecture overview](openwiki/architecture/overview.md) for corepack enable pnpm install --frozen-lockfile cp .env.example .env -pnpm dev -``` - -Before starting the app, configure `.env` and bring the database up to date: - -```bash -STORYRAIL_DATABASE_URL=postgresql://user:password@127.0.0.1:5432/storyrail pnpm migrate ``` -`pnpm migrate:status` reports what a database has, what it still needs, and anything contradictory — a migration edited after it was applied, two files that took the same number, or one that started and never finished. Migrations are still plain SQL in `database/migrations/`, applied in numeric order, and can still be run with any tool you prefer; what the runner adds is that the database records which ones it has, in `public.storyrail_schema_migrations`. - -A database created before that ledger existed has the schema and no record of it. Tell the runner what is already there, naming the last migration you know was applied: +Set `STORYRAIL_DATABASE_URL`, `STORYRAIL_OPERATOR_ID`, and `STORYRAIL_CREDENTIAL_KEY` in `.env`. The operator ID identifies local editorial actions; the credential key is needed to save encrypted provider credentials in Settings. Apply migrations to your development database with Node's env-file option: ```bash -STORYRAIL_DATABASE_URL=... pnpm migrate:adopt --through 0061-durable-policy-runs.sql +node --env-file=.env scripts/migrate.ts up +pnpm dev ``` -Adopted migrations are recorded as adopted rather than applied, because the runner did not run them and cannot vouch for them. Anything after the one you name is then applied normally by `pnpm migrate`. - -Open [http://localhost:3133](http://localhost:3133) to use the development newsroom. StoryRail runs on port 3133 so it does not collide with other local services on the usual Next.js default. +Open [http://localhost:3133](http://localhost:3133). The `pnpm migrate` shortcut reads `STORYRAIL_DATABASE_URL` from the process environment, so export that variable before using the shortcut. Migrations are run explicitly; the app does not apply them at startup. -## Environment variables +### Configuration -| Variable | Required for | Purpose | -| ------------------------------- | -------------------------------- | ------------------------------------------------------------------------------------------- | -| `STORYRAIL_DATABASE_URL` | All persisted workflows | PostgreSQL connection string for editorial state. | -| `STORYRAIL_CREDENTIAL_KEY` | Any stored connector credential | 32 random bytes in base64 that every stored credential is encrypted under. Not recoverable. | -| `STORYRAIL_SITE_ID` | Installations with a second Site | Selects the Site this process serves; unset means the Site the installation started with. | -| `STORYRAIL_OPERATOR_ID` | Operator-attributed HTTP actions | Identifies the current fixed development operator; this is not authentication. | -| `STORYRAIL_OPENROUTER_BASE_URL` | Optional provider override | Absolute HTTP(S) base URL for an OpenRouter-compatible provider endpoint. | -| `STORYRAIL_TEST_DATABASE_URL` | PostgreSQL and browser tests | Points to a disposable database named exactly `storyrail_test`. | +`.env.example` lists the supported server environment variables: -Connector credentials and model selection are no longer environment variables. The OpenRouter and Firecrawl keys are per-Site secrets held encrypted in `storyrail.site_credentials`, and the model each agent role runs on is per-Site configuration in `storyrail.site_settings`. Both are resolved when a run needs them rather than when a process starts, so a change takes effect on the next request. Credentials are write-only over HTTP: `PUT` and `DELETE /api/site-credentials/[slot]` set and remove one, and `GET /api/site-settings` reports only which slots are configured and the last four characters of each. No endpoint returns a stored credential. +- `STORYRAIL_DATABASE_URL` — PostgreSQL connection for persisted editorial state. +- `STORYRAIL_CREDENTIAL_KEY` — Base64-encoded 32-byte key used to encrypt stored provider credentials. Generate with `openssl rand -base64 32`; keep it outside the database. +- `STORYRAIL_SITE_ID` — Selects the Site served by this process when multiple Sites exist. +- `STORYRAIL_OPERATOR_ID` — Fixed operator identity for development HTTP actions; this is not authentication. +- `STORYRAIL_OPENROUTER_BASE_URL` — Optional base URL override for an OpenRouter-compatible provider. -Normal Story, Inbox, triage, inspection, Agent Profile, and manual Assignment workflows need no credential at all, and an installation with none configured stays usable: only the runs that reach outside fail, and they fail as recorded Agent Run and extraction failures rather than as errors out of a route. +Provider credentials and model choices are configured per Site in the newsroom settings. Do not commit `.env` or real credentials. Because the app has no authentication, keep the development server private to your machine or a trusted network. -## Development and validation +## Development -Available project scripts: +Useful commands from `package.json`: ```bash pnpm dev +pnpm migrate:status pnpm format:check pnpm lint pnpm typecheck @@ -212,23 +105,20 @@ pnpm test:e2e pnpm build ``` -`pnpm test:postgres` requires `STORYRAIL_TEST_DATABASE_URL`. See [OpenWiki's engineering workflow](openwiki/engineering-workflow.md) and [CONTRIBUTING.md](CONTRIBUTING.md) for the full maintainer-owned verification sequence. - -The browser acceptance suite uses a disposable database and a dedicated Next.js server. Install -its browser once with `pnpm exec playwright install chromium`, then run it with -`STORYRAIL_TEST_DATABASE_URL=postgresql://.../storyrail_test pnpm test:e2e`. The suite applies -migrations to that database and refuses to reuse another running development server. - -## Project status and limitations +PostgreSQL integration and browser tests require a disposable database named `storyrail_test` through `STORYRAIL_TEST_DATABASE_URL`. The browser suite also needs Chromium (`pnpm exec playwright install chromium`). Follow [CONTRIBUTING.md](CONTRIBUTING.md) for the maintainer-owned verification sequence and database precautions. -StoryRail is a development-oriented pre-alpha. It has no authentication, migrations are applied deliberately rather than on startup, and some anti-bot publishers remain inaccessible through Firecrawl. +## Documentation -The editorial path is implemented from Source intake through research, assignment, cited drafting, review, and publication, and can run unattended. Autopilot is now a durable record with a reconciliation pass that closes out work whose process disappeared, and a tool call is recorded before it reaches outside rather than after. What remains is delivery: StoryRail has no publishing destination and no way to read an archive other than its own. Free and low-cost models also fail a meaningful share of the time; StoryRail records those failures rather than retrying silently, so an unattended run may stop partway with a durable reason. +- [Product vision](docs/product/vision.md) +- [MVP scope and current limitations](docs/product/mvp.md) +- [Terminology](docs/product/terminology.md) +- [Architecture decisions](docs/architecture/README.md) +- [Generated OpenWiki technical documentation](openwiki/index.md) (optional; refresh locally when needed) -## Technical documentation +## Project status -The generated [OpenWiki documentation](openwiki/index.md) provides deeper, code-grounded coverage of the architecture, domain model, workflows, persistence schema, HTTP API, newsroom UI, and engineering workflow. Human-authored product direction lives under [`docs/product/`](docs/product/), and architectural decisions are indexed under [`docs/architecture/`](docs/architecture/README.md). +StoryRail is under active development. Features and interfaces can change, and the project has not been reviewed or hardened for production use. Authentication is not implemented. Provider availability and output quality vary; failed external work is preserved as a failure record for operator review. See the [MVP scope](docs/product/mvp.md) for current boundaries and deferred work. ## License -StoryRail is licensed under the [GNU Affero General Public License version 3](LICENSE). +StoryRail is licensed under the [GNU Affero General Public License, version 3 only](LICENSE). diff --git a/docs/screenshots/README.md b/docs/screenshots/README.md new file mode 100644 index 0000000..f41f61e --- /dev/null +++ b/docs/screenshots/README.md @@ -0,0 +1,8 @@ +# Screenshot notes + +These screenshots show real StoryRail interface components rendered with a small, fictional in-memory newsroom. Names, headlines, publication details, source excerpts, and editorial states are illustrative and do not represent a real publisher or persisted StoryRail data. + +- `newsroom.png` shows the `NewsroomShell` with a selected in-progress Story, its editorial rail, and current Article. +- `article-provenance.png` shows the `ArticleReader` with the citation support expanded for a cited claim. + +The images were captured from the repository's React components and styles in a temporary local browser harness. The harness is not part of the repository; no application data or external service was used. diff --git a/docs/screenshots/article-provenance.png b/docs/screenshots/article-provenance.png new file mode 100644 index 0000000..50a3467 Binary files /dev/null and b/docs/screenshots/article-provenance.png differ diff --git a/docs/screenshots/newsroom.png b/docs/screenshots/newsroom.png new file mode 100644 index 0000000..f95a574 Binary files /dev/null and b/docs/screenshots/newsroom.png differ