From 7505383441cddce7b56a2b32a07f175c2739ac6d Mon Sep 17 00:00:00 2001 From: Edward Becker Date: Sun, 15 Nov 2026 10:21:35 -0500 Subject: [PATCH 01/40] chore(plan): complete planning for OKF-11 Plan-Finalized: 53f1486f30f003fa1e8d5962f18e5c55c0ec595408ef820b612b659b7ae906bd --- .pan/drafts/OKF-11.md | 642 +++++++++++++ ...-with-visibility-and-redaction.xbrief.json | 893 ++++++++++++++++++ 2 files changed, 1535 insertions(+) create mode 100644 .pan/drafts/OKF-11.md create mode 100644 .pan/specs/2026-11-15-OKF-11-publish-a-bundle-as-a-public-static-site-with-visibility-and-redaction.xbrief.json diff --git a/.pan/drafts/OKF-11.md b/.pan/drafts/OKF-11.md new file mode 100644 index 0000000..26df13d --- /dev/null +++ b/.pan/drafts/OKF-11.md @@ -0,0 +1,642 @@ +# OKF-11: Publish a bundle as a public static site with visibility and redaction + +Issue: https://github.com/eltmon/okf/issues/11 · Planned 2026-11-15 against `main` at `92a21fe`. + +This PRD is the implementation brief. It names every file, function signature, output path, finding code and test. An executor must not re-research or redesign; where a choice exists, this document makes it. + +--- + +## 1. Glossary + +- **Bundle.** A directory tree of UTF-8 Markdown files in Open Knowledge Format (OKF). `references/spec.md` is the vendored spec. +- **Concept.** A `.md` file in the bundle other than the reserved `index.md` and `log.md`. It has YAML frontmatter between `---` lines and a Markdown body. +- **Concept ID.** The concept's bundle-relative path without `.md`, in POSIX form, for example `rules/no-spoilers`. `okf_common.concept_id_for_path` computes it. +- **Index file.** A reserved `index.md` in any bundle directory. `scripts/reindex.py` keeps a generated listing between `` and `` markers; prose outside the markers is hand-written. +- **Visibility.** Whether a concept may appear on the published site: `public` or `private`. +- **Public concept.** A concept whose resolved visibility is `public` (§4.2). Every other concept is private. +- **Published directory.** A bundle directory (root included, written `""`) that contains at least one public concept in itself or in any subdirectory. +- **Schema file.** `schema.md` at the bundle root. OKF v0.2 plans it as a `type: Policy` concept (`.pan/drafts/okf-v0.2-redesign.md` §4.2). For this issue it only carries the bundle visibility default. +- **Redaction profile.** The YAML file `okf-publish.yaml` at the bundle root (or `--profile `). It lists regex rules that rewrite published text. +- **Rule.** One redaction entry: an `id`, a Python `re` pattern and a replacement string. +- **Built-in rules.** The rules in `DEFAULT_RULES` in `scripts/publish_redact.py` (§4.5). They apply even with no profile file. +- **Published text.** Every string the site shows: concept title, description, type, status, tags, source entries, body; and the Markdown of published index pages. +- **Trust tier.** A label derived from OKF v0.2 `verified` frontmatter: `human-reviewed`, `machine-confirmed` or `unverified` (§4.4). +- **Site root.** The `--out` directory. Every site path in this PRD is relative to it. +- **Page-relative href.** An href computed from the linking page's own location, for example `../schema.html` from `rules/no-spoilers.html`. The site never uses `/`-rooted hrefs, so it works under a subpath (GitHub Pages project sites) and on any static host. +- **Output marker.** The file `.okf-publish` that `publish.py` writes at the site root. Its presence proves `publish.py` created the directory and may delete its contents. +- **Finding.** One error line printed by `publish.py`, shaped `ERROR : ` (§4.8). +- **Check mode.** `publish.py --check`: scan without writing; exit 2 if any rule would fire in published text. +- **`PYRUN`.** Shorthand used only in this PRD for `uv run --no-project --python 3.12 --with pyyaml --with pytest`. On the dev machine `python3` is 3.14 without PyYAML, so plain `python3` fails (verified 2026-11-15). Verify commands in the xBRIEF spell the full form. + +## 2. Problem and context + +A bundle can be viewed only locally today (`/okf open`, which delegates to `pan knowledge open`). Some projects want a public, read-only site. The motivating case is a live-stream project whose viewers propose the rules the AI follows; viewers want to see each rule, who suggested it, and what replaced it. + +Publishing the raw bundle is unsafe. Concepts can hold operator-only notes, machine paths, internal incident detail and names of private people. So the site needs per-concept visibility, a deterministic redaction pass, and a gate (`--check`) that fails CI when a rule would fire in a public concept. + +**Current code state (verified 2026-11-15, `main` = `92a21fe`):** + +- Scripts are separate CLIs under `scripts/` sharing `scripts/okf_common.py`. There is no `okf.py` CLI, no `tests/` directory and no pytest. The gate is `scripts/selftest.sh` (bash greps plus inline Python), run by `.github/workflows/ci.yml`. +- OKF-1 (one CLI and pytest), OKF-4 (`schema.md`), OKF-6 (trust fields) and OKF-8 (viewer) are all still open. The issue says publishing "fits after #6". This plan does not wait (decision D1). +- `okf_common.iter_concept_paths` (`scripts/okf_common.py`, line 174, `def iter_concept_paths(bundle_root: Path) -> Iterable[Path]:`) walks every `*.md`, including `.github/` and `.git/`. Publishing needs its own walker (§4.1). +- `okf_common.resolve_concept_link` (line 126) returns `None` for `index.md` and for directory links, so publishing needs its own link classifier (§4.3). +- `okf_common.LINK_PATTERN` (line 17) and `markdown_links` (line 109) skip fenced code. Publishing reuses `markdown_links` for backlinks. +- The house extension-key convention is an `x_` prefix (`x_embed: exclude`, `scripts/embed.py` line 187). The issue names `x_visibility`, so this plan uses `x_visibility` and `x_visibility_default`. + +## 3. Requirements + +### Functional + +- **FR-1.** `python3 scripts/publish.py --bundle --out ` writes one HTML page per public concept at `.html`, an `index.html` for every published directory (root always), `search.html`, `search-index.json`, `assets/style.css`, `assets/search.js` and the output marker `.okf-publish`. It writes nothing else. +- **FR-2.** Visibility resolves per concept from frontmatter `x_visibility`, falling back to the bundle default `x_visibility_default` in root `schema.md` frontmatter. A missing `schema.md` or a missing key means the default is `private`. Any value other than the exact strings `public` or `private` is finding `P_VISIBILITY_INVALID`, and the run exits 2 without writing. +- **FR-3.** A private concept appears nowhere on the site. It has no page. Its index listing line is dropped. It has no search record and appears in no backlink list. A body or source link to it renders as plain text (the link text stays; the issue requires this). +- **FR-4.** Links in published Markdown resolve this way: a link to a public concept becomes a page-relative href to its page, keeping any `#fragment`. A link to a published directory or its `index.md` becomes a page-relative href to that directory's `index.html`. `http:`, `https:` and `mailto:` links stay as written. A `#fragment`-only link stays. Every other target renders as plain text: private concepts, missing concepts, unpublished directories, non-Markdown files and any other scheme. A private target and a missing target render identically. +- **FR-5.** Each concept page shows the concept ID, title, type, status, trust tier, tags, description, rendered body, a "Sources" list and a "Linked from" list (backlinks from public concepts only). +- **FR-6.** The trust tier is `human-reviewed` when any `verified` entry has a `by` string that starts with `human:`. Otherwise it is `machine-confirmed` when there is at least one `verified` entry. Otherwise it is `unverified`. A bare mapping counts as a one-element list. Status is frontmatter `status` or `stable` when absent. +- **FR-7.** Search is client-side. `search-index.json` holds one record per public concept. `search.html` loads `assets/search.js`, which fetches the JSON and lists matches. No server code, no CDN and no third-party script. +- **FR-8.** The redaction profile `okf-publish.yaml` (schema in §4.5) adds regex rules and literal `private_strings`, and can disable built-in rules by id. Built-in rules apply with no profile file. Every rule applies to all published text before rendering, including fenced code. +- **FR-9.** `publish.py --check` scans the published text of public concepts and of published index pages. Each rule that matches gives one `P_REDACTION` finding with the rule id and the location (file and line for bodies and index pages, field name for frontmatter). It never prints the matched text. It exits 2 on any finding and 0 otherwise. It writes no files. +- **FR-10.** A rule that matches a public concept ID or a published directory path gives finding `P_ID_REDACTION` in both modes, and the run exits 2 without writing, because URLs cannot be redacted. +- **FR-11.** `--out` safety. The run refuses (finding `P_OUT_UNSAFE`, exit 2) when `--out` resolves to the bundle root or an ancestor of it, is an existing non-directory, or is a non-empty directory without the output marker. When the marker is present, the run deletes every entry in `--out` and rebuilds, so a page from a concept that has turned private never survives. +- **FR-12.** Bundle walking skips any path with a component that starts with `.` (for example `.git`, `.github`, `.okf`, `.okf-index`) or equals `node_modules`, and skips `--out` when it lies inside the bundle. A concept whose frontmatter does not parse gives finding `P_FRONTMATTER_PARSE`, and the run exits 2 without writing. `log.md` is never published. Non-Markdown files are never copied. +- **FR-13.** Documentation: SKILL.md and README command-table rows and a SKILL.md section for `/okf publish`; a README "Publishing a bundle" section covering visibility, the redaction profile and a sample Cloudflare Pages setup; `references/publish.md`; a `docs/USAGE.md` section. +- **FR-14.** A pytest harness under `tests/` runs in CI next to `scripts/selftest.sh`. + +### Non-functional + +- **NFR-1.** Standard library plus PyYAML only. Python 3.10+ syntax (no `tomllib`, no `typing.Self`, no `ExceptionGroup`). Every new module starts with `#!/usr/bin/env python3`, a one-line docstring and `from __future__ import annotations`. +- **NFR-2.** Deterministic output: two runs on the same bundle produce byte-identical files. No timestamps, sorted iteration everywhere, `json.dumps(..., sort_keys=True, ensure_ascii=False, separators=(",", ":"))`. +- **NFR-3.** No existing script changes except the `scripts/selftest.sh` command loop (§6, WI-9). `okf_common.py`, `validate.py`, `reindex.py` and the template copies under `templates/repo/.okf/scripts/` stay byte-identical. No Overdeck imports (selftest greps for them). +- **NFR-4.** HTML safety. All text from the bundle is HTML-escaped. Raw HTML in Markdown renders as escaped text. No `on*=` attributes, no `style=` attributes, no `javascript:` or `data:` hrefs. The only `` on `search.html`. `search.js` writes results with `textContent` and `setAttribute("href", …)`, never `innerHTML`. +- **NFR-5.** Findings never echo matched secret text, because CI logs of public repos are public. + +## 4. Design (decisions made) + +### 4.1 Module layout + +All new code is flat in `scripts/`, matching the existing `from okf_common import …` style. Files: + +| File | Role | +| --- | --- | +| `scripts/publish_markdown.py` | Stdlib Markdown-subset → escaped HTML renderer. | +| `scripts/publish_redact.py` | Built-in rules, profile loading, `redact`, `scan`. | +| `scripts/publish_model.py` | Bundle walk, visibility, trust tier, sources, link classification, backlinks. | +| `scripts/publish_render.py` | Page shell, concept pages, index pages, index filtering, `assets/style.css`. | +| `scripts/publish_search.py` | `search-index.json`, `search.html`, `assets/search.js`. | +| `scripts/publish.py` | CLI: argparse, profile, `--out` safety, redaction application, `--check`, exit codes. | +| `scripts/publish_assets/style.css` | Site stylesheet, copied verbatim to `assets/style.css`. | +| `scripts/publish_assets/search.js` | Search script, copied verbatim to `assets/search.js`. | +| `templates/okf-publish.yaml` | Starter profile users copy to their bundle root. | + +Modules locate assets with `Path(__file__).resolve().parent / "publish_assets"`. + +### 4.2 Visibility (`publish_model.py`) + +```python +VISIBILITY_KEY = "x_visibility" +DEFAULT_KEY = "x_visibility_default" +SCHEMA_FILE = "schema.md" +VISIBILITY_VALUES = ("public", "private") +``` + +1. Bundle default: if `/schema.md` does not exist → `private`, reason `"no schema.md at bundle root"`. If it parses and lacks `x_visibility_default` → `private`, reason `"schema.md has no x_visibility_default"`. If the value is `public` or `private` → that value, reason `"schema.md x_visibility_default: "`. Any other value → finding `P_VISIBILITY_INVALID schema.md: x_visibility_default must be public or private`, default `private`. +2. Per concept: `x_visibility` absent → bundle default. `public`/`private` → that value. Anything else (including `Public`, `true`, `yes`) → finding `P_VISIBILITY_INVALID : x_visibility must be public or private`, concept treated as private. +3. `schema.md` is an ordinary concept for publishing: its own `x_visibility` or the default decides whether it gets a page. + +Fail closed: no input ever resolves to public by accident. When zero concepts are public, `publish.py` prints the default reason (§4.8). + +### 4.3 Link classification (`publish_model.py`) + +```python +@dataclass(frozen=True) +class LinkTarget: + kind: str # "concept" | "directory" | "external" | "fragment" | "other" + value: str # concept ID, directory path ("" = root), or the raw href + fragment: str # text after "#", without the "#"; "" when none + +def classify_link(source_dir: str, target: str) -> LinkTarget: ... +``` + +`source_dir` is the bundle-relative POSIX directory of the page holding the link (`""` for the root). Rules, in order: + +1. Split `target` on the first `#` into `path` and `fragment`. +2. `path == ""` → `fragment` kind, value `"#" + fragment`. +3. `path` lowercased starts with `http://`, `https://` or `mailto:` → `external`, value = `target` unchanged. +4. `path` contains `:` → `other` (any other scheme, including `javascript:`, `data:`, `tel:`, `repo:`). +5. Resolve the path: a leading `/` makes it bundle-root relative; otherwise join it to `source_dir`. Normalise `.` and `..`. A `..` that would climb above the root → `other`. +6. Normalised path ends with `/`, or is empty after normalising → `directory`, value = path without trailing `/`. +7. Last segment is `index.md` → `directory`, value = its parent (`""` for root). +8. Last segment is `log.md` → `other`. +9. Ends with `.md` → `concept`, value = path without `.md`. +10. Anything else → `other`. + +Whether a `concept` or `directory` target is *published* is decided by the caller (`model.is_public(id)`, `dir in model.public_dirs`). + +### 4.4 Model types (`publish_model.py`) + +```python +@dataclass(frozen=True) +class Source: + title: str # "" when absent + resource: str # required; entries without it are skipped + author: str # "" when absent + +@dataclass(frozen=True) +class PublishConcept: + concept_id: str + path: str # bundle-relative POSIX path, e.g. "rules/no-spoilers.md" + directory: str # "" for root + title: str # frontmatter title, else last ID segment + description: str + type: str + tags: tuple[str, ...] + status: str # default "stable" + trust_tier: str + sources: tuple[Source, ...] + body: str + body_start_line: int # 1-based file line of the first body line + visibility: str # "public" | "private" + links: tuple[str, ...] # raw link targets from markdown_links(body), in order + +@dataclass(frozen=True) +class Finding: + code: str + location: str # "rules/a.md", "rules/a.md:14", "index.md:5" + message: str + def line(self) -> str: + return f"ERROR {self.code} {self.location}: {self.message}" + +@dataclass +class PublishModel: + bundle_root: Path + default_visibility: str + default_reason: str + concepts: dict[str, PublishConcept] # every parsed concept, keyed by ID + index_files: dict[str, str] # directory -> index.md text with any frontmatter removed + def public_ids(self) -> list[str]: ... # sorted + def is_public(self, concept_id: str) -> bool: ... + def public_dirs(self) -> set[str]: ... # "" always included + def backlinks(self, concept_id: str) -> list[str]: ... # sorted public IDs that link to it + +def walk_markdown(bundle_root: Path, exclude: Path | None = None) -> list[Path]: ... +def trust_tier(frontmatter: dict) -> str: ... +def build_model(bundle_root: Path, exclude: Path | None = None) -> tuple[PublishModel, list[Finding]]: ... +``` + +Normalisation rules: + +- `tags`: a list → `str()` of each item; a scalar → one-element tuple; absent → `()`. +- `sources`: a list; each item is either a mapping (use `resource`, `title`, `author`, all `str()`; skip an item without `resource`) or a string (becomes `Source("", s, "")`). A non-list value is ignored. +- `verified` for `trust_tier`: `None`/absent → `[]`; a mapping → `[mapping]`; a list → mapping items only. +- `body_start_line`: index (1-based) of the closing `---` line plus 1; plus 1 more when the following line is empty (because `okf_common.split_frontmatter` drops one leading blank line). +- `backlinks(x)`: IDs `s` with `is_public(s)`, `s != x`, and any `classify_link(s.directory, link)` with kind `concept` and value `x`. Sorted, unique. +- `public_dirs()`: for every public concept, its directory and every ancestor, plus `""`. +- `index_files`: for each `index.md` the walker finds, keyed by its directory. Strip frontmatter when the text starts with `---\n` (the root may carry `okf_version`). +- A concept that fails `okf_common.parse_concept` gives `P_FRONTMATTER_PARSE : ` and is left out of `concepts`. +- `walk_markdown` returns sorted `*.md` paths. It skips any path whose bundle-relative parts include a component starting with `.` or equal to `node_modules`, and skips paths under `exclude` when `exclude` is inside the bundle. It does not skip `log.md`; `build_model` ignores `log.md` itself. + +### 4.5 Redaction (`publish_redact.py`) + +Profile schema (`okf-publish.yaml`): + +```yaml +okf_publish_version: "0.1" # required, must be the string "0.1" +disable_default_rules: [] # optional; ids from DEFAULT_RULES +rules: # optional; applied after built-in rules + - id: stream-box-host # required, unique, matches ^[a-z0-9][a-z0-9-]*$ + pattern: '\bstream-box\.lan\b' # required, Python re syntax; inline flags such as (?i) allowed + replace: '[internal-host]' # optional, default "[redacted]" +private_strings: # optional; literal strings, matched case-insensitively + - Alice Example +``` + +Built-in rules (exact patterns, in this order): + +```python +DEFAULT_RULES: tuple[tuple[str, str, str], ...] = ( + ("home-path-unix", r"/(?:home|Users)/[^/\s'\"`)\]>]+", "~"), + ("home-path-windows", r"[A-Za-z]:\\Users\\[^\\\s'\"`)\]>]+", "~"), + ("ipv4", r"\b(?:25[0-5]|2[0-4]\d|1?\d?\d)(?:\.(?:25[0-5]|2[0-4]\d|1?\d?\d)){3}\b", "[ip]"), + ("github-token", r"\bgh[pousr]_[A-Za-z0-9]{36,}\b", "[token]"), + ("openai-style-key", r"\bsk-[A-Za-z0-9_-]{20,}", "[token]"), + ("aws-access-key", r"\bAKIA[0-9A-Z]{16}\b", "[token]"), + ("slack-token", r"\bxox[abprs]-[A-Za-z0-9-]{10,}", "[token]"), + ("private-key-block", r"-----BEGIN [A-Z ]*PRIVATE KEY-----", "[private-key]"), +) +``` + +`private_strings` entry *n* (1-based) becomes rule `private-string-` with pattern `re.escape(s)`, flag `re.IGNORECASE`, replacement `[redacted]`. Order of application: enabled built-in rules, then `rules`, then `private_strings`. + +```python +@dataclass(frozen=True) +class Rule: + id: str + pattern: re.Pattern[str] + replace: str + origin: str # "default" | "profile" | "private_strings" + +@dataclass(frozen=True) +class Profile: + rules: tuple[Rule, ...] + path: Path | None # None when built-in rules only + +def default_profile() -> Profile: ... +def load_profile(bundle_root: Path, profile_path: Path | None = None) -> tuple[Profile | None, list[str]]: ... +def redact(text: str, profile: Profile) -> tuple[str, dict[str, int]]: ... +def scan(text: str, profile: Profile) -> list[tuple[str, int]]: ... +``` + +- `load_profile`: with `profile_path` given, that file must exist. Without it, use `/okf-publish.yaml` when it exists, else `default_profile()`. Return `(None, errors)` on any error: unreadable YAML, top level not a mapping, unknown top-level key, `okf_publish_version` not `"0.1"`, unknown id in `disable_default_rules`, rule without `id` or `pattern`, invalid id, duplicate id (including a clash with a built-in id), a pattern that fails `re.compile` (message names the rule id and the `re.error`), an empty private string, or a non-string value. Each error string is one sentence naming the field. +- `redact`: apply each rule with `pattern.sub(replace, text)` in order. Return the new text and `{rule_id: substitutions}` for rules that fired. `replace` is passed as a literal (escape backslashes with `replace.replace("\\", "\\\\")` before `sub`, so `\1` is not a group reference). +- `scan`: for each rule, each match in the *original* text gives `(rule.id, line)` where `line = text.count("\n", 0, match.start()) + 1`. Return sorted unique pairs. +- `templates/okf-publish.yaml` contains `okf_publish_version: "0.1"`, `disable_default_rules: []`, commented examples of one rule and one private string, and a comment listing every built-in rule id. A test asserts the commented id list equals `[r[0] for r in DEFAULT_RULES]`. + +False positives: `ipv4` also matches `127.0.0.1` and four-part version numbers. A bundle that hits this either rewrites the text or adds `ipv4` to `disable_default_rules`. This is documented, and there is no per-match allow list (non-goal). + +### 4.6 Rendering (`publish_markdown.py`, `publish_render.py`) + +```python +LinkResolver = Callable[[str], Optional[str]] +def render_markdown(text: str, resolve_link: LinkResolver) -> str: ... +``` + +`resolve_link(raw_target)` returns an href or `None`; `None` renders only the link text. The renderer re-checks every href it emits: an href whose part before the first `/`, `?` or `#` contains `:` must start with `http:`, `https:` or `mailto:` (case-insensitive), otherwise the link renders as text. + +Supported Markdown (everything else is escaped text inside a paragraph): + +| Construct | Output | +| --- | --- | +| ATX heading `#`–`######` | `…`; slug = lowercase, runs of non-`[a-z0-9]` → `-`, trimmed; repeats get `-2`, `-3` | +| Paragraph (consecutive non-blank lines) | `

…

`, lines joined with `\n` | +| Fenced code ```` ``` ```` or `~~~` with optional info word | `
escaped
`; class only when the word matches `^[A-Za-z0-9_+-]+$` | +| Blockquote `> ` | `
` + recursive render | +| Unordered list `-`, `*`, `+` and ordered `1.` | `