diff --git a/docs/installation.md b/docs/installation.md index 0b662e8..3e5414b 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -39,6 +39,7 @@ directory: mkdir -p "${CODEX_HOME:-$HOME/.codex}/skills" cp -R skills/dtm "${CODEX_HOME:-$HOME/.codex}/skills/" cp -R skills/voice "${CODEX_HOME:-$HOME/.codex}/skills/" +cp -R skills/markdown-to-pdf "${CODEX_HOME:-$HOME/.codex}/skills/" ``` Start a thread in the installed vault and invoke `$dtm`. That thread remains in @@ -49,6 +50,28 @@ Invoke `$voice` after representative writing has reached `ready/` or `published/`, or after representative internal documents have reached `documents/final/`. It updates the private voice pack without analysing drafts. +Invoke `$markdown-to-pdf` when you want a shareable PDF from a Markdown note or +managed document, including Mermaid diagrams when present. + +## Optional PDF deliverable toolchain + +If you want managed Markdown-to-PDF generation with Mermaid support, install: + +```sh +brew install pandoc typst mermaid-cli +npx puppeteer browsers install chrome-headless-shell +``` + +The PDF command is: + +```sh +python3 tools/document_deliverables.py pdf documents/drafts/example.md +``` + +Mermaid rendering launches a headless browser. Inside sandboxed agent +environments, that step may require unsandboxed execution approval even after +the dependencies are installed. + ## Validate ```sh diff --git a/framework/AGENTS.md b/framework/AGENTS.md index 712e4ff..0840d8f 100644 --- a/framework/AGENTS.md +++ b/framework/AGENTS.md @@ -92,6 +92,40 @@ compound across sessions. same change. - Never present inference as sourced fact. +### `projects/` — DTM-managed durable project workspace + +- `projects/index.md` is the curated catalogue of durable project pages. Keep + every managed project listed exactly once under its current lifecycle state + with a one-line description concrete enough to route future work without + opening each file. +- Project pages hold stable objectives, current state, next actions, decisions, + and major activity for work that spans days or needs durable coordination. +- Valid project statuses are `active`, `on-hold`, and `completed`. +- `completed` projects are retained for reference; completion does not imply + deletion or automatic relocation. +- Keep stable lowercase kebab-case filenames and update `projects/index.md`, + the relevant Daily Note references, and the root `log.md` when the change is + materially significant. + +### `work/` — DTM-managed working-note workspace + +- `work/index.md` is the curated catalogue of managed working notes. Keep every + managed top-level `work/*.md` note listed exactly once under its current + lifecycle state with a one-line description concrete enough to avoid loading + each note just to understand what still matters. +- This area is for active or retained operational working material such as + meeting prep, research packs, temporary plans, scratch analyses, examples, + and intermediate outputs that do not yet belong in `wiki/`, `projects/`, + `documents/`, or `writing/`. +- Valid working-note statuses are `current`, `parked`, and `reference`. +- `reference` means the note is no longer an active work item but is retained + because its reasoning, examples, or context still have value. +- Keep stable lowercase kebab-case filenames for managed working notes and + update `work/index.md` plus relevant Daily Note references when lifecycle + state or scope changes materially. +- `work/wiki-open-questions.md` remains the DTM-managed operational queue for + unresolved wiki questions and follows the same workspace conventions. + ### `documents/` — collaborative internal documents workspace - The DTM owns collaborative drafting, revision support, and lifecycle @@ -423,8 +457,11 @@ Summary of what changed, with links to affected pages. Allowed operation labels are `setup`, `ingest`, `query`, `lint`, `schema`, and `dtm`. DTM log entries are reserved for significant operational changes such as -project milestones, durable decisions, or created/updated wiki artefacts; routine -daily rollover does not need a wiki log entry. +project milestones, durable decisions, created or materially updated artefacts, +or unattended automated actions that created, promoted, published, or +materially changed something in the system. Routine daily rollover does not +need a wiki log entry unless it produced a material unattended change beyond +ordinary note lifecycle housekeeping. Mention unresolved conflicts, gaps, or follow-ups in the same entry. ## Completion standard diff --git a/framework/DTM.md b/framework/DTM.md index 13f3afc..84fa202 100644 --- a/framework/DTM.md +++ b/framework/DTM.md @@ -49,6 +49,10 @@ When work is delegated to another thread: - The DTM may create or update wiki pages when durable knowledge emerges. Apply the wiki schema and citation rules in `AGENTS.md`, update `wiki/index.md`, and log the material change. +- When DTM-scoped work creates or materially extends a wiki artefact, capture it + in both today's Daily Note and the root `log.md`. Daily Note activity preserves + day-level operational continuity; the root log preserves system-level + traceability for the wiki change itself. - Do not treat conversational claims as externally verified facts. Attribute personal decisions and observations to the user/context where useful. - The Knowledge Agent owns systematic source ingestion. The DTM owns Daily @@ -63,7 +67,8 @@ When work is delegated to another thread: ## DTM workspace -- `daily/YYYY-MM-DD.md` — canonical Daily Notes. +- `daily/YYYY-MM-DD.md` — active Daily Notes for the current 14-day retro period. +- `daily/archive/YYYY-MM-DD.md` — archived Daily Notes retained for reference once they age out of the active window. - `projects/` — durable project plans, status, milestones, and next actions. - `work/` — working documents, drafts, scratch analyses, and deliverables that are not yet durable wiki knowledge. @@ -77,6 +82,30 @@ Prefer links rooted at the vault, for example `[[projects/example-project|Example project]]`. A project page should hold stable context and current state; its day-specific activity belongs in the Daily Note. +`projects/index.md` and `work/index.md` are the curated navigation layers for +those workspaces. Keep every managed note listed exactly once under its current +lifecycle state with a concrete one-line description. + +Project statuses are: + +- `active` — live delivery or active coordination work. +- `on-hold` — intentionally paused but expected to resume. +- `completed` — retained for reference after active delivery ends. + +Working-note statuses are: + +- `current` — active work material still supporting a live task or integration step. +- `parked` — inactive for now but likely to resume. +- `reference` — no longer active work, but retained because the note still has useful context or reasoning. + +Keep project filenames and managed working-note filenames in stable lowercase +kebab-case. Use `templates/project.md` for projects and +`templates/working-note.md` for new managed working notes. + +Run `python3 tools/workspaces.py write` followed by +`python3 tools/workspaces.py lint` after structural or lifecycle-status changes +in `projects/` or managed top-level `work/*.md` notes. + For Daily Notes, project pages, work notes, wiki edits performed under DTM authority, and internal documents, keep ordinary prose paragraphs on one physical line and rely on Obsidian for visual wrapping. Use new lines only when @@ -160,7 +189,16 @@ completion. Significant DTM actions also receive an append-only `log.md` entry using the `dtm` operation label. Significant means a durable decision, project -milestone, or created/updated wiki artefact—not ordinary task edits or rollover. +milestone, created or materially updated artefact, or an unattended automated +action that created, promoted, published, or materially changed something in +the system. Ordinary task edits, conversational clarification, and routine day +rollover do not need a vault log entry unless they were performed unattended as +part of an automation whose outcome materially changed the system. + +If the significant action is specifically the creation or material extension of +a wiki artefact, do not rely on Daily Note capture alone. Record the outcome in +today's Daily Note and also append a root-log entry that makes the wiki change +traceable at system level. ## Tasks @@ -253,6 +291,11 @@ The scheduled lifecycle runs at 00:01 in the user's local timezone. It should: outcomes, meaningful developments, and unfinished threads. 4. Refine today's `Focus` into a short ranked recommendation based on carried work, active projects, questions, deadlines, and outcomes. + `Focus` is for actionable priorities only. Do not place blocked items there + when no material progress is possible under the user's control. Track those + items under a separate `Blockers` section instead. A blocked item may appear + in `Focus` only if there is a genuine actionable step the user can take + today beyond merely waiting or monitoring. 5. Keep the mechanically carried personal/professional tasks and open questions; correct duplicates or categorisation errors if necessary. 6. Confirm scheduled and recurring instances are relevant for the date. diff --git a/framework/automation-definitions/dtm-daily-rollover.json b/framework/automation-definitions/dtm-daily-rollover.json index a57fb56..e1c6ce9 100644 --- a/framework/automation-definitions/dtm-daily-rollover.json +++ b/framework/automation-definitions/dtm-daily-rollover.json @@ -4,5 +4,5 @@ "schedule": "00:01 daily in the user's local timezone", "execution_environment": "local", "workspace": "{{VAULT_PATH}}", - "prompt": "Read AGENTS.md and DTM.md, run python3 tools/dtm.py rollover, finalise the previous note from recorded evidence, populate today's Previous Day and Focus, preserve valid carried tasks and questions, instantiate recurrence, link active projects, record DTM activity, capture only concise bullet-point durable decisions in the Decisions section, log only significant durable changes, and run DTM and wiki lint checks." + "prompt": "Read AGENTS.md and DTM.md, run python3 tools/dtm.py rollover, finalise the previous note from recorded evidence, populate today's Previous Day and actionable-only Focus, keep genuine blockers in a separate Blockers section, preserve valid carried tasks and questions, instantiate recurrence, link active projects, record DTM activity, capture only concise bullet-point durable decisions in the Decisions section, log only significant durable changes, and run DTM and wiki lint checks." } diff --git a/framework/defaults/documents/README.md b/framework/defaults/documents/README.md index 4867133..090086b 100644 --- a/framework/defaults/documents/README.md +++ b/framework/defaults/documents/README.md @@ -93,7 +93,27 @@ python3 tools/document_deliverables.py docx documents/drafts/example.md ## Deliverable generation -Use `tools/document_deliverables.py` when a managed Markdown document needs a shareable `.docx` copy under `documents/deliverables//`. +Use `tools/document_deliverables.py` when a managed Markdown document needs a shareable deliverable under `documents/deliverables//`. + +Word copy: + +```sh +python3 tools/document_deliverables.py docx documents/drafts/example.md +``` + +PDF copy, including Mermaid diagrams when present: + +```sh +python3 tools/document_deliverables.py pdf documents/drafts/example.md +``` + +To omit an author-facing section from the reader copy, add `--exclude-heading`, for example: + +```sh +python3 tools/document_deliverables.py pdf documents/drafts/example.md --exclude-heading Notes +``` + +The PDF path depends on `pandoc`, `typst`, and `mermaid-cli`. Mermaid rendering launches a headless browser, so agents running inside a sandbox may need unsandboxed execution approval when a source contains Mermaid diagrams. The current supported path is: diff --git a/framework/defaults/dtm/recurring-tasks.json b/framework/defaults/dtm/recurring-tasks.json index 6f5ca30..6927091 100644 --- a/framework/defaults/dtm/recurring-tasks.json +++ b/framework/defaults/dtm/recurring-tasks.json @@ -14,7 +14,7 @@ { "id": "monday-review-diary-post", "task": "Review the generated weekly diary draft", - "area": "professional", + "area": "schedule", "enabled": true, "schedule": { "frequency": "weekly", @@ -24,7 +24,7 @@ { "id": "tuesday-review-wiki-blog-post", "task": "Review the generated weekly wiki blog draft", - "area": "professional", + "area": "schedule", "enabled": true, "schedule": { "frequency": "weekly", diff --git a/framework/publication-manifest.json b/framework/publication-manifest.json index 8889a92..3feb21b 100644 --- a/framework/publication-manifest.json +++ b/framework/publication-manifest.json @@ -9,6 +9,7 @@ {"source": "tools/wiki.py", "destination": "framework/tools/wiki.py"}, {"source": "tools/dtm.py", "destination": "framework/tools/dtm.py"}, {"source": "tools/documents.py", "destination": "framework/tools/documents.py"}, + {"source": "tools/document_deliverables.py", "destination": "framework/tools/document_deliverables.py"}, {"source": "tools/writing.py", "destination": "framework/tools/writing.py"}, {"source": "tools/export_framework.py", "destination": "framework/tools/export_framework.py"}, {"source": "tools/publish_framework.py", "destination": "framework/tools/publish_framework.py"}, diff --git a/framework/skills/markdown-to-pdf/SKILL.md b/framework/skills/markdown-to-pdf/SKILL.md new file mode 100644 index 0000000..9ecd5e7 --- /dev/null +++ b/framework/skills/markdown-to-pdf/SKILL.md @@ -0,0 +1,42 @@ +--- +name: markdown-to-pdf +description: Generate a shareable PDF from a SecondBrain Markdown note or managed document, including Mermaid diagrams when present. Use when the user asks for a PDF deliverable, a shareable PDF copy, or a Markdown-to-PDF export inside the vault. +--- + +# Markdown to PDF + +Create a PDF artefact from a Markdown source using the managed SecondBrain deliverables tool. + +## Use the managed command + +1. Confirm the working directory is a SecondBrain vault containing `AGENTS.md` and `tools/document_deliverables.py`. +2. If the source is a managed document under `documents/drafts/` or `documents/final/`, prefer the default output location under `documents/deliverables//`. +3. Run: + +```sh +python3 tools/document_deliverables.py pdf +``` + +4. For a non-document Markdown note, provide an explicit output path: + +```sh +python3 tools/document_deliverables.py pdf --output /path/to/output.pdf +``` + +5. If the note contains author-facing sections that should not be included in the reader copy, omit them with one or more `--exclude-heading` arguments: + +```sh +python3 tools/document_deliverables.py pdf --exclude-heading Notes +``` + +## Mermaid and sandboxing + +- The PDF workflow supports Mermaid diagrams automatically. +- Mermaid rendering depends on `pandoc`, `typst`, `mermaid-cli`, and a working headless browser runtime. +- If the source contains Mermaid and browser launch fails inside a sandboxed agent environment, rerun the same command with unsandboxed execution approval rather than replacing the diagram workflow by hand. + +## Validation + +- Confirm the command prints the output path. +- Inspect the PDF visually before treating it as ready to share. +- For managed document deliverables created under DTM authority, update the Daily Note and relevant document index or log entries when the artefact materially changes the workspace state. diff --git a/framework/skills/markdown-to-pdf/agents/openai.yaml b/framework/skills/markdown-to-pdf/agents/openai.yaml new file mode 100644 index 0000000..fa0f5ff --- /dev/null +++ b/framework/skills/markdown-to-pdf/agents/openai.yaml @@ -0,0 +1,6 @@ +interface: + display_name: "Markdown to PDF" + short_description: "Generate shareable PDFs from vault Markdown" + default_prompt: "Use $markdown-to-pdf to create a shareable PDF from this Markdown note." +policy: + allow_implicit_invocation: false diff --git a/framework/templates/daily-note.md b/framework/templates/daily-note.md index 2eb6c4a..5d1e63b 100644 --- a/framework/templates/daily-note.md +++ b/framework/templates/daily-note.md @@ -16,6 +16,8 @@ tags: ## Focus +## Blockers + ## Personal To-Do ## Professional To-Do diff --git a/framework/templates/project.md b/framework/templates/project.md index beb9dc5..a2d59ce 100644 --- a/framework/templates/project.md +++ b/framework/templates/project.md @@ -34,3 +34,5 @@ Relevant decisions and links back to the supporting Daily Note or working contex ## Activity - YYYY-MM-DD — Material change with a link to the Daily Note. + + diff --git a/framework/templates/working-note.md b/framework/templates/working-note.md new file mode 100644 index 0000000..121b6ce --- /dev/null +++ b/framework/templates/working-note.md @@ -0,0 +1,25 @@ +--- +title: Working note title +type: working-note +status: current +created: YYYY-MM-DD +updated: YYYY-MM-DD +tags: + - work +--- + +# Working note title + +Concise summary of what this note is for and why it exists. + +## Context + +Relevant background. + +## Current content + +Working material, drafts, analysis, or interim outputs. + +## Next use + +How this note is expected to be used next, integrated, or retired to reference status. diff --git a/framework/tools/document_deliverables.py b/framework/tools/document_deliverables.py new file mode 100644 index 0000000..e581563 --- /dev/null +++ b/framework/tools/document_deliverables.py @@ -0,0 +1,600 @@ +#!/usr/bin/env python3 +"""Build shareable document deliverables from managed Markdown sources.""" + +from __future__ import annotations + +import argparse +import json +import os +import re +import subprocess +import sys +import tempfile +from dataclasses import dataclass +from pathlib import Path +from typing import Iterable + +def preferred_runtime_python() -> Path: + configured = os.environ.get("CODEX_PRIMARY_RUNTIME_PYTHON") + if configured: + return Path(configured).expanduser() + return Path.home() / ".cache" / "codex-runtimes" / "codex-primary-runtime" / "dependencies" / "python" / "bin" / "python3" + + +def preferred_runtime_site_packages() -> Path: + configured = os.environ.get("CODEX_PRIMARY_RUNTIME_SITE_PACKAGES") + if configured: + return Path(configured).expanduser() + return Path.home() / ".cache" / "codex-runtimes" / "codex-primary-runtime" / "dependencies" / "python" / "lib" / "python3.12" / "site-packages" + + +RUNTIME_PYTHON = preferred_runtime_python() +RUNTIME_SITE_PACKAGES = preferred_runtime_site_packages() + +if RUNTIME_PYTHON.exists() and Path(sys.executable).resolve() != RUNTIME_PYTHON.resolve(): + os.execv(str(RUNTIME_PYTHON), [str(RUNTIME_PYTHON), __file__, *sys.argv[1:]]) + +if RUNTIME_SITE_PACKAGES.exists(): + sys.path.insert(0, str(RUNTIME_SITE_PACKAGES)) + +from docx import Document # type: ignore +from docx.enum.style import WD_STYLE_TYPE +from docx.enum.text import WD_ALIGN_PARAGRAPH, WD_BREAK +from docx.oxml import OxmlElement # type: ignore +from docx.oxml.ns import qn # type: ignore +from docx.shared import Inches, Pt, RGBColor # type: ignore + + +ROOT = Path(__file__).resolve().parent.parent +DOCUMENTS = ROOT / "documents" +DELIVERABLES = DOCUMENTS / "deliverables" + +PAGE_WIDTH_IN = 8.5 +PAGE_HEIGHT_IN = 11 +MARGIN_IN = 1.0 +CONTENT_WIDTH_IN = PAGE_WIDTH_IN - (MARGIN_IN * 2) +IMAGE_SUFFIXES = {".png", ".jpg", ".jpeg", ".gif", ".svg", ".webp"} +MERMAID_BLOCK_RE = re.compile(r"```mermaid\s*\n(.*?)\n```", re.S) +OBSIDIAN_EMBED_RE = re.compile(r"!\[\[([^\]]+)\]\]") +OBSIDIAN_LINK_RE = re.compile(r"(? tuple[dict[str, str], str]: + if not text.startswith("---\n"): + return {}, text + end = text.find("\n---\n", 4) + if end < 0: + return {}, text + + raw = text[4:end] + body = text[end + 5 :] + meta: dict[str, str] = {} + key: str | None = None + for line in raw.splitlines(): + if re.match(r"^\s+-\s+", line) and key: + meta[key] = (meta.get(key, "") + "\n" + re.sub(r"^\s+-\s+", "", line)).strip() + continue + match = re.match(r"^([A-Za-z_][\w-]*):(?:\s*(.*))?$", line) + if not match: + key = None + continue + key, value = match.groups() + meta[key] = (value or "").strip().strip("\"'") + return meta, body + + +def publish_body(text: str) -> str: + if "" in text and "" in text: + return text.split("", 1)[1].split("", 1)[0].strip() + return text + + +def split_blocks(text: str) -> list[str]: + lines = [line.rstrip() for line in text.splitlines()] + blocks: list[str] = [] + current: list[str] = [] + for line in lines: + if not line.strip(): + if current: + blocks.append("\n".join(current).strip()) + current = [] + continue + current.append(line) + if current: + blocks.append("\n".join(current).strip()) + return blocks + + +def parse_sections(text: str) -> list[Section]: + sections: list[Section] = [] + current = Section(level=0, title="", blocks=[]) + for block in split_blocks(text): + heading = re.match(r"^(#{1,6})\s+(.*)$", block) + if heading: + if current.title or current.blocks: + sections.append(current) + current = Section(level=len(heading.group(1)), title=heading.group(2).strip(), blocks=[]) + else: + current.blocks.append(block) + if current.title or current.blocks: + sections.append(current) + return sections + + +def clean_inline(text: str) -> str: + text = re.sub(r"\[\[([^\]|#]+)(?:#[^\]|]+)?\|([^\]]+)\]\]", r"\2", text) + text = re.sub(r"\[\[([^\]|#]+)(?:#[^\]|]+)?\]\]", lambda m: Path(m.group(1)).stem.replace("-", " "), text) + text = re.sub(r"\[([^\]]+)\]\(([^)]+)\)", r"\1", text) + text = re.sub(r"`([^`]+)`", r"\1", text) + text = re.sub(r"\*\*([^*]+)\*\*", r"\1", text) + text = re.sub(r"\*([^*]+)\*", r"\1", text) + return text.strip() + + +def default_output_for(source: Path, suffix: str) -> Path: + slug = source.stem + if source.is_relative_to(DOCUMENTS / "drafts") or source.is_relative_to(DOCUMENTS / "final"): + return DELIVERABLES / slug / f"{slug}{suffix}" + raise ValueError( + "Default output is only available for managed documents under documents/drafts or documents/final; use --output for other Markdown notes." + ) + + +def display_text_for_target(target: str) -> str: + return Path(target).stem.replace("-", " ") + + +def resolve_embed(raw_target: str, source: Path) -> Path | None: + target = raw_target.split("|", 1)[0].split("#", 1)[0].strip() + if not target: + return None + candidate = (source.parent / target).resolve() + if candidate.exists(): + return candidate + root_candidate = (ROOT / target).resolve() + if root_candidate.exists(): + return root_candidate + return None + + +def replace_obsidian_embeds(text: str, source: Path) -> str: + def repl(match: re.Match[str]) -> str: + resolved = resolve_embed(match.group(1), source) + if resolved is None: + return display_text_for_target(match.group(1)) + if resolved.suffix.lower() in IMAGE_SUFFIXES: + return f"![](<{resolved.as_posix()}>)" + return resolved.name + + return OBSIDIAN_EMBED_RE.sub(repl, text) + + +def replace_obsidian_links(text: str) -> str: + def repl(match: re.Match[str]) -> str: + target, alias = match.groups() + label = alias if alias else display_text_for_target(target) + if target.startswith("#"): + if target.startswith("#^ref-"): + return f"[{label}](#references)" + anchor = target[1:] + if anchor.startswith("^"): + anchor = anchor[1:] + return f"[{label}](#{anchor})" + return label + + return OBSIDIAN_LINK_RE.sub(repl, text) + + +def replace_obsidian_block_ids(text: str) -> str: + return OBSIDIAN_BLOCK_ID_RE.sub("", text) + + +def render_mermaid_diagrams(text: str, working: Path) -> str: + counter = 0 + cached_browsers = sorted( + Path.home().glob( + ".cache/puppeteer/chrome-headless-shell/*/chrome-headless-shell-mac-arm64/chrome-headless-shell" + ) + ) + browser = cached_browsers[-1] if cached_browsers else Path("/Applications/Chromium.app/Contents/MacOS/Chromium") + puppeteer_config: Path | None = None + if browser.exists(): + puppeteer_config = working / "puppeteer-config.json" + puppeteer_config.write_text( + json.dumps( + { + "executablePath": str(browser), + "args": ["--no-sandbox"], + } + ), + encoding="utf-8", + ) + + def repl(match: re.Match[str]) -> str: + nonlocal counter + counter += 1 + diagram = match.group(1).strip() + "\n" + source_path = working / f"diagram-{counter:02d}.mmd" + image_path = working / f"diagram-{counter:02d}.png" + source_path.write_text(diagram, encoding="utf-8") + command = [ + "mmdc", + "--input", + str(source_path), + "--output", + str(image_path), + "--theme", + "neutral", + "--backgroundColor", + "transparent", + "--scale", + "2", + ] + if puppeteer_config is not None: + command.extend(["--puppeteerConfigFile", str(puppeteer_config)]) + try: + subprocess.run(command, check=True, capture_output=True, text=True) + except FileNotFoundError as exc: + raise RuntimeError("mmdc was not found on PATH. Install mermaid-cli before building PDF deliverables.") from exc + except subprocess.CalledProcessError as exc: + stderr = (exc.stderr or "").strip() + raise RuntimeError(f"Mermaid rendering failed for diagram {counter}: {stderr or exc}") from exc + return f"![Mermaid diagram {counter}](<{image_path.as_posix()}>)" + + return MERMAID_BLOCK_RE.sub(repl, text) + + +def strip_named_sections(text: str, headings: list[str]) -> str: + if not headings: + return text + heading_set = {heading.strip().casefold() for heading in headings if heading.strip()} + if not heading_set: + return text + + lines = text.splitlines() + result: list[str] = [] + skip_level: int | None = None + + for line in lines: + heading = re.match(r"^(#{1,6})\s+(.*)$", line) + if heading: + level = len(heading.group(1)) + title = heading.group(2).strip().casefold() + if skip_level is not None and level <= skip_level: + skip_level = None + if skip_level is None and title in heading_set: + skip_level = level + continue + if skip_level is None: + result.append(line) + + return "\n".join(result).strip() + + +def preprocess_markdown_for_pdf(source: Path, working: Path, excluded_headings: list[str]) -> Path: + meta, raw_body = parse_frontmatter(source.read_text(encoding="utf-8")) + body = publish_body(raw_body).strip() + body = strip_named_sections(body, excluded_headings) + body = replace_obsidian_embeds(body, source) + body = replace_obsidian_links(body) + body = replace_obsidian_block_ids(body) + body = render_mermaid_diagrams(body, working) + + if meta.get("title") and not re.search(r"^#\s+", body, re.M): + body = f"# {meta['title']}\n\n{body}" + + prepared = working / f"{source.stem}.md" + prepared.write_text(body + "\n", encoding="utf-8") + return prepared + + +def build_pdf(source: Path, output: Path, excluded_headings: list[str]) -> Path: + output.parent.mkdir(parents=True, exist_ok=True) + with tempfile.TemporaryDirectory(prefix="secondbrain-pdf-") as temporary: + working = Path(temporary) + prepared = preprocess_markdown_for_pdf(source, working, excluded_headings) + resource_path = os.pathsep.join([str(source.parent), str(ROOT), str(working)]) + command = [ + "pandoc", + str(prepared), + "--from", + "gfm+pipe_tables", + "--standalone", + "--pdf-engine=typst", + "--resource-path", + resource_path, + "--output", + str(output), + ] + try: + subprocess.run(command, check=True, capture_output=True, text=True) + except FileNotFoundError as exc: + raise RuntimeError("pandoc was not found on PATH. Install pandoc before building PDF deliverables.") from exc + except subprocess.CalledProcessError as exc: + stderr = (exc.stderr or "").strip() + raise RuntimeError(f"Pandoc PDF generation failed: {stderr or exc}") from exc + return output + + +def list_kind(block: str) -> str | None: + lines = block.splitlines() + if all(re.match(r"^- ", line) for line in lines): + return "bullet" + if all(re.match(r"^\d+\. ", line) for line in lines): + return "number" + return None + + +def set_font(run, name: str, size: float, *, bold: bool = False, italic: bool = False, color: str = "000000") -> None: + run.font.name = name + run._element.rPr.rFonts.set(qn("w:ascii"), name) + run._element.rPr.rFonts.set(qn("w:hAnsi"), name) + run.font.size = Pt(size) + run.font.bold = bold + run.font.italic = italic + run.font.color.rgb = RGBColor.from_string(color) + + +def style_paragraph(paragraph, *, after: int, before: int = 0, line: int = 280) -> None: + fmt = paragraph.paragraph_format + fmt.space_before = Pt(before) + fmt.space_after = Pt(after) + fmt.line_spacing = line / 240 + + +def configure_styles(doc: Document) -> None: + normal = doc.styles["Normal"] + normal.font.name = "Calibri" + normal._element.rPr.rFonts.set(qn("w:ascii"), "Calibri") + normal._element.rPr.rFonts.set(qn("w:hAnsi"), "Calibri") + normal.font.size = Pt(11) + + style_specs = { + "Title": ("Calibri", 18, True, "1F1F1F"), + "Heading 1": ("Calibri", 16, True, "2E74B5"), + "Heading 2": ("Calibri", 13, True, "2E74B5"), + "Heading 3": ("Calibri", 12, True, "1F4D78"), + } + for name, (font, size, bold, color) in style_specs.items(): + style = doc.styles[name] + style.font.name = font + style._element.rPr.rFonts.set(qn("w:ascii"), font) + style._element.rPr.rFonts.set(qn("w:hAnsi"), font) + style.font.size = Pt(size) + style.font.bold = bold + style.font.color.rgb = RGBColor.from_string(color) + + for missing in ("List Bullet", "List Number"): + if missing not in [style.name for style in doc.styles]: + doc.styles.add_style(missing, WD_STYLE_TYPE.PARAGRAPH) + + +def set_page_geometry(doc: Document) -> None: + section = doc.sections[0] + section.page_width = Inches(PAGE_WIDTH_IN) + section.page_height = Inches(PAGE_HEIGHT_IN) + section.top_margin = Inches(MARGIN_IN) + section.bottom_margin = Inches(MARGIN_IN) + section.left_margin = Inches(MARGIN_IN) + section.right_margin = Inches(MARGIN_IN) + section.header_distance = Inches(0.492) + section.footer_distance = Inches(0.492) + + +def add_footer(doc: Document, title: str) -> None: + section = doc.sections[0] + footer = section.footer + para = footer.paragraphs[0] + para.alignment = WD_ALIGN_PARAGRAPH.RIGHT + style_paragraph(para, after=0, line=240) + run = para.add_run(title + " | ") + set_font(run, "Calibri", 9, color="666666") + + fld_begin = OxmlElement("w:fldChar") + fld_begin.set(qn("w:fldCharType"), "begin") + instr = OxmlElement("w:instrText") + instr.set(qn("xml:space"), "preserve") + instr.text = " PAGE " + fld_separate = OxmlElement("w:fldChar") + fld_separate.set(qn("w:fldCharType"), "separate") + fld_end = OxmlElement("w:fldChar") + fld_end.set(qn("w:fldCharType"), "end") + + page_run = para.add_run() + page_run._r.append(fld_begin) + page_run._r.append(instr) + page_run._r.append(fld_separate) + page_run._r.append(fld_end) + set_font(page_run, "Calibri", 9, color="666666") + + +def add_title_block(doc: Document, title: str, audience: str, updated: str) -> None: + title_p = doc.add_paragraph(style="Title") + title_p.alignment = WD_ALIGN_PARAGRAPH.LEFT + style_paragraph(title_p, after=6, line=240) + set_font(title_p.add_run(title), "Calibri", 18, bold=True, color="1F1F1F") + + meta = doc.add_paragraph() + style_paragraph(meta, after=12, line=240) + meta.alignment = WD_ALIGN_PARAGRAPH.LEFT + meta_run = meta.add_run(f"Audience: {audience} | Updated: {updated}") + set_font(meta_run, "Calibri", 10, color="666666") + + +def add_heading(doc: Document, title: str, level: int) -> None: + style = "Heading 1" if level <= 2 else "Heading 2" if level == 3 else "Heading 3" + para = doc.add_paragraph(style=style) + para.alignment = WD_ALIGN_PARAGRAPH.LEFT + before = 16 if style == "Heading 1" else 12 if style == "Heading 2" else 8 + after = 8 if style == "Heading 1" else 6 if style == "Heading 2" else 4 + style_paragraph(para, before=before, after=after, line=240) + run = para.add_run(title) + if style == "Heading 1": + set_font(run, "Calibri", 16, bold=True, color="2E74B5") + elif style == "Heading 2": + set_font(run, "Calibri", 13, bold=True, color="2E74B5") + else: + set_font(run, "Calibri", 12, bold=True, color="1F4D78") + + +def add_paragraph(doc: Document, text: str) -> None: + para = doc.add_paragraph() + style_paragraph(para, after=6, line=264) + para.alignment = WD_ALIGN_PARAGRAPH.LEFT + run = para.add_run(clean_inline(text)) + set_font(run, "Calibri", 11, color="000000") + + +def add_list(doc: Document, block: str, ordered: bool) -> None: + for line in block.splitlines(): + item = re.sub(r"^(- |\d+\. )", "", line).strip() + para = doc.add_paragraph(style="List Number" if ordered else "List Bullet") + style_paragraph(para, after=6, line=280) + para.paragraph_format.left_indent = Inches(0.25) + para.paragraph_format.first_line_indent = Inches(-0.25) + run = para.add_run(clean_inline(item)) + set_font(run, "Calibri", 11, color="000000") + + +def add_summary_table(doc: Document, title: str, summary: str, recommendation: str) -> None: + table = doc.add_table(rows=3, cols=2) + table.style = "Table Grid" + table.autofit = False + widths = [Inches(1.6), Inches(CONTENT_WIDTH_IN - 1.6)] + labels = ["Document", "Summary", "Recommendation"] + values = [title, summary, recommendation] + for row, label, value in zip(table.rows, labels, values, strict=True): + for idx, width in enumerate(widths): + row.cells[idx].width = width + left, right = row.cells + left.text = "" + right.text = "" + lp = left.paragraphs[0] + rp = right.paragraphs[0] + style_paragraph(lp, after=0, line=240) + style_paragraph(rp, after=0, line=264) + set_font(lp.add_run(label), "Calibri", 10, bold=True, color="1F1F1F") + set_font(rp.add_run(value), "Calibri", 10.5, color="000000") + + +def build_docx(source: Path, output: Path) -> Path: + meta, raw_body = parse_frontmatter(source.read_text(encoding="utf-8")) + body = publish_body(raw_body) + sections = parse_sections(body) + if sections and sections[0].level == 1: + sections = sections[1:] + + doc = Document() + configure_styles(doc) + set_page_geometry(doc) + + title = meta.get("title", source.stem.replace("-", " ").title()) + audience = meta.get("audience", "") + updated = meta.get("updated", "") + add_title_block(doc, title, audience, updated) + add_footer(doc, title) + + summary_section = next((section for section in sections if section.title.lower() == "summary"), None) + recommendation = "" + for section in sections: + if section.title.lower() == "recommendation": + recommendation = " ".join(clean_inline(block) for block in section.blocks[:2]) + break + if summary_section: + summary = " ".join(clean_inline(block) for block in summary_section.blocks[:2]) + add_summary_table(doc, title, summary, recommendation or "Treat the staged model as a pilot and refine it with evidence.") + doc.add_paragraph() + + for section in sections: + lowered = section.title.lower() + if lowered in {"summary", "notes"}: + continue + if lowered == "document" and not section.blocks: + continue + if section.title and lowered != "document": + add_heading(doc, section.title, section.level) + for block in section.blocks: + kind = list_kind(block) + if kind == "bullet": + add_list(doc, block, ordered=False) + elif kind == "number": + add_list(doc, block, ordered=True) + else: + add_paragraph(doc, block) + + output.parent.mkdir(parents=True, exist_ok=True) + doc.save(output) + return output + + +def command_docx(args: argparse.Namespace) -> int: + source = Path(args.source).resolve() + if not source.exists(): + print(f"ERROR source not found: {source}") + return 1 + try: + output = Path(args.output).resolve() if args.output else default_output_for(source, ".docx") + except ValueError as exc: + print(f"ERROR {exc}") + return 1 + built = build_docx(source, output) + print(built) + return 0 + + +def command_pdf(args: argparse.Namespace) -> int: + source = Path(args.source).resolve() + if not source.exists(): + print(f"ERROR source not found: {source}") + return 1 + try: + output = Path(args.output).resolve() if args.output else default_output_for(source, ".pdf") + except ValueError as exc: + print(f"ERROR {exc}") + return 1 + try: + built = build_pdf(source, output, args.exclude_heading) + except RuntimeError as exc: + print(f"ERROR {exc}") + return 1 + print(built) + return 0 + + +def parser() -> argparse.ArgumentParser: + p = argparse.ArgumentParser(description=__doc__) + sub = p.add_subparsers(dest="command", required=True) + + docx = sub.add_parser("docx", help="Build a .docx deliverable from a managed Markdown document") + docx.add_argument("source", help="Path to the source Markdown document") + docx.add_argument("--output", help="Optional explicit output path for the .docx") + docx.set_defaults(func=command_docx) + + pdf = sub.add_parser("pdf", help="Build a .pdf deliverable from a Markdown note or managed document") + pdf.add_argument("source", help="Path to the source Markdown document") + pdf.add_argument("--output", help="Optional explicit output path for the .pdf") + pdf.add_argument( + "--exclude-heading", + action="append", + default=[], + help="Heading title to omit from the rendered PDF; may be passed more than once", + ) + pdf.set_defaults(func=command_pdf) + return p + + +def main(argv: Iterable[str] | None = None) -> int: + args = parser().parse_args(argv) + return args.func(args) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/framework/tools/dtm.py b/framework/tools/dtm.py index 2fe7f90..6729efe 100644 --- a/framework/tools/dtm.py +++ b/framework/tools/dtm.py @@ -18,6 +18,7 @@ SECTIONS = [ "Previous Day", "Focus", + "Blockers", "Personal To-Do", "Professional To-Do", "Schedule & Recurring", @@ -34,12 +35,21 @@ TASK_RE = re.compile(r"^- \[ \] .+$", re.M) QUESTION_RE = re.compile(r"^- (?!\[[ xX]\] )(?! + ## Personal To-Do {block(personal, 'No carried personal tasks.')} @@ -231,31 +329,31 @@ def replace_frontmatter_value(text: str, field: str, value: str) -> str: def close_day(day: date, next_day: date | None = None) -> bool: path = note_path(day) if not path.exists(): - print(f"No Daily Note to close: {path.relative_to(ROOT)}") + print(f"No Daily Note to close: {path.relative_to(vault_root())}") return False text = path.read_text(encoding="utf-8") changed = replace_frontmatter_value(text, "status", "closed") if next_day: changed = replace_frontmatter_value( - changed, "next", f'"[[daily/{next_day.isoformat()}|{next_day.isoformat()}]]"' + changed, "next", f'"[[{note_link(next_day)}|{next_day.isoformat()}]]"' ) if changed != text: path.write_text(changed, encoding="utf-8") - print(f"Closed {path.relative_to(ROOT)}") + print(f"Closed {path.relative_to(vault_root())}") else: - print(f"Already closed {path.relative_to(ROOT)}") + print(f"Already closed {path.relative_to(vault_root())}") return True def open_day(day: date) -> bool: - path = note_path(day) + path = active_note_path(day) if path.exists(): - print(f"Already open/present: {path.relative_to(ROOT)}") + print(f"Already open/present: {path.relative_to(vault_root())}") return False previous_text = read_note(day - timedelta(days=1)) path.parent.mkdir(parents=True, exist_ok=True) path.write_text(render_note(day, previous_text), encoding="utf-8") - print(f"Created {path.relative_to(ROOT)}") + print(f"Created {path.relative_to(vault_root())}") return True @@ -274,10 +372,10 @@ def append_activity(day: date, actor: str, message: str, timestamp: str | None = text = path.read_text(encoding="utf-8") notes_match = re.search(r"^## Notes & Activity\s*$", text, re.M) if notes_match is None: - raise DTMError(f"{path.relative_to(ROOT)} is missing the Notes & Activity section") + raise DTMError(f"{path.relative_to(vault_root())} is missing the Notes & Activity section") decisions_match = re.search(r"^## Decisions\s*$", text, re.M) if decisions_match is None: - raise DTMError(f"{path.relative_to(ROOT)} is missing the Decisions section") + raise DTMError(f"{path.relative_to(vault_root())} is missing the Decisions section") entry = f"- {time_value} — {actor.strip()} — {message.strip()}" notes_heading = "## Notes & Activity" @@ -290,7 +388,7 @@ def append_activity(day: date, actor: str, message: str, timestamp: str | None = notes_body = text[notes_end:marker_start] notes_lines = notes_body.rstrip("\n").splitlines() if entry in notes_lines: - print(f"Activity already present in {path.relative_to(ROOT)}") + print(f"Activity already present in {path.relative_to(vault_root())}") return False if notes_lines and notes_lines[-1].strip(): notes_lines.append(entry) @@ -300,7 +398,7 @@ def append_activity(day: date, actor: str, message: str, timestamp: str | None = updated_before = f"{notes_prefix}{notes_heading}\n{chr(10).join(notes_lines)}\n" path.write_text(f"{updated_before}## Decisions{after}", encoding="utf-8") - print(f"Updated {path.relative_to(ROOT)}") + print(f"Updated {path.relative_to(vault_root())}") return True @@ -314,6 +412,37 @@ def rollover(day: date) -> int: return 0 +def archive_notes(keep: int) -> int: + if keep < 1: + raise DTMError("Keep count must be at least 1") + note_files = active_note_files() + if len(note_files) <= keep: + print(f"No Daily Notes archived; {len(note_files)} active note(s) within keep window {keep}.") + return 0 + archive_dir = daily_archive_dir() + archive_dir.mkdir(parents=True, exist_ok=True) + to_move = note_files[:-keep] + moved_days: list[date] = [] + for path in to_move: + if not DATE_RE.fullmatch(path.stem): + raise DTMError(f"Cannot archive non-date Daily Note {path.relative_to(vault_root())}") + day = date.fromisoformat(path.stem) + target = archived_note_path(day) + if target.exists(): + raise DTMError(f"Archive destination already exists: {target.relative_to(vault_root())}") + path.rename(target) + moved_days.append(day) + print(f"Archived {path.relative_to(vault_root())} -> {target.relative_to(vault_root())}") + affected_days = set(moved_days) + for moved_day in moved_days: + affected_days.add(moved_day - timedelta(days=1)) + affected_days.add(moved_day + timedelta(days=1)) + for affected_day in sorted(affected_days): + sync_note_links(affected_day) + print(f"Archived {len(moved_days)} Daily Note(s); kept {keep} active.") + return 0 + + def validate_recurrence() -> tuple[list[str], list[str]]: errors: list[str] = [] warnings: list[str] = [] @@ -380,19 +509,25 @@ def require_valid_recurrence() -> None: def lint() -> int: errors, warnings = validate_recurrence() - note_files = sorted(path for path in DAILY.glob("*.md") if path.name != "README.md") + note_files = all_note_files() for path in note_files: if not DATE_RE.fullmatch(path.stem): warnings.append(f"{path.relative_to(ROOT)}: filename is not an ISO date") continue + note_day = date.fromisoformat(path.stem) text = path.read_text(encoding="utf-8") - for heading in SECTIONS: + required_sections = [ + heading + for heading in SECTIONS + if heading != "Blockers" or note_day >= BLOCKERS_REQUIRED_FROM + ] + for heading in required_sections: count = len(re.findall(rf"^## {re.escape(heading)}\s*$", text, re.M)) if count != 1: errors.append( f"{path.relative_to(ROOT)}: section {heading!r} occurs {count} times" ) - positions = [text.find(f"## {heading}") for heading in SECTIONS] + positions = [text.find(f"## {heading}") for heading in required_sections] if all(position >= 0 for position in positions) and positions != sorted(positions): errors.append(f"{path.relative_to(ROOT)}: required sections are out of order") for field in ("type: daily-note", f"date: {path.stem}", "status:"): @@ -412,7 +547,7 @@ def lint() -> int: def status() -> int: require_valid_recurrence() - notes = sorted(DAILY.glob("*.md")) + notes = active_note_files() if not notes: print("No Daily Notes.") return 0 @@ -444,6 +579,8 @@ def main() -> int: activity_parser.add_argument("--actor", default="DTM") activity_parser.add_argument("--time", dest="timestamp") activity_parser.add_argument("message") + archive_parser = sub.add_parser("archive") + archive_parser.add_argument("--keep", type=int, default=14) sub.add_parser("lint") sub.add_parser("status") args = parser.parse_args() @@ -459,6 +596,8 @@ def main() -> int: if args.command == "activity": append_activity(args.date, args.actor, args.message, args.timestamp) return 0 + if args.command == "archive": + return archive_notes(args.keep) if args.command == "lint": return lint() return status() diff --git a/framework/tools/export_framework.py b/framework/tools/export_framework.py index 8dbeaa5..4716f01 100644 --- a/framework/tools/export_framework.py +++ b/framework/tools/export_framework.py @@ -56,6 +56,8 @@ def copy_entry(source: Path, destination: Path) -> None: if source.is_symlink(): raise ExportError(f"Symlinks are not publishable: {source.relative_to(LIVE_ROOT)}") if source.is_file(): + if source.name == ".DS_Store": + return destination.parent.mkdir(parents=True, exist_ok=True) shutil.copy2(source, destination) return @@ -65,13 +67,15 @@ def copy_entry(source: Path, destination: Path) -> None: if item.is_symlink(): raise ExportError(f"Symlinks are not publishable: {item.relative_to(LIVE_ROOT)}") if item.is_file(): + if item.name == ".DS_Store": + continue target = destination / item.relative_to(source) target.parent.mkdir(parents=True, exist_ok=True) shutil.copy2(item, target) def staged_files(root: Path) -> list[Path]: - return sorted(path for path in root.rglob("*") if path.is_file()) + return sorted(path for path in root.rglob("*") if path.is_file() and path.name != ".DS_Store") def privacy_scan(root: Path, manifest: dict[str, object]) -> None: diff --git a/tests/test_framework.py b/tests/test_framework.py index dd81d39..f7e5ef4 100644 --- a/tests/test_framework.py +++ b/tests/test_framework.py @@ -57,6 +57,16 @@ def test_voice_skill_excludes_drafts_and_is_explicit(self): self.assertIn("writing/drafts/", instructions) self.assertIn("allow_implicit_invocation: false", metadata) + def test_markdown_to_pdf_skill_points_to_managed_command(self): + skill = ROOT / "framework" / "skills" / "markdown-to-pdf" + instructions = (skill / "SKILL.md").read_text(encoding="utf-8") + metadata = (skill / "agents" / "openai.yaml").read_text(encoding="utf-8") + + self.assertIn("name: markdown-to-pdf", instructions) + self.assertIn("tools/document_deliverables.py pdf", instructions) + self.assertIn("Mermaid", instructions) + self.assertIn("allow_implicit_invocation: false", metadata) + class RecurrenceSafetyTests(unittest.TestCase): def test_invalid_rule_fails_before_rollover_mutates_notes(self): @@ -96,6 +106,105 @@ def test_invalid_rule_fails_before_rollover_mutates_notes(self): self.assertIn("status: open", previous.read_text(encoding="utf-8")) self.assertFalse((daily / f"{today.isoformat()}.md").exists()) + def test_rollover_does_not_carry_incomplete_recurring_schedule_items_to_non_applicable_days(self): + dtm = load_dtm_module() + saturday = date(2030, 1, 5) + friday = saturday - timedelta(days=1) + + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + daily = root / "daily" + daily.mkdir() + previous = daily / f"{friday.isoformat()}.md" + previous.write_text( + "\n".join( + [ + "---", + f"title: Daily Note — {friday.isoformat()}", + "type: daily-note", + f"date: {friday.isoformat()}", + "status: open", + "---", + "", + "## Previous Day", + "", + "Previous note: [[daily/2029-01-03|2029-01-03]]", + "", + "## Focus", + "", + "1. Finish Friday work.", + "", + "## Blockers", + "", + "", + "", + "## Personal To-Do", + "", + "", + "", + "## Professional To-Do", + "", + "", + "", + "## Schedule & Recurring", + "", + "- [ ] Weekday recurring item ", + "- [ ] One-off carryable schedule item", + "", + "## Notes & Activity", + "", + "- 17:00 — DTM — Wrapped up Friday.", + "", + "## Decisions", + "", + "", + "", + "## References", + "", + "", + "", + "## Open Questions", + "", + "", + "", + ] + ), + encoding="utf-8", + ) + recurrence = root / "recurring-tasks.json" + recurrence.write_text( + json.dumps( + { + "version": 1, + "tasks": [ + { + "id": "weekday-rule", + "task": "Weekday recurring item", + "area": "schedule", + "enabled": True, + "schedule": { + "frequency": "weekly", + "weekdays": ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday"], + }, + } + ], + } + ), + encoding="utf-8", + ) + dtm.DAILY = daily + dtm.RECURRENCE = recurrence + + dtm.rollover(saturday) + + created = (daily / f"{saturday.isoformat()}.md").read_text(encoding="utf-8") + self.assertIn("- [ ] One-off carryable schedule item", created) + self.assertNotIn("weekday-rule", created) + self.assertIn( + "## Schedule & Recurring\n\n- [ ] One-off carryable schedule item", + created, + ) + if __name__ == "__main__": unittest.main()