diff --git a/docs/content/deep-dives/_index.md b/docs/content/deep-dives/_index.md index cac0fce..2cf384e 100644 --- a/docs/content/deep-dives/_index.md +++ b/docs/content/deep-dives/_index.md @@ -7,7 +7,7 @@ bookCollapseSection = true # Deep dives Understanding-oriented discussion of the *why* behind Katalyst: the -[vision and scope]({{< relref "vision.md" >}}), the +[vision and scope]({{< relref "why-katalyst/_index.md" >}}), the [domain model]({{< relref "domain-model/_index.md" >}}) the tool is built on, and the deeper design discussions that no single page or package owns: how [checks work]({{< relref "domain-model/checks.md" >}}) and the libraries that diff --git a/docs/content/deep-dives/domain-model/_index.md b/docs/content/deep-dives/domain-model/_index.md index f40de4f..7ef5734 100644 --- a/docs/content/deep-dives/domain-model/_index.md +++ b/docs/content/deep-dives/domain-model/_index.md @@ -15,7 +15,7 @@ The most central concept in Katalyst is a **base**: a storage system that holds An **operation** is something a base lets you do with data: read, list, aggregate, write, and eventually query. Which operations a base supports, and what structural commitments those operations require, is the subject of -[progressive operations]({{< relref "../progressive-operations.md" >}}). +[progressive operations]({{< relref "../why-katalyst/progressive-operations.md" >}}). In addition to natively-supported operations for various backends, Katalyst provides two very useful kinds of operation. diff --git a/docs/content/deep-dives/vision.md b/docs/content/deep-dives/vision.md deleted file mode 100644 index 20f8bd7..0000000 --- a/docs/content/deep-dives/vision.md +++ /dev/null @@ -1,109 +0,0 @@ -+++ -title = "Vision and scope" -weight = 10 -+++ - -# Vision and scope - -Traditional data management often forces teams into binary choices: -structured or unstructured, rigid or chaotic. Katalyst is an experimental -framework aimed at enabling fast, low-risk evolution through progressive typing -across bases and operations. - -## Database management is risky and rigid - -Backend architecture and database changes are frequently high-risk operations. -Teams therefore wrap data systems in heavy governance for access control, -schema changes, and migrations. Those controls are necessary, but the cost can -make change rare. Rare change creates rigidity. - -## AI-native apps want flexibility - -AI-driven systems increasingly involve non-engineers designing prompts and -workflows that generate and transform data. To move quickly, teams often store -semi-structured or unstructured data. - -Over time, familiar pain appears: - -1. More frequent bugs. -2. Slower debugging and incident resolution. -3. Confusion around ownership and system boundaries. -4. Hard-to-read workflows and data paths. -5. Risky refactors and fragile changes. -6. Fewer bugs caught before production. -7. Growing dependence on manual QA. - -## Move past the structured/unstructured dichotomy - -When schema changes are rare and expensive, teams are pushed toward two bad -extremes: rigid systems with high coordination costs, or flexible systems with -weak guarantees. - -Katalyst explores a third path: make schema changes fast and safe enough that -teams can progressively add structure as it becomes necessary. - -This follows a proven pattern from the application layer (TypeScript, mypy, -Pydantic) and extends it deeper into storage systems. - -## What Katalyst is - -Katalyst is an experimental framework for progressive typing in the storage -layer, designed for AI-readiness: - -- Validate content across structured and unstructured forms. -- Apply one validation model across filesystems and databases. -- Use validation rules not only for checks, but as primitives for migration - and system evolution. - -The core is a framework for **content validation**. Where practical, it builds -on existing systems rather than replacing them (for example JSON Schema and -other established validators). Unlike most validation frameworks, Katalyst is -intended to: - -- Span both unstructured and structured content. -- Support deterministic and non-deterministic rule evaluation. - -## Why this matters - -Katalyst is designed as infrastructure for AI harnesses and agentic systems. -Representative use cases include: - -- Guardrails for agents creating or updating content, including memory stores - and shared knowledge bases. -- Validation across mixed content types, rather than only typed records. -- Support for storage-layer migrations (for example markdown to SQLite or - other backends) without losing validation guarantees. - -## Multiple form factors - -The same DSL is intended to be exposed through several form factors: - -1. A linter that applies Katalyst rules to files in a filesystem. -2. A CLI that enforces rules on write operations in a filesystem (markdown, - YAML, CSV, and related formats). -3. A server that enforces rules on write operations for SQL and NoSQL stores - (for example SQLite, PostgreSQL, MongoDB). - -These form factors share one core idea: schemas and linters are closely -related and should compose across storage boundaries. The conceptual basis, -why each backend tier unlocks new operations, is in -[Progressive operations]({{< relref "progressive-operations.md" >}}) and the -[domain model]({{< relref "domain-model/_index.md" >}}). - -## Current implementation status - -The current implementation in this repo is intentionally narrower than the -full direction above: - -- Filesystem-first CLI. -- Markdown frontmatter validation via JSON Schema. -- Config-driven schema resolution through `.katalyst/`. - -The DSL is expected to grow to support validation for object-like data (YAML, -JSON, SQL-backed records), markdown content, and file/directory structures, -along with more storage backends and tooling that reuses validation rules for -additional operations (especially migrations). - -This page is intentionally directional: treat it as scope and rationale, not -a frozen specification. The implementation here represents an early, practical -slice of the broader goal. diff --git a/docs/content/deep-dives/why-katalyst/_index.md b/docs/content/deep-dives/why-katalyst/_index.md new file mode 100644 index 0000000..aa32dbb --- /dev/null +++ b/docs/content/deep-dives/why-katalyst/_index.md @@ -0,0 +1,32 @@ ++++ +title = "Why Katalyst?" +weight = 10 +bookCollapseSection = true ++++ + +# Why Katalyst? + +In order for agents to become capable of real work, the next frontier seems to revolve around two things: + +1. Making operational context more legible to agents. +2. Enabling agents to curate their own memory—individual or shared—in a way that's robust, durable, and efficient. + +These problems have several things in common: +* Content that's a mix of text and more structured data. +* A compute model that's a mix of LLMs and deterministic software. +* The need for humans and agents need to make sense of the same information. +* UI/UX questions that end up being grounded in shared primitives. + +I've come to see the two problems as two faces of the same coin. By enabling agents to curate internally consistent, always-up-to-date knowledge bases, I believe we can serve both needs. + +Katalyst is designed to provide the right content primitives and large fraction of the deterministic compute required to solve this problem. + +## How this section is organized + +This section contains the first-principles reasoning underlying Katalyst's primitives. This isn't necessary if you just want to use the library. It will mostly be useful for those who want a solid, well-grounded perspective on how to build AI knowledge bases. + +- [What is curation?]({{< relref "what-is-curation.md" >}}) defines curation and the criteria that make curated information useful. +- [Internal consistency]({{< relref "internal-consistency.md" >}}) explains how a knowledge base decides which contradictions count. +- [Completeness]({{< relref "completeness.md" >}}) covers the scope of information a knowledge base claims to contain. +- [Up-to-dateness]({{< relref "up-to-dateness.md" >}}) describes how a knowledge base stays connected to the world it represents. +- [Progressive operations]({{< relref "progressive-operations.md" >}}) explains how storage backends grow richer as query complexity increases. \ No newline at end of file diff --git a/docs/content/deep-dives/why-katalyst/completeness.md b/docs/content/deep-dives/why-katalyst/completeness.md new file mode 100644 index 0000000..4dc2a25 --- /dev/null +++ b/docs/content/deep-dives/why-katalyst/completeness.md @@ -0,0 +1,16 @@ ++++ +title = "Completeness" +weight = 17 ++++ + +# Completeness + + + +Completeness means covering all the relevant material within some scope. + +The scope matters. A knowledge base does not need to contain every true fact in the world. It needs to contain the material required by the purpose it claims to serve. + + + +* Claims about ordering rely on claims about completeness \ No newline at end of file diff --git a/docs/content/deep-dives/why-katalyst/internal-consistency.md b/docs/content/deep-dives/why-katalyst/internal-consistency.md new file mode 100644 index 0000000..16cd627 --- /dev/null +++ b/docs/content/deep-dives/why-katalyst/internal-consistency.md @@ -0,0 +1,94 @@ ++++ +title = "Internal consistency" +weight = 16 ++++ + +# Internal consistency + + + +Consistency means being free from internal contradiction. On its surface, this seems simple: the knowledge base can't say "A is true" in one place and "A is false" in another. + + + +## Content claims vs structural claims + +However, there's some subtlety here. Imagine a folder containing customer feedback interviews. In one transcript, customer A says, "this product is amazing!" In another, customer B says "the product is terrible." Those statements are in direct contradiction, but is the knowledge base inconsistent? + +I'd argue no. The knowledge base isn't claiming that both customer opinions are true descriptions of the product. It is claiming that both interviews happened and that both customers said what the transcripts record. + +Imagine adding a README in the folder: "This folder contains interview transcripts from many customers. Customers may disagree among themselves." The README makes the *structure* of the folder explicit. It tells a reader what kind of content the folder contains, how to interpret disagreement between items, and which guarantees the folder is making. + +> **Structure** is a set of rules and conventions that distinguish structural claims from ordinary content within a knowledge base. + +If the README said, "This folder contains interview transcripts from many customers. All customers absolutely love the product," that would contradict customer B's statement, and the folder would be internally inconsistent. + +In other words, we need to distinguish between two types of claims. + +> A **content claim** says something *within* the knowledge base. + +> A **structural claim** says something *about* the knowledge base: what kind of content it contains, how that content is organized, and any other guarantees the system makes about it. + +For determining consistency, only structural claims matter. + +> **Internal consistency**: A knowledge base is internally consistent if it is free from contradictory structural claims. + + +## Defining structure + + + + + +In the customer feedback example, the README defines a simple structure. There are lots of other examples of + +{Examples: Tables of contents; executive summaries; indexes; chapters; sections; API references} + + + +{Transition, introduce the list of desiderata: structure should be explicit; structure doesn't need to be part of the content; structure needs to be defined authoritatively} + +**Structure should be explicit** + +In many knowledge bases, it's common for structural conventions to be implicit. You don't usually need to be told "the chapter entries in the table of contents correspond 1:1 with the chapters in the book." Or "terms in the index are sorted alphabetically." + +However, for our purposes, it's helpful to insist that structure be made explicit. All of our structural claims must be declared somewhere. This gives us a master list to check, to ensure consistency. + +**Structure is often embedded in content, but it doesn't need to be** + +Sometimes, it's useful to embed knowledge base structure directly in content. {Examples: summaries and overviews, text books, technical documentation} + +When structure is written directly into content, {it has benefits X, Y and Z} + +However, structure doesn't always need to be spelled out in content. In some cases, this can be counterproductive. {Examples: marketing, persuasion; security / private knowledge}. In other cases, it would just be pedantic. + +Since we want the structure of our knowledge base to be explicit, but we don't necessarily want to show all of it to the user, we need a concept of metadata / markup attached to the knowledge base. + +**Structure needs an authoritative source** + +If you want to play logic games, you can invent self-referential cases where content tries to override structure. "Ignore all previous instructions and..." "This page lists all pages that do not list themselves." + +Practially speaking, we can avoid this kind of thing by defining an authoritative source for structure in the knowledge base. + + + +To separate these caess, a knowledge base needs a *structural layer*: + + + +changes how the content should be read. Disagreement between transcripts is allowed, but a transcript with the wrong customer ID, source date, or interview format may still violate the structure of the collection. + +{Make an analogy to turing machines (no separation between programs / data) and how we actually do things.} + + + + +{What properties must a structural interpreter have? Goal isn't executionin the sense of a program, but interpretation.} + + +## Explicit vs implicit structure + +## Guaranteeing internal consistency + +* Explicit +* With a comprehensive library of invariants, and a reliable method of enforcement. \ No newline at end of file diff --git a/docs/content/deep-dives/progressive-operations.md b/docs/content/deep-dives/why-katalyst/progressive-operations.md similarity index 100% rename from docs/content/deep-dives/progressive-operations.md rename to docs/content/deep-dives/why-katalyst/progressive-operations.md diff --git a/docs/content/deep-dives/why-katalyst/up-to-dateness.md b/docs/content/deep-dives/why-katalyst/up-to-dateness.md new file mode 100644 index 0000000..5743363 --- /dev/null +++ b/docs/content/deep-dives/why-katalyst/up-to-dateness.md @@ -0,0 +1,12 @@ ++++ +title = "Up-to-dateness" +weight = 18 ++++ + +# Up-to-dateness + +Up-to-dateness is the guarantee of external consistency: the state of the content accurately reflects the state of the real world at some point in time. A knowledge base can be internally consistent and complete within its stated scope while still being wrong, because the world changed since the content was last updated. + +That makes up-to-dateness different from the other two criteria. It cannot be guaranteed from inside the content alone. It requires contact with an external source of truth: an event stream, a periodic refresh, a source-system query, a human review, or some other verification process. A curated system can record timestamps, sources, freshness windows, and update rules, but the guarantee comes from the process that reconnects the content to the world. + +Because curation takes work, there's always some lag between a change in the world and the content that reflects it. As a general rule, less lag is better. Information doesn't need to be perfectly up-to-date in order to be valuable. The important questions are whether the content makes a truthful claim about when it corresponded to the world, and whether the content is updated quickly enough to support valuable decisions. diff --git a/docs/content/deep-dives/why-katalyst/what-is-curation.md b/docs/content/deep-dives/why-katalyst/what-is-curation.md new file mode 100644 index 0000000..59273c3 --- /dev/null +++ b/docs/content/deep-dives/why-katalyst/what-is-curation.md @@ -0,0 +1,91 @@ ++++ +title = "What is curation?" +weight = 15 ++++ + +# What is curation? + + +If AI systems are going to curate content effectively, they need a clear target: what counts as curation, and what good curation should accomplish. + + + +Let's start from examples. The work of curation shows up in many practical tasks: + +- summarizing the useful parts of a conversation thread in a document that people can find later +- grouping related notes so they are easier to browse +- adding dates, owners, tags, or status labels so people know what they are looking at +- rewriting headings so a reader can scan the page before committing to it +- removing duplicate or stale material + + + +These tasks look different, but they all share the same purpose: making content usable. + +> Curation is the act of making content usable. + + + +Curation trades work up front for convenience later. An analogy: in data systems, constraints like indexes and schemas add overhead when data is written, in order to make later reads faster and more reliable. Curation applies the same trade to content: extra care when information is written, edited, or organized, in exchange for less work when someone needs to use it. + +## Making information usable + +Truly first-class curation requires user empathy: putting yourself in the shoes of a user, thinking about their needs, and what will be most helpful to them. + + + +For example, in technical documentation, different readers need different shapes of content. For someone evaluating the project, the most useful content is a landing page or README that explains of what it does and the problems it solves. For a new user, it's an onboarding tutorial that gets them to a working setup. For an experienced user, it might be an API reference with exact details. Each page type curates the same body of knowledge around a different reader need. + +Here are some other examples of information curated for specific needs: + +- A bug report that includes exact reproduction steps, environment details, expected behavior, actual behavior, logs, screenshots, and why the reporter thinks it matters. +- An account brief helps a customer success manager prepare for a renewal call by gathering usage trends, open risks, support history, and likely expansion opportunities. +- A battle card helps a salesperson prepare for a competitive sales call by summarizing the competitor the prospect already uses and the objections most likely to come up. +- A board packet that condenses financials, customer signals, hiring plans, and risks into the few questions directors need to weigh in on. + +Beyond the information itself, curation can also involve the presentation of the information. + +- formatting a document so the most important information is visible before the supporting detail +- adding headings, summaries, and tables of contents so readers can scan before reading +- making content searchable through clear titles, stable terminology, tags, and aliases +- choosing UI conventions that match the task, such as filters for comparison, status badges for freshness, or callouts for warnings +- managing information density so a page gives enough context without burying the answer +- exposing affordances for agents, such as stable identifiers, structured metadata, machine-readable links, and clear boundaries between source material and interpretation + + +Curating information to this level of detail is valuable because it saves the reader from reconstructing context at the moment they need it. It is also costly. To do this effectively, the curator must know the audience, anticipate their needs, gather the source material, and shape it for that use. Part of the promise of AI curation is that it should make more of that work possible. + +## Universal properties for good curation + + + +However, even if the cost of curation drops significantly, it's often impossible to anticipate every need that every kind of user will have. + +* A user might be interested in comparing cases along a new axis. "Which product requests came from customers who are expanding this year rather than customers at risk?" +* Or they might need to answer a question where the answer depends on a unique combination of components. "What guidance applies to a customer using SSO, SCIM, and audit logs, but only on the team plan?" +* Or maybe something fundamental in the environment has changed, and past conclusions need to be reexamined with new foundational assumptions. “Now that we've changed our pricing, which of our old conclusions about customer segmentation still hold?” + + +Happily, there are some properties that are almost always helpful. Even when exact future needs are unknown, content can still be curated toward a few baseline criteria. + +This document focuses on three: +* *Internal consistency*: free from internal contradiction +* *Completeness*: covers all the relevant material within some scope +* *Up-to-dateness*: accurately reflects the state of the real world at some point in time + + +These properties are powerful because they create a trustworthy substrate for logical reasoning: answering questions, making decisions, and drawing valid conclusions. + +When an information system has all three of these properties, it becomes something stronger than a pile of information. +They are *universal* because they can support valid reasoning regardless of subject matter. Even without knowing what a body of content contains or how it will be used, it is still a safe bet that well-curated content should have these properties. + +## Defining "knowledge base" + + + +It will be useful to have a name for an information system that satisfies all three criteria at once. The closest ordinary term is "knowledge base," but that term is often used loosely. Here, I want to give it a stricter meaning: + +> A **knowledge base** is a body of information curated so that it is internally consistent, complete within a useful domain, and up to date. + +As we will see, this definition imposes enough structure to sketch useful technical requirements for AI knowledge bases. The following pages take each property in turn. + diff --git a/docs/content/reference/glossary.md b/docs/content/reference/glossary.md index 86a6059..485fb28 100644 --- a/docs/content/reference/glossary.md +++ b/docs/content/reference/glossary.md @@ -38,7 +38,7 @@ how each term maps onto today's code is documented in the per-package | **Data surface** | A representation Katalyst exposes for checks, inspectors, or `fix` to read from content: markdown body text, plain text, structured object, or file metadata. See [Data surfaces]({{< relref "data-surfaces/_index.md" >}}). | | **Measurement primitive** | A reusable building block the inspectors are built from: `object_fields` (a data dictionary over object maps), `markdown_body` (body structure), and file-metadata. | | **Metadata** | The parsed, in-memory structure of the frontmatter (a `map[string]any`). | -| **Operation** | Something a base lets you do with its data: read, list, query, aggregate, write. Each has a scope (item, collection, across collections) and structural requirements the backend must satisfy. See [progressive operations]({{< relref "../deep-dives/progressive-operations.md" >}}). | +| **Operation** | Something a base lets you do with its data: read, list, query, aggregate, write. Each has a scope (item, collection, across collections) and structural requirements the backend must satisfy. See [progressive operations]({{< relref "../deep-dives/why-katalyst/progressive-operations.md" >}}). | | **Profile class** | A group of near-identical profiles the summarizer collapses together, so output is proportional to the number of distinct profiles, not directories. | | **Project** | The whole katalyst workspace: a repo root with a `.katalyst/` **Config** that declares the bases, collections, and checks katalyst operates over. The top-level scope an empty selector addresses, and what `katalyst init` creates. Collections live within a project; the `project` package (`internal/project`) is its code home, holding the `.katalyst/` loader while the collection implementation lives under `storage/`. | | **Raw base layer** | Inspectors that profile a base directly, before any collection configuration, addressed by base-native reference (a path today). The onboarding case: "what's in this base?" | diff --git a/docs/content/welcome.md b/docs/content/welcome.md index d680df2..2fd9d79 100644 --- a/docs/content/welcome.md +++ b/docs/content/welcome.md @@ -109,7 +109,7 @@ What if structure were light: easy to add, easy to maintain, easy to change? In a world of unbounded creative collaboration with agents, the limiting factor isn't generating new ideas or gathering more information: it's having a shared language and structure to organize what we've learned, and to act on it together. -For more on these ideas, see [Deep dives]({{< relref "deep-dives/_index.md" >}}), especially [Vision and scope]({{< relref "deep-dives/vision.md" >}}). +For more on these ideas, see [Deep dives]({{< relref "deep-dives/_index.md" >}}), especially [Vision and scope]({{< relref "deep-dives/why-katalyst/_index.md" >}}). ## No, really, you should try it out