okf — Open Knowledge Format

Project-level knowledge base system for AI Agents, with automatic Git repository scanning, specification linting, and automated updates.

okf turns your Git repository into a living, queryable knowledge base that humans and AI Agents can use. Every piece of knowledge is a Markdown concept file with YAML frontmatter, generated from your code and documents automatically and kept up to date on every commit.

Features

📁 Open Knowledge Format — Open knowledge format based on Markdown + YAML Frontmatter
📄 Document Import — Import PDF, DOCX, XLSX, PPTX, HTML, CSV, TXT directly (pure-Go conversion, no Python/CGO); okf add report.pdf just works
🔍 Auto-Generation — Automatically generates knowledge base by scanning Git repository source code
⚡ Incremental Updates — Incremental updates based on Git commits
🛠 Git Hook — One-click installation, automatic knowledge base updates on every commit
📋 Lint Checking — Built-in specification compliance checker (16 rules)
🔎 Advanced Query — Filter by type, tags, or full-text search
🧠 Hybrid Semantic Search — Local natural-language search: chunk-level MiniLM embeddings + BM25, fused with weighted RRF (fully offline, no CGO)
🤖 Agent-facing MCP — Standard MCP tools for repository status/init/refresh/query/context plus durable note/event/feedback capture
🏗 Modular Architecture — Clean, layered design following Go best practices

How it works

flowchart LR
    A[Your Git repository] -->|"okf init / scan"| B[.okf/knowledge<br/>Markdown concepts]
    C["PDF · DOCX · XLSX · PPTX<br/>HTML · CSV · TXT"] -->|"okf add"| B
    D[git commit] -->|"okf hook / sync"| B
    B --> E["okf lint<br/>OKF v0.2 checks"]
    B --> F["okf search / query"]
    B --> G["MCP server<br/>status · init · refresh · query · context"]
    G --> H[AI Agents]

Installation — Quick Start (30 seconds)

Pick one of these three install methods:

1. One-click installer (recommended)

Linux / macOS:

curl -fsSL https://raw.githubusercontent.com/superops-team/okf/main/scripts/install.sh | bash

Windows (PowerShell):

irm https://raw.githubusercontent.com/superops-team/okf/main/scripts/install.ps1 | iex

If the one-liner fails with Unexpected token / " parse errors (caused by proxies HTML-encoding the response), use the download-then-run method:
iwr -useb "https://raw.githubusercontent.com/superops-team/okf/main/scripts/install.ps1" -OutFile install.ps1; .\install.ps1

The installer:

Automatically detects your OS (Linux / macOS) and CPU architecture (amd64 / arm64)
Downloads the latest pre-built binary from GitHub Releases
Verifies SHA256 checksums
Installs to /usr/local/bin/ (or ~/.local/bin/ without sudo)

2. Install via Go

go install github.com/superops-team/okf/cmd/okf@latest

3. Download from releases

Download pre-built binaries for your platform from the Releases page.

OS	Architecture	Archive
Linux	amd64 (x86_64)	`okf_<version>_linux_amd64.tar.gz`
Linux	arm64 (aarch64)	`okf_<version>_linux_arm64.tar.gz`
macOS	amd64 (Intel)	`okf_<version>_darwin_amd64.tar.gz`
macOS	arm64 (Apple Silicon)	`okf_<version>_darwin_arm64.tar.gz`
Windows	amd64	`okf_<version>_windows_amd64.zip`
Windows	arm64	`okf_<version>_windows_arm64.zip`

Usage

# Initialize knowledge base from your repo
cd /your/repo
okf init

# Show knowledge base information
okf show

# Search concepts
okf search -q "database"

# Import a real document (converts PDF/DOCX/XLSX/... to Markdown)
okf add report.pdf

# Lint check
okf lint

# Semantic (natural-language) search — build the index once, then search
okf vector index
okf search -q "check my notes for errors" -semantic

# Install Git Hook (automatic updates on every commit)
okf hook -type post-commit

# Start the MCP server for a repository. Relative --dir values resolve under --repo;
# absolute --dir values remain absolute.
okf mcp --repo /your/repo --dir .okf/knowledge

Agent-facing MCP tools

The MCP server exposes the repository knowledge service through okf_status, okf_init, okf_refresh, okf_query, and okf_context. Durable knowledge capture is available through okf_note, okf_log, and okf_feedback; okf_ask queries only those durable note/event/feedback concepts. Existing bundle/list/get/search/lint/document-import tools remain available.

Writes require a stable idempotency_key, use deterministic identities, reject unknown or incorrectly typed fields, and fail closed for path escape, symlink-root, size-limit, and credential-like metadata violations. The server persists only feedback explicitly submitted by the caller; it does not inspect a host application's private event bus. See docs/knowledge/mcp-server.md and docs/knowledge/durable-capture.md.

Semantic Search

okf search -semantic performs natural-language search over concepts, using a locally embedded MiniLM model (384-dim vectors) and an HNSW index — no network, no external runtime required. Long documents are split into heading-aware chunks, and results blend a semantic channel with a BM25 lexical channel via weighted Reciprocal Rank Fusion.

# Build (or incrementally update) the vector index — one-time, per knowledge base
okf vector index
# Inspect index state (chunks, concepts, index format version)
okf vector status
# Full rebuild (after content changes, or when upgrading index format)
okf vector rebuild
# Search semantically (hybrid: semantic + BM25, fused with weighted RRF)
okf search -q "check my notes for errors" -semantic
# Pure semantic, no lexical channel
okf search -q "how do I rebuild the index" -semantic -lexical-weight 0

Results are annotated with their source: semantic, lexical, or both. If no index exists, -semantic warns and falls back to lexical search. The MCP server exposes the same capability via okf_semantic_search.

Measuring retrieval quality

okf eval scores retrieval against a golden query set and can compare strategies side by side:

okf eval -golden pkg/eval/testdata/golden_semantic.json -path docs/knowledge -compare

Measured on this repository's own knowledge base (28 queries, 26 positive, K=5):

Strategy	Recall@5	MRR
`lexical-substring` (pre-0.5.0 behaviour)	0.0769	0.0769
`bm25-only`	0.8077	0.6538
`semantic-only`	0.9615	0.7096
`hybrid-default`	0.9615	0.7256

How it works & constraints

Chunked indexing: concepts are split on ##–#### headings into ≤1024-character chunks (code fences and tables are never split), each carrying a Title > Section breadcrumb. This matters because MiniLM truncates at 256 tokens: indexing whole concepts dropped 70.6% of this repository's knowledge-base content, so text past the truncation point was unsearchable.
Hybrid retrieval: the semantic channel (HNSW over chunk vectors) and the BM25 channel are fused with weighted RRF (k=60, equal weights by default). Tune with -lexical-weight; 0 disables the lexical channel. BM25 tokenizes identifiers into subwords (okf_semantic_search → okf/semantic/search) and CJK text into overlapping bigrams, with no dictionary dependency.
Reproducibility: indexes below 2048 chunks are searched by exact scan rather than HNSW's approximate traversal, because the approximate path does not return every node even when asked for all of them, and which nodes it misses shifts between rebuilds. Combined with a fixed RNG seed and deterministic tie-breaks, this makes results identical across rebuilds — verified by rebuilding this knowledge base repeatedly and confirming the evaluation metrics do not move.
Index cost (measured, 7 concepts → 97 chunks): chunking increases index size ~20x (14.5 KB → 287 KB) and build time ~6x (128 ms → 800 ms). Both scale with content volume, not concept count.
Index format v2 is not backward compatible: chunk-level keys differ from the old concept-level keys. okf vector status reports the format version, and loading an older index fails with an explicit prompt to run okf vector rebuild (search falls back to lexical meanwhile) rather than silently returning wrong results.
Embedded resources: the ONNX Runtime CPU library (per-OS, ~10–15 MB) plus a quantized MiniLM model (~23 MB) are embedded into the binary via go:embed and extracted to the user cache directory on first use (checksum-verified). Building for each platform only embeds that platform's resources (scripts/fetch-ort.sh / scripts/fetch-model.sh fetch them at build time; the runtime never goes online).
Dynamic loading (transparency): the ONNX Runtime shared library is loaded at runtime via dlopen from the extracted cache — the binary is self-contained but not statically linked. Cache location: os.UserCacheDir()/okf/ (override with OKF_ORT_DIR).
Limits: MiniLM embeddings are English-centric. Chunking and BM25's CJK bigrams improve Chinese retrieval, but a purely Chinese query against English content still relies on the semantic channel alone. Embedder is an interface, leaving room for stronger models (e.g. BGE-M3) or remote APIs later.
Licenses: pure-onnx (MIT), coder/hnsw (CC0-1.0), ONNX Runtime (MIT), MiniLM-L6-v2 model (Apache-2.0).

Documentation

Project Structure

.
├── cmd/okf/          # CLI entry point
│   └── main.go      # Main application
├── pkg/
│   ├── okf/         # Core types and public API
│   │   ├── types.go # Concept, KnowledgeBundle definitions
│   │   ├── api.go   # LoadBundle, SaveBundle
│   │   ├── errors.go # Error types
│   │   ├── helpers.go # Helper functions
│   │   └── meta/    # Version information
│   ├── parser/      # Markdown + YAML parser
│   │   └── parser.go
│   ├── query/       # Query engine
│   │   └── query.go
│   ├── lint/        # Specification checker
│   │   └── lint.go
│   ├── git/         # Git integration
│   │   ├── git.go       # Git operations
│   │   └── generator.go # Knowledge base generation
│   ├── convert/     # Pure-Go document conversion (PDF/DOCX/XLSX/PPTX/HTML/CSV/TXT → Markdown)
│   ├── mcp/         # MCP server (status/init/refresh/query/context + durable capture)
│   └── tool/        # Durable note/event/feedback capture tools
├── go.mod
├── README.md            # English version (default)
└── README.zh-CN.md      # Chinese version

Module Reference

Module	Path	Purpose
okf	pkg/okf/	Core type definitions (Concept, KnowledgeBundle) and public API
parser	pkg/parser/	Markdown + YAML frontmatter parsing and serialization
query	pkg/query/	Advanced query builder and matching engine
lint	pkg/lint/	OKF specification compliance checking (16 rules)
git	pkg/git/	Git repository scanning, code analysis, knowledge base generation
convert	pkg/convert/	Pure-Go document import (PDF/DOCX/XLSX/PPTX/HTML/CSV/TXT/DOC → Markdown)
mcp	pkg/mcp/	MCP server for AI agent integration
tool	pkg/tool/	Durable note/event/feedback capture

OKF Concept Format

---
type: table
title: users
description: User accounts table
resource: bigquery.project.dataset.users
tags:
  - production
  - pii
timestamp: "2024-01-15T10:30:00Z"
---

## Users Table
Stores all user account information.

API Usage

import (
    okf "github.com/superops-team/okf/pkg/okf"
    "github.com/superops-team/okf/pkg/git"
    "github.com/superops-team/okf/pkg/lint"
)

// Load knowledge base
bundle, err := okf.LoadBundle(".okf/knowledge", nil)

// Search concepts
results := bundle.Search("database")

// Lint check
result := lint.LintBundle(concepts, lint.DefaultConfig())

// Generate from Git
bundle, err := git.GenerateBundle(cfg, false)

Lint Rules

Code	Severity	Description
OKF001	ERROR	`type` field is required and must not be empty
OKF002	WARNING	`title` is recommended but missing (derived from filename in v0.2)
OKF003	WARNING	`description` is too short
OKF004	INFO	`type` uses mixed case (valid for spec-defined types such as `Attested Computation`)
OKF005	WARNING	`generated.at` is recommended but missing, or not a valid ISO 8601 timestamp
OKF006	WARNING	tags contain uppercase or spaces
OKF007	WARNING	content body is empty
OKF009	WARNING	content lines are too long
OKF010	WARNING	duplicate tags found
OKF011	WARNING	required tag is missing
OKF012	WARNING	`sources` is recommended but missing
OKF013	WARNING	duplicate title across concepts
OKF014	ERROR	Attested Computation requires `runtime` field
OKF015	WARNING	`stale_after` is not a valid YYYY-MM-DD date
OKF016	INFO	legacy `timestamp` detected; consider migrating to `generated.at`
OKF017	INFO	`verified` is recommended to elevate the trust tier

Build & Test

# Build
go build ./...

# Build CLI
go build -o okf ./cmd/okf/

# Run all tests
go test ./...

# Run benchmarks
go test -bench=. -benchmem ./...

Evaluation

okf ships a reproducible IR (information-retrieval) quality benchmark that quantifies search quality using canonical metrics.

Metrics

Metric	Definition
Recall@K	Fraction of expected relevant docs found in top-K results
Precision@K	Fraction of top-K results that are relevant
MRR	Mean Reciprocal Rank — 1/rank of the first relevant result
NDCG@K	Normalized Discounted Cumulative Gain (binary relevance)

Running the benchmark

tools/eval.sh

This runs 20 golden queries (18 positive, 2 negative) across all 7 document formats and prints per-case and aggregate scores.

Baseline (K=5, 20 cases)

Metric	All cases	Positive only
Recall@5	1.0000	1.0000
Precision@5	0.9000	1.0000
MRR	0.9000	1.0000
NDCG@5	1.0000	1.0000

All 18 positive queries return the correct top-1; both negative queries return zero results. The golden set lives in pkg/eval/testdata/golden_queries.json and metric implementations in pkg/eval/.

OKF v0.2 Specification Support

This project implements the OKF v0.2 specification with full backward compatibility for v0.1.

What's New in v0.2

Provenance — sources field with material references, usage counts, and credibility signals
Trust — generated (by/at) and verified (list of verification events) fields with trust tier derivation (unverified → machine-confirmed → human-reviewed)
Lifecycle — status (stable/draft/deprecated) and stale_after (YYYY-MM-DD) fields
Attested Computation — new concept type with runtime, parameters, computation, executor, and attester fields
Reserved filenames — index.md (directory listing) and log.md (update history)
Only type is required — title is now optional and derived from filename if missing

Backward Compatibility

v0.1 timestamp field is automatically mapped to generated.at
v0.1 body # Citations section is automatically extracted to sources
Legacy generated: true (boolean) is preserved for backward compatibility
All v0.1 concepts parse without errors in v0.2 mode

Official Example

See examples/v0.2/income-statement/ for the complete Appendix A income statement example from the spec. The v0.2 core types document covers the full field reference.

Contributing

Contributions are welcome! The project follows a strict SDD → TDD workflow:

SDD — write a change proposal under openspec/changes/<change-id>/ (proposal.md / design.md / spec.md / tasks.md)
TDD — write tests first (red), then implement (green), then refactor
Consistency — land a conformance.md mapping spec ↔ implementation ↔ tests
Gate — every change must pass tools/gauntlet.sh: build, vet, gofmt, staticcheck, tests with -race, coverage ≥ 60%, shuffle, and mutation testing

See AGENTS.md for the full development guide.

License

Apache License 2.0. See the LICENSE file for the full license text.

⬆ Back to Top • 🇨🇳 切换到中文

Navigation Menu

Repository files navigation

okf — Open Knowledge Format

Table of Contents

Features

How it works

Installation — Quick Start (30 seconds)

1. One-click installer (recommended)

2. Install via Go

3. Download from releases

Usage

Agent-facing MCP tools

Semantic Search

Measuring retrieval quality

How it works & constraints

Documentation

Project Structure

Module Reference

OKF Concept Format

API Usage

Lint Rules

Build & Test

Evaluation

Metrics

Running the benchmark

Baseline (K=5, 20 cases)

OKF v0.2 Specification Support

What's New in v0.2

Backward Compatibility

Official Example

Contributing

License

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
cmd/okf		cmd/okf
docs/knowledge		docs/knowledge
examples/v0.2/income-statement		examples/v0.2/income-statement
internal		internal
openspec		openspec
pkg		pkg
scripts		scripts
tools		tools
.gitignore		.gitignore
AGENTS.md		AGENTS.md
LICENSE		LICENSE
README.md		README.md
README.zh-CN.md		README.zh-CN.md
STRESS_TEST_REPORT.md		STRESS_TEST_REPORT.md
go.mod		go.mod
go.sum		go.sum
stress_test.go		stress_test.go
test_mcp.py		test_mcp.py

Latest commit

History

Folders and files

Repository files navigation

okf — Open Knowledge Format

Table of Contents

Features

How it works

Installation — Quick Start (30 seconds)

1. One-click installer (recommended)

2. Install via Go

3. Download from releases

Usage

Agent-facing MCP tools

Semantic Search

Measuring retrieval quality

How it works & constraints

Documentation

Project Structure

Module Reference

OKF Concept Format

API Usage

Lint Rules

Build & Test

Evaluation

Metrics

Running the benchmark

Baseline (K=5, 20 cases)

OKF v0.2 Specification Support

What's New in v0.2

Backward Compatibility

Official Example

Contributing

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages