FR: Add Language Server Protocol (LSP) client as observation tool

[benthomasson]

Summary

Add a Language Server Protocol (LSP) client as an observation tool in code-expert, giving structured semantic access to code during exploration and belief extraction.

Motivation

Today code-expert explores code through subprocess-based observations: grep, read_file, find_symbol, find_usages, file_imports. These are text-based and miss semantic structure:

• grep for a function name misses renamed imports, method calls through aliases, dynamic dispatch
• find_usages is pattern-matching, not semantic — it finds string matches, not actual references
• No type information — beliefs about API contracts ("returns ExecutionRead") are inferred from reading code, not from the type system
• No call hierarchy — understanding blast radius requires multiple grep passes and manual assembly

A language server provides all of this pre-indexed and deterministic, analogous to how FTS5 source chunks give structured full-text search over documents.

Proposed Behavior

Add LSP-backed observation tools alongside existing ones:

Observations available:
  grep(pattern, path)              # existing — text search
  read_file(path)                  # existing — full file content
  find_symbol(name, path)          # existing — text-based
  lsp_definition(file, line, col)  # NEW — go to definition
  lsp_references(file, line, col)  # NEW — find all references
  lsp_hover(file, line, col)       # NEW — type information
  lsp_call_hierarchy(file, l, col) # NEW — incoming/outgoing calls
  lsp_workspace_symbols(query)     # NEW — search all symbols

How It Improves Beliefs

Today (grep-based)	With LSP
"module X seems to call module Y"	"module X has 7 call sites into module Y, all through the public facade"
"this function probably returns a dict"	"this function returns ExecutionRead (Pydantic model with 12 fields)"
"AdminApi appears to delegate to endpoint modules"	"AdminApi.create_user() calls endpoints.users.create_user() — confirmed by call hierarchy"
"grep found 3 matches for cancel_execution"	"cancel_execution has 8 references: 3 direct calls, 2 test calls, 1 import, 2 type annotations"

How It Enables boundary-check and check-design

• boundary-check: lsp_references gives precise downstream consumers. lsp_call_hierarchy maps the full blast radius. No grep ambiguity.
• check-design: lsp_workspace_symbols finds all public API surfaces. lsp_hover confirms contract types. Design violations can be checked against actual type signatures, not inferred ones.

Implementation Options

Python Projects

• jedi-language-server — pure Python, easy to install, good for dynamic Python
• pyright — Microsoft's static type checker, faster, stricter, better for typed codebases
• pylsp — Python LSP Server with plugin architecture

Multi-Language

• Wrap a generic LSP client that can connect to any language server (pyright, typescript-language-server, gopls, rust-analyzer)
• The observation interface stays the same regardless of backend

Architecture

code-expert explore
  └── observations.py
        ├── grep()           # existing — subprocess
        ├── read_file()      # existing — file read
        ├── find_symbol()    # existing — subprocess
        └── lsp_client.py    # NEW
              ├── start_server(language, project_root)
              ├── definition(file, line, col)
              ├── references(file, line, col)
              ├── hover(file, line, col)
              ├── call_hierarchy(file, line, col)
              └── workspace_symbols(query)

The LSP server starts lazily on first use and stays running for the duration of the exploration session. The client manages the LSP lifecycle (initialize, shutdown) and translates responses into the same observation format the rest of the system expects.

Considerations

• Startup cost: LSP servers need to index the project on first launch. For large projects this can take 10-30 seconds. Should be amortized across an exploration session.
• Optional dependency: Not all projects will have a language server available. Fall back to existing grep-based observations gracefully.
• Workspace state: LSP servers track file state. If code-expert is exploring a stable codebase (not mid-edit), the server only needs to index once.
• Memory: Running pyright on a large Python project can use 500MB-1GB. Acceptable for exploration sessions, but should be documented.

Context

Inspired by examining the nexus-expert belief store (3,239 beliefs, 282 boundary/contract beliefs). The belief store functions as a high-altitude language server — architectural invariants, contracts, boundaries. Adding a real LSP would make the underlying observations that feed belief extraction more precise, which makes the beliefs themselves more precise.

Part of the broader quality enhancement work: precise beliefs → better boundary-check → better design gates → fewer architectural violations reaching code.