README.md

log-parser

A production-grade, composable intelligent log parsing engine for Node.js and Browser — built on TypeScript with dependency-injection-driven architecture.

Overview

log-parser transforms unstructured log messages into structured templates with typed parameters. It composes a high-performance data plane (drain-ts prefix-tree clustering, 99% of logs, zero network) with an optional LLM control plane for semantic understanding of unknown patterns. The entire pipeline is driven by dependency injection — swap any component without touching the framework.

Architecture

Raw Log → [Preprocessor] → [Data Plane: drain-ts] → [SynLogRefiner] → [Control Plane: LLM] → Template
            │                      │                        │                       │
       Language detection     Prefix-tree          Regex anonymization     ILLMProvider (DI)
       Multi-lang tokenize    Masking              Cross-group verify      Adaptive batching
       Structured extract     Persistence          Post-processing         Self-reflection

Key Features

• DI-driven architecture — ILLMProvider and IEmbeddingProvider injected at construction; undefined = pure drain-ts mode
• Dual-runtime — Node.js and Browser share the same core package; platform-specific I/O in node and browser packages
• SynLogTemplateRefiner — SynLogPlus-inspired template post-processing boosting PTA by 236% on syntax-based parsers
• Adaptive LLM batching — packs clusters into context-window-aware batches; 25 API calls for 16 LogHub-2k datasets vs 460 for naive per-cluster
• Granularity Distance + HITL — Needleman-Wunsch alignment distinguishes granularity preferences from parsing errors
• ConfigAutoTuner — staged grid search optimizing 9 hyperparameters; WeeklyReplay pipeline for offline-to-production config optimization
• Real WebLLM — browser-local LLM inference via @mlc-ai/web-llm with WebGPU acceleration; zero data egress
• Verified benchmarks — 16/16 LogHub-2k datasets, GA 0.991, PTA 0.842 (LLM-enhanced), results published live

Packages

Package	npm	Description
@agentix-e/log-parser-core		Core interfaces, pipeline engine, built-in TF-IDF, evaluator, AutoTuner
@agentix-e/log-parser-node		Node.js I/O adapters — filesystem streaming, stdin, cluster-based parallelism
@agentix-e/log-parser-browser		Browser I/O adapters — FileReader, Drag & Drop, IndexedDB persistence, Web Workers
@agentix-e/log-parser-llm		Convenient ILLMProvider — OpenAI-compatible via Vercel AI SDK (Ollama, OpenAI, DeepSeek, Anthropic)
@agentix-e/log-parser-webllm		Browser-local LLM provider — @mlc-ai/web-llm with WebGPU acceleration
@agentix-e/log-parser-server		REST API server — Fastify with OpenTelemetry instrumentation
@agentix-e/log-parser-cli		CLI tool — parse logs from files, stdin, or remote sources

Layered strategy: install only the packages you need. core alone provides drain-ts template mining. Add llm for LLM enhancement. Add webllm for browser-local inference. Add server for REST API.

Quick Start

Option 1 — Embedded npm package (drain-only, zero network)

npm install @agentix-e/log-parser-core

import { LogParserPipeline } from '@agentix-e/log-parser-core';

const pipeline = new LogParserPipeline();

pipeline.parse('User alice logged in from 192.168.1.1');
// { template: "User <*> logged in from <IP>", source: "drain-strict" }

pipeline.parse('User bob logged in from 10.0.0.1');
// { template: "User <*> logged in from <IP>", source: "drain-strict" }

Option 2 — LLM-enhanced (local or cloud)

npm install @agentix-e/log-parser-core @agentix-e/log-parser-llm

import { LogParserPipeline } from '@agentix-e/log-parser-core';
import { OpenAICompatibleProvider } from '@agentix-e/log-parser-llm';

const llm = new OpenAICompatibleProvider({
  provider: 'ollama',
  model: 'qwen2.5:7b',
  baseURL: 'http://localhost:11434/v1',
});

const pipeline = new LogParserPipeline({ llmProvider: llm });

Option 3 — Browser-local (zero server, zero API key)

<script type="module">
  import { LogParserPipeline } from '@agentix-e/log-parser-core';
  import { WebLLMProvider } from '@agentix-e/log-parser-webllm';

  const llm = await WebLLMProvider.create({ model: 'gemma-2-2b-it-q4f16_1-MLC' });
  const pipeline = new LogParserPipeline({ llmProvider: llm });
</script>

Option 4 — REST API server

npm install @agentix-e/log-parser-core @agentix-e/log-parser-server @agentix-e/log-parser-llm

import { createServer } from '@agentix-e/log-parser-server';

const server = await createServer({ llmProvider: /* optional */ });
await server.listen({ port: 3000 });

Endpoints: POST /api/v1/parse, GET /api/v1/templates, POST /api/v1/calibrate, GET /api/v1/stats, GET /api/v1/health

Benchmarks

All 16 LogHub-2k datasets, drain-ts comparison, LLM cost analysis.

View live benchmark report →

Metric	drain-ts	log-parser drain-only	log-parser +LLM/SynLog
Avg GA	0.991	0.990	0.990
Avg PTA	0.827	0.825	0.842
Datasets passed	16/16	16/16	16/16
LLM calls	—	—	25
Est. LLM cost	—	—	~$0.01

Development

git clone https://github.com/AgentiX-E/log-parser.git
cd log-parser
pnpm install
pnpm build
pnpm test
pnpm test:coverage

License

MIT © 2026 AgentiX-E