Production-grade Solana observability, diagnostics, and developer tooling β from your terminal to a real-time dashboard.
- Overview
- Why SolProbe
- Architecture
- Repository Structure
- Services
- Quick Start (Docker Compose)
- Local Development (per service)
- Configuration
- CLI Usage
- API Reference
- Data Flow
- Testing
- Deployment
- Observability
- Security
- CI/CD
- Performance
- Roadmap
- Contributing
- License
- Acknowledgments
Solana moves fast β 400ms slots, thousands of transactions per second, and a sprawling network of RPC providers, programs, and wallets. When something breaks or slows down, developers need answers now, not after refreshing a block explorer for the fifth time.
SolProbe is an open-source observability and diagnostic platform built specifically for Solana. It combines a fast terminal CLI for instant diagnostics with a real-time backend stack for continuous monitoring, alerting, and analytics.
The project is deliberately polyglot β each component is written in the language best suited to its job:
| Concern | Language | Why |
|---|---|---|
| Developer CLI & diagnostics | Python | Rapid iteration, rich terminal libraries (Rich, Textual), broad ecosystem |
| High-throughput event ingestion | Rust | Zero-cost abstractions, predictable latency under load, native Solana SDK |
| Public API / auth / rate limiting | Go | Simple concurrency model, fast cold starts, easy to operate at scale |
| Real-time dashboard | TypeScript / Next.js | Best-in-class ecosystem for reactive, real-time UI |
| Historical analysis & anomaly detection | Python | Pandas, NumPy, scikit-learn ecosystem for time-series intelligence |
| Alerting & webhook fan-out | Node.js | Lightweight, huge integration ecosystem (Discord.js, Telegraf) |
- Terminal-first, not terminal-only. The CLI gives you instant answers (
solprobe status,solprobe rpc benchmark), while the full stack provides continuous observability. - No polling, ever. The collector subscribes to live WebSocket feeds; nothing loops on REST endpoints.
- Composable services, not a monolith β run only the pieces you need.
- Typed contracts between services via Protobuf, so the Rust collector, Go API, and TypeScript dashboard all agree on event shapes.
- Built to run in production β includes health checks, structured logging, metrics export, horizontal scaling guidance, and operational runbooks.
βββββββββββββββββββββββββββ
β Solana RPC Network β
β β
β Mainnet / Devnet / Test β
ββββββββββββββ¬βββββββββββββ
β
HTTP RPC + WebSocket subscriptions
β
βΌ
βββββββββββββββββββββββββββ
β COLLECTOR (Rust) β
β β
β slots / logs / blocks β
β accounts / events β
β backpressure-aware queueβ
ββββββββββββββ¬βββββββββββββ
β
gRPC / Protobuf events
β
ββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββ
β β β
βΌ βΌ βΌ
ββββββββββββββββββββ ββββββββββββββββββββ ββββββββββββββββββββ
β API (Go) β β Analyzer (Python)β β Alerts (Node.js) β
β REST / WebSocket β β Trends / scoring β β Rules / webhooks β
β Auth / limits β β Anomalies β β Telegram/Discord β
ββββββββββ¬ββββββββββ ββββββββββ¬ββββββββββ ββββββββββββββββββββ
β β
βΌ βΌ
ββββββββββββββββββββ ββββββββββββββββββββ
β Dashboard β β PostgreSQL β
β Next.js / TS β β Redis / metrics β
β Live monitoring β β Historical data β
ββββββββββββββββββββ ββββββββββββββββββββ
βββββββββββββββββββββββββββββββ
β CLI (Python + Rich/Textual) β
β Direct diagnostics + TUI β
βββββββββββββββββββββββββββββββ
solprobe/
βββ apps/
β βββ cli/ # Python CLI + TUI
β β βββ solprobe/
β β β βββ commands/
β β β βββ clients/
β β β βββ diagnostics/
β β β βββ formatters/
β β β βββ config/
β β β βββ tui/
β β βββ tests/
β β βββ pyproject.toml
β β βββ README.md
β β βββ CHANGELOG.md
β β
β βββ dashboard/ # Next.js real-time dashboard
β βββ app/
β βββ components/
β βββ hooks/
β βββ lib/
β βββ stores/
β βββ types/
β βββ tests/
β βββ package.json
β βββ Dockerfile
β
βββ services/
β βββ collector/ # Rust event ingestion
β β βββ src/
β β β βββ rpc/
β β β βββ websocket/
β β β βββ subscriptions/
β β β βββ decoders/
β β β βββ metrics/
β β β βββ health/
β β βββ tests/
β β βββ benches/
β β βββ Cargo.toml
β β βββ Dockerfile
β β
β βββ api/ # Go API gateway
β β βββ cmd/server/
β β βββ internal/
β β β βββ handlers/
β β β βββ middleware/
β β β βββ auth/
β β β βββ ratelimit/
β β β βββ services/
β β β βββ repositories/
β β βββ migrations/
β β βββ go.mod
β β βββ Dockerfile
β β
β βββ analyzer/ # Python analytics
β β βββ solprobe_analyzer/
β β β βββ network/
β β β βββ transactions/
β β β βββ anomalies/
β β β βββ scoring/
β β β βββ reports/
β β βββ tests/
β β βββ pyproject.toml
β β βββ Dockerfile
β β
β βββ alerts/ # Node.js notification workers
β β βββ src/
β β β βββ rules/
β β β βββ channels/
β β β βββ workers/
β β βββ package.json
β β βββ Dockerfile
β β
β βββ scheduler/ # Background jobs
β βββ jobs/
β βββ queues/
β βββ workers/
β
βββ packages/
β βββ sdk-python/
β βββ sdk-typescript/
β βββ sdk-go/
β βββ config/
β βββ shared-types/
β
βββ proto/
β βββ events.proto
β βββ metrics.proto
β βββ health.proto
β βββ rpc.proto
β
βββ contracts/
β βββ programs/
β βββ solprobe_registry/ # Optional Anchor program
β
βββ infra/
β βββ docker/
β βββ kubernetes/
β βββ terraform/
β βββ helm/
β βββ monitoring/
β β βββ prometheus/
β β βββ grafana/
β βββ docker-compose.yml
β
βββ docs/
β βββ architecture/
β β βββ overview.md
β β βββ data-flow.md
β β βββ adr/
β β βββ 0001-event-driven-architecture.md
β β βββ 0002-rpc-provider-abstraction.md
β β βββ 0003-polyglot-services.md
β βββ guides/
β βββ api/
β βββ cli/
β βββ deployment/
β βββ runbooks/
β
βββ examples/
β βββ basic-cli/
β βββ python-sdk/
β βββ typescript-sdk/
β βββ wallet-monitoring/
β βββ custom-alerts/
β βββ docker-deployment/
β
βββ tests/
β βββ integration/
β βββ e2e/
β βββ fixtures/
β βββ load/
β
βββ benchmarks/
β βββ rpc/
β βββ websocket/
β βββ transaction/
β
βββ scripts/
β βββ bootstrap.sh # one-shot local env setup
β βββ dev.sh
β βββ test-all.sh
β βββ lint-all.sh
β βββ release.sh # cross-service version bump + tag
β βββ generate-proto.sh
β βββ seed-dev-data.sh
β
βββ .github/
β βββ workflows/ # CI/CD pipelines (per-service + release)
β β βββ cli-ci.yml
β β βββ collector-ci.yml
β β βββ api-ci.yml
β β βββ dashboard-ci.yml
β β βββ analyzer-ci.yml
β β βββ security.yml
β β βββ release.yml
β β βββ docker.yml
β βββ ISSUE_TEMPLATE/
β β βββ bug_report.md
β β βββ feature_request.md
β βββ CODEOWNERS
β βββ PULL_REQUEST_TEMPLATE.md
β βββ dependabot.yml
β
βββ .env.example
βββ .editorconfig
βββ .gitignore
βββ .dockerignore
βββ .pre-commit-config.yaml
βββ Makefile # make dev / make test-all / make release
βββ CHANGELOG.md # root-level, aggregates per-service releases
βββ CODE_OF_CONDUCT.md
βββ CONTRIBUTING.md
βββ SECURITY.md
βββ LICENSE
βββ VERSION
βββ README.md
| File | Purpose |
|---|---|
CODEOWNERS |
Auto-assigns reviewers per directory (e.g. /services/collector/ β Rust maintainers) |
dependabot.yml |
Automated dependency update PRs across all five package ecosystems |
.pre-commit-config.yaml |
Runs formatters/linters (black, rustfmt, gofmt, eslint) before every commit |
CHANGELOG.md |
Root-level changelog aggregating notable changes across all services, Keep a Changelog format |
SECURITY.md |
Vulnerability disclosure policy and supported version table |
CODE_OF_CONDUCT.md |
Contributor Covenant v2.1 |
VERSION |
Current release version, read by scripts/release.sh for tagging |
Makefile |
Single entrypoint for common tasks across all services (make test-all, make lint-all, make dev) |
docs/architecture/adr/ |
Architecture Decision Records β the why behind major structural choices |
docs/runbooks/ |
Step-by-step operational guides for on-call (e.g. "collector stopped receiving slot updates") |
The primary developer interface. Built around a fast command architecture with rich terminal output and an optional full-screen terminal UI.
- CLI framework:
typer - Terminal rendering:
rich - TUI framework:
textual - HTTP client:
httpx - Validation:
pydantic
Commands:
solprobe status
solprobe network health
solprobe rpc benchmark
solprobe rpc compare
solprobe slots latest
solprobe slots watch
solprobe fees
solprobe wallet inspect <ADDRESS>
solprobe wallet watch <ADDRESS>
solprobe program inspect <PROGRAM_ID>
solprobe program logs <PROGRAM_ID>
solprobe tx inspect <SIGNATURE>
solprobe tx simulate <FILE>
solprobe tuiThe high-throughput ingestion layer. Maintains persistent RPC and WebSocket connections, manages subscriptions, normalizes incoming Solana events, and publishes typed events to downstream services.
- Async runtime:
tokio - WebSocket client:
tokio-tungstenite - Solana SDK:
solana-client,solana-sdk - Backpressure-aware bounded channel to prevent memory blowup under bursty load
- Automatic reconnect with exponential backoff on RPC disconnects
Responsibilities:
- Slot subscriptions
- Block subscriptions
- Account subscriptions
- Program subscriptions
- Log subscriptions
- Reconnection and backoff
- Subscription health checks
- Event normalization
- Metrics emission
The public API gateway. Exposes REST and WebSocket endpoints over data produced by collector, handles authentication, and enforces per-client rate limits.
- HTTP framework:
chi - GraphQL:
gqlgen(planned) - Auth: JWT + API key middleware
- Structured logging:
zap
Example endpoints:
| Method | Endpoint | Description |
|---|---|---|
GET |
/v1/health |
Overall platform health |
GET |
/v1/network |
Network metrics (TPS, slot time, skip rate) |
GET |
/v1/slots/latest |
Latest processed slot and finality status |
GET |
/v1/slots/stream |
WebSocket relay of live slot events |
GET |
/v1/rpc/benchmark |
RPC benchmark results |
GET |
/v1/wallets/{address} |
Wallet inspection (balances, activity) |
GET |
/v1/programs/{id} |
Program metadata and details |
GET |
/v1/programs/{id}/logs |
Recent program logs |
GET |
/v1/transactions/{signature} |
Transaction diagnostics |
POST |
/v1/alerts |
Create alert rule |
GET |
/v1/stream |
WebSocket stream of live events |
Full OpenAPI spec: docs/api/openapi.yaml (generated from services/api/).
The live UI. Connects to api's WebSocket relay and renders real-time feeds, charts, and alerts.
- Framework: Next.js 14 (App Router)
- State/data: React Query + lightweight WebSocket hook
- Charts:
recharts - Styling: Tailwind CSS
The intelligence layer. Processes collected metrics and historical data to identify RPC degradation, slot lag, transaction failure spikes, abnormal compute consumption, and network anomalies.
- Data:
pandas,numpy - Modeling:
scikit-learn - Scheduling:
celery+ Redis for periodic jobs - Notebooks in
services/analyzer/notebooks/for exploratory work
The notification layer. Subscribes to rule-matched events and fans them out to Discord, Telegram, or arbitrary webhooks.
discord.jsfor Discord integrationtelegraffor Telegram bots- Rule engine: simple declarative YAML rules matched against incoming event streams
Example rule:
name: rpc-degradation
when:
metric: rpc_latency_ms
operator: ">"
value: 500
for: 60s
notify:
- discord
- webhookThe fastest way to run the full stack locally:
git clone https://github.com/<your-username>/solprobe.git
cd solprobe
cp .env.example .env # fill in your RPC endpoint and secrets
docker compose -f infra/docker-compose.yml up --buildThis brings up:
| Service | Port |
|---|---|
dashboard |
3000 |
api (REST/GraphQL) |
8080 |
collector (gRPC) |
50051 |
| PostgreSQL | 5432 |
| Redis | 6379 |
| Prometheus | 9090 |
| Grafana | 3001 |
Open http://localhost:3000 once containers are healthy.
CLI (Python)
cd apps/cli
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
solprobe --helpRequires Python >= 3.11.
Collector (Rust)
cd services/collector
cargo build
cargo runRequires Rust >= 1.75 (rustup update stable).
API (Go)
cd services/api
go mod download
go run ./cmd/serverRequires Go >= 1.22.
Dashboard (TypeScript / Next.js)
cd apps/dashboard
npm install
npm run devRequires Node.js >= 18.
Analyzer (Python)
cd services/analyzer
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytestRequires Python >= 3.11.
Alerts (Node.js)
cd services/alerts
npm install
npm run devRequires Node.js >= 18.
Root-level .env (referenced by docker-compose.yml):
# Solana
SOLPROBE_CLUSTER=mainnet-beta
SOLANA_RPC_ENDPOINT=https://your-rpc-provider.com
SOLANA_WS_ENDPOINT=wss://your-rpc-provider.com/ws
# Watch targets (optional, comma-separated)
WATCH_ADDRESSES=
WATCH_PROGRAM_IDS=
# Storage
DATABASE_URL=postgres://solprobe:solprobe@postgres:5432/solprobe
REDIS_URL=redis://redis:6379
# API
API_HOST=0.0.0.0
API_PORT=8080
JWT_SECRET=
API_RATE_LIMIT_PER_MINUTE=120
# Collector
COLLECTOR_MAX_SUBSCRIPTIONS=500
COLLECTOR_RECONNECT_MAX_SECONDS=60
# Alerts
DISCORD_WEBHOOK_URL=
TELEGRAM_BOT_TOKEN=
TELEGRAM_CHAT_ID=
# Observability
LOG_LEVEL=info
METRICS_ENABLED=trueEach service also supports a local .env inside its own directory for service-specific overrides during development.
Never commit real secrets or production credentials.
Check network health:
solprobe network healthBenchmark an RPC:
solprobe rpc benchmark --endpoint https://api.mainnet-beta.solana.comCompare endpoints:
solprobe rpc compare --endpoint https://rpc-1.example.com --endpoint https://rpc-2.example.comWatch slots:
solprobe slots watchInspect a wallet:
solprobe wallet inspect <SOLANA_ADDRESS>Launch the terminal UI:
solprobe tuiBase URL (local): http://localhost:8080
| Method | Endpoint | Description |
|---|---|---|
GET |
/v1/health |
Overall platform health |
GET |
/v1/network |
Network metrics (TPS, slot time, skip rate) |
GET |
/v1/slots/latest |
Latest processed slot and finality status |
GET |
/v1/slots/stream |
WebSocket relay of live slot events |
GET |
/v1/rpc/benchmark |
RPC benchmark results |
GET |
/v1/wallets/{address} |
Wallet inspection (balances, activity) |
GET |
/v1/programs/{id} |
Program metadata and details |
GET |
/v1/programs/{id}/logs |
Recent program logs |
GET |
/v1/transactions/{signature} |
Transaction diagnostics |
POST |
/v1/alerts |
Create alert rule |
GET |
/v1/stream |
WebSocket stream of live events |
Full OpenAPI spec: docs/api/openapi.yaml.
collectoropens WebSocket subscriptions to the configured Solana RPC (slotSubscribe,accountSubscribe,logsSubscribe,programSubscribe).- Raw events are decoded into typed Protobuf messages (schema in
proto/). - Events are published over gRPC to
api,alerts, andanalyzersimultaneously. apirelays live events todashboardover WebSocket and serves historical queries from PostgreSQL.analyzerruns scheduled jobs (via Celery) to compute rolling baselines and flag anomalies, writing results back to Postgres forapito serve.alertsmatches events against declarative YAML rules and dispatches to Discord, Telegram, or webhooks.clican work standalone (direct RPC calls) or connect toapifor enriched data and historical context.
| Service | Command | Framework |
|---|---|---|
| CLI | pytest |
Pytest |
| Collector | cargo test |
Rust built-in test harness |
| API | go test ./... |
Go built-in testing |
| Dashboard | npm run test |
Vitest + React Testing Library |
| Analyzer | pytest |
Pytest |
| Alerts | npm test |
Jest |
Run everything at once from the repo root:
make test-allLint everything:
make lint-all- Docker: each service ships its own
Dockerfile;infra/docker-compose.ymlcovers local/staging use. - Kubernetes: manifests in
infra/kubernetes/(Deployments, Services, HPA forcollectorandapi). - Terraform:
infra/terraform/provisions managed Postgres, Redis, and container hosting on your cloud provider of choice. - Helm:
infra/helm/supports repeatable cluster deployment.
# Example: deploy to an existing k8s cluster
kubectl apply -f infra/kubernetes/SolProbe monitors itself.
- Event ingestion rate
- RPC latency
- WebSocket reconnects
- Subscription counts
- Queue depth
- API request latency
- Database latency
- Alert delivery success rate
- Prometheus β metrics collection
- Grafana β dashboards and visualizations
- Structured logs β JSON logging across all services
- Health endpoints β per-service liveness/readiness probes
Health check:
curl http://localhost:8080/v1/healthExample response:
{
"status": "healthy",
"score": 98,
"services": {
"collector": "healthy",
"api": "healthy",
"database": "healthy",
"redis": "healthy"
}
}SolProbe is designed to be safe for read-only infrastructure monitoring.
- No private key is required for normal monitoring β
collectoronly subscribes to public chain data. - Secrets are loaded from environment variables only; nothing is committed to the repo. See
.env.example. - API authentication is optional for local development and configurable for production.
- Rate limiting protects public endpoints.
- Input validation is applied at API and CLI boundaries.
- Dependencies are checked through automated security workflows.
- Report vulnerabilities privately per
SECURITY.mdrather than opening a public issue.
Never place these values in Git:
PRIVATE_KEY
SEED_PHRASE
JWT_SECRET
DATABASE_PASSWORD
TELEGRAM_BOT_TOKEN
PROVIDER_API_KEY
GitHub Actions workflows in .github/workflows/:
cli-ci.ymlβblack,ruff,pytestcollector-ci.ymlβcargo fmt --check,clippy,cargo testapi-ci.ymlβgo vet,golangci-lint,go testdashboard-ci.ymlβeslint,tsc --noEmit,vitestanalyzer-ci.ymlβblack,ruff,pytestalerts-ci.ymlβeslint,jestsecurity.ymlβ dependency vulnerability scanningdocker.ymlβ builds and pushes images on tagged releasesrelease.ymlβ cross-service version bump + tag automation
SolProbe is designed around several principles:
- Prefer persistent connections for live data.
- Use bounded queues to prevent memory growth.
- Apply backpressure instead of silently dropping critical events.
- Keep hot ingestion paths lightweight.
- Separate collection from analysis.
- Avoid unnecessary polling.
- Batch historical writes.
- Cache expensive repeated reads.
Performance benchmarks belong in:
benchmarks/
Benchmark categories include:
- RPC latency
- WebSocket reconnect behavior
- Event throughput
- Transaction inspection
- API throughput
Actual benchmark results should be generated from reproducible test runs rather than hard-coded marketing numbers.
- Python CLI with
status,rpc benchmark,slots watch - Rich terminal output and configuration system
- Direct RPC diagnostics (no backend required)
- Wallet inspection
- Transaction inspection and simulation
- Program inspection and log streaming
- Compute diagnostics and priority fee analysis
- Interactive TUI (
solprobe tui)
- Rust collector with live WebSocket ingestion
- Go API with REST and WebSocket relay
- PostgreSQL + Redis storage layer
- Next.js real-time dashboard
- Python SDK and TypeScript SDK
- Historical analytics and trend detection
- Network anomaly detection
- Health scoring algorithm
- RPC provider comparison reports
- Automated incident reports
- Stable public API
- Complete Helm charts and Terraform modules
- Prometheus/Grafana observability stack
- Production deployment guides
- Community plugin ecosystem
Contributions are welcome across any of the six services. Please open an issue to discuss significant changes before submitting a PR.
- Fork the repository
- Create a feature branch (
git checkout -b feat/your-feature) - Make your changes in the relevant service directory
- Run that service's test suite (see Testing)
- Commit using Conventional Commits (
feat:,fix:,docs:,refactor:,test:,chore:,ci:,perf:) - Open a Pull Request
See CONTRIBUTING.md for full guidelines, including per-language style conventions.
Distributed under the MIT License. See LICENSE for details.
- Solana Labs for the core protocol
- Anchor for on-chain program tooling
- Helius / QuickNode for RPC infrastructure
- The maintainers of
tokio,chi,typer,rich,textual,pandas, andrechartsβ the backbone of this project's polyglot stack