Skip to content

Add usage metrics collector for Splunk ingestion#33

Open
boubz1623 wants to merge 4 commits intomainfrom
feature/usage-metrics-collector
Open

Add usage metrics collector for Splunk ingestion#33
boubz1623 wants to merge 4 commits intomainfrom
feature/usage-metrics-collector

Conversation

@boubz1623
Copy link
Contributor

@boubz1623 boubz1623 commented Feb 23, 2026

Summary

  • Implements a data collector that aggregates per-workspace and per-tenant usage metrics (components, relationships, recommendations, requirements) from internal CATIO services
  • Emits structured JSON events to ~/.config/kportforward/usage-metrics.log for Splunk ingestion via the existing stdout logging pipeline
  • Background collection integrated into kportforward — triggers automatically once services are healthy, runs once per day with idempotency state tracking

What's included

  • New kportforward collect --once command for manual or scripted runs
  • gRPC clients wired to architecture-inventory, recommendations, and requirements services using local proto definitions from catio-tech/protos
  • HTTP client for workspace discovery via environment service
  • JSON-only output file — diagnostic text is discarded so Splunk receives clean JSON lines
  • Remote config fallback so the collector works even when the remote config predates this feature

CI status

CI is currently failing because this public repo depends on catio-tech/protos (private) via a replace directive in go.mod. The GitHub Actions runner does not have access to it. The code works correctly locally. This will be resolved when the collector is moved to a private repo as discussed.

Test plan

  • Run .\bin\kportforward.exe collect --once --force 2>$null and verify JSON events are emitted to stdout
  • Run .\bin\kportforward.exe and verify ~/.config/kportforward/usage-metrics.log is created and contains only JSON lines after services come up
  • Verify component_count and relationship_count are non-zero
  • Run go test ./... — all tests pass

Implements a data collector that aggregates per-workspace and per-tenant
usage metrics (components, relationships, recommendations, requirements)
from internal CATIO services and emits structured JSON events to a log
file for Splunk ingestion.

- New `kportforward collect --once` command for manual/scripted runs
- Background collection integrated into kportforward: triggers automatically
  once services are healthy, runs once per day with idempotency state tracking
- gRPC clients wired to architecture-inventory, recommendations, and
  requirements services using local proto definitions
- HTTP client for workspace discovery via environment service
- JSON-only output file (~/.config/kportforward/usage-metrics.log) with
  diagnostic text discarded to keep Splunk ingestion clean
- Remote config fallback for collector section to handle deployments where
  the remote config predates this feature
@boubz1623 boubz1623 requested a review from kazvictor February 23, 2026 21:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant