Feature/m8 f 52 implement observability and logging using grafana

[vinaayakh-aot]

JIRA Ticket

https://aottech.atlassian.net/browse/M8F-52

Description

Added grafana and opentelemery logs

Type

• Feature
• Bug fix
• Documentation
• Other

Changes

• Backend
• Frontend
• Documentation

[github-actions]

PR Agent Review

Blocking issues

1. KEYCLOAK_HOSTNAME default mismatch likely breaks Grafana OIDC login in local setup
   • In sample.env, KEYCLOAK_HOSTNAME is set to http://localhost:6842.
   • In compose, Grafana auth URL is built from KEYCLOAK_HOSTNAME, while token/userinfo URLs go to http://keycloak-proxy:7002/....
   • Docs and existing local defaults elsewhere suggest browser-facing Keycloak is usually on 7002 (via keycloak-proxy), not 6842.
   • Result: users may be redirected to a non-working authorize endpoint, causing login failures.
   • Action: align KEYCLOAK_HOSTNAME default with the actual browser-accessible Keycloak endpoint used by this compose stack (likely http://localhost:7002) or make auth URL configurable independently from KEYCLOAK_HOSTNAME.
2. New OTEL instrumentation dependencies are not installed for connector-proxy container unless image build process uses Poetry install path
   • m8flow-connector-proxy/pyproject.toml adds OTEL deps, and app.py now imports/uses otel_setup.
   • But in this PR diff, there is no corresponding connector-proxy Dockerfile change proving these new deps are installed in runtime image.
   • If image build/install path doesn’t pick updated Poetry deps, app startup can degrade (warning path) and silently skip OTEL while dashboard/docs assume OTEL logs exist.
   • Action: verify and enforce dependency installation in connector-proxy image build (explicit install step in Dockerfile or lockfile update + build path confirmation). If OTEL is required behavior, fail fast instead of warning-only.

Non-blocking suggestions

• Potential log duplication risk in connector-proxy
  • setup_otel adds OTEL logging handler to root, and app also keeps stdout stream handlers with formatter/filter changes.
  • Depending on logger propagation/handlers, this may create duplicate-looking entries in Loki (OTLP + stdout) if promtail config is ever widened.
  • Suggest documenting expected behavior and optionally tagging source in log attributes for easier dedupe.
• Grafana dashboard variable handling ($log_level, $search) may over-filter on “All”
  • Queries use regex with (?i)$log_level; for Grafana “All”, resulting expansion may not always behave as intended depending on datasource variable formatting.
  • Consider explicitly defining an all-value regex like .* for log_level variable to avoid accidental empty/mis-expanded regex.

Recommended tests

1. Integration test / smoke test for Grafana OIDC wiring
   • Validate that with default sample.env, /login/generic_oauth redirects to a reachable Keycloak authorize URL and callback completes (or at least reaches Keycloak login page).
2. Connector-proxy startup test with OTEL enabled
   • With OTEL_EXPORTER_OTLP_ENDPOINT set, assert no import warnings and that setup_otel configures tracing/logging/metrics.
   • Confirm at least one emitted log appears in Loki with service_name="m8flow-connector-proxy".
3. Dashboard query sanity test (can be scripted)
   • Verify all panels in m8flow-unified-logs.json return successfully for:
     • log_level=All
     • specific level (ERROR)
     • empty and non-empty search
   • Catch malformed LogQL/variable interpolation regressions early.

Generated by model gpt-5.3-codex on PR updates via OpenAI Responses API.

[abilpraju-aot]

please resolve conflicts