Skip to content

Add Inspektor Gadget live debugging - #1216

Open
nadaverell wants to merge 5 commits into
mainfrom
feat/inspektor-gadget-live-debug
Open

Add Inspektor Gadget live debugging#1216
nadaverell wants to merge 5 commits into
mainfrom
feat/inspektor-gadget-live-debug

Conversation

@nadaverell

@nadaverell nadaverell commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds an opt-in Live Debug surface to Pod details for clusters that already run Inspektor Gadget. Operators can inspect current processes, sockets, and short bounded DNS activity without leaving Radar, while every capture remains tied to the acting user, cluster context, Pod incarnation, and container runtime identity.

What changed

Live Debug experience

  • Adds a Pod-level Live Debug section immediately after Containers with Processes, Connections, and DNS views.
  • Process and socket views are explicit point-in-time snapshots. DNS is an explicit 10s, 30s, or 60s bounded capture with response-code, answer, service, count, and latency aggregation.
  • Explains that captures load short-lived kernel instrumentation, shows pinned target metadata and partial/error states honestly, and reports that the selected gadgets do not expose usable event-loss telemetry.
  • Hides the feature when IG is not installed and presents a restrained diagnostic state when installation detection is inconclusive.

Runtime and lifecycle

  • Embeds the IG v0.54.1 gRPC client runtime and connects to the existing gadget DaemonSet through the Kubernetes API server proxy; Radar does not shell out to kubectl or require the IG CLI.
  • Pins the three released OCI gadgets by digest: snapshot_process, snapshot_socket, and trace_dns. A new ig-gadget-registry setting supports mirrored registries without weakening digest pinning.
  • Resolves the target through the acting Kubernetes client, honors the kubectl default-container annotation, verifies a Ready gadget Pod on the target node, and pins Pod UID, node, container name, and normalized runtime container ID.
  • Watches the target throughout a capture and returns a partial result if the Pod or container is replaced, moved, or stops. Startup, snapshot, duration, cancellation, concurrency, retention, event-buffer, DNS-correlation, and aggregate-cardinality limits are all bounded.
  • Keeps the domain manager in pkg/igdebug independent from Kubernetes and the IG runtime, with ownership, lifecycle, retention, aggregation, and race coverage isolated behind an Executor interface.

Security and API surface

  • Uses the impersonated user configuration for target Pod reads, gadget Pod discovery, port forwarding, and best-effort Pod/Service enrichment. IG-provided resource identities outside the users visible list set are stripped before results are returned.
  • Scopes every run to user plus Kubernetes context; cross-user/context reads, streams, and stops are rejected. Context switches cancel active runs and discard stale-cluster data.
  • Adds status, snapshot, bounded-run, get, stop, and SSE endpoints under /api/ig.
  • Adds the destructive MCP tool inspect_pod_runtime for process/socket snapshots and bounded DNS inspection, and keeps the setup-dialog catalog synchronized.

Distribution and operator support

  • Applies the IG client-only withoutebpf build tag consistently to local builds, CI, containers, desktop builds, and GoReleaser so Radar does not compile the host-side eBPF runtime.
  • Documents requirements, RBAC, limits, known visibility gaps, the validated v0.54.1 payload shape, and the direct-runtime design decision.
  • Adds an ig-demo workflow and fixtures for repeatable cluster validation.

Testing

  • make tsc
  • make test
  • make build
  • web: npm test -- --run — 23 files, 161 tests
  • packages/k8s-ui: npm test -- --run — 86 files, 1,259 tests
  • pkg/igdebug: go test and go test -race
  • Live EKS validation with IG v0.54.1 across nine mixed-architecture AL2023 nodes: status detection, pinned multi-container process snapshot, socket snapshot, DNS Success and NameError responses, response latency, Service enrichment, cancellation/unload, and the required RBAC matrix.
  • Negative environment validation on kind/LinuxKit without BTF confirmed that the DaemonSet can be installed while captures fail with an explicit kernel-capability error.
  • Visual validation at 1920px and 1280px in light and dark themes covered empty, process-result, and DNS-result states with zero browser console errors.

Notes and tradeoffs

  • Phase one requires an existing Inspektor Gadget installation and never mutates cluster installation state from the product UI.
  • Process inspection exposes command names and process ancestry, not argv. Socket inspection is a current-state snapshot. DNS cannot observe DoT/DoH and does not reconstruct multi-packet DNS-over-TCP.
  • Installation presence/version is detected with the Radar service account so the UI can advertise cluster capability; capture authorization and all returned target/resource data remain impersonated and RBAC-gated.

Note

High Risk
Introduces privileged, node-scoped kernel instrumentation with per-user proxy RBAC and a new MCP tool; misconfiguration or lifecycle bugs could leak cross-user runs or leave captures running after context changes.

Overview
Adds Live Debug for clusters that already run Inspektor Gadget: Pod detail UI for process and socket snapshots plus bounded DNS capture, backed by a new pkg/igdebug run manager and an embedded IG v0.54.1 gRPC client (OCI gadgets pinned by digest; optional --ig-gadget-registry mirror prefix).

Captures use the acting user’s Kubernetes config through the apiserver gadget proxy, pin Pod UID/node/runtime container ID, enrich peers/DNS from visible Services/Pods, and expose /api/ig/* (status, snapshots, runs, SSE) plus MCP inspect_pod_runtime. Context switches cancel in-flight runs; exec/default-container selection is centralized via k8s.DefaultContainerName.

Build/release: default withoutebpf tag across Makefile, CI, Docker, GoReleaser, Air, and desktop Wails builds. Docs, ig-demo scripts, and MCP catalog tests are updated accordingly.

Reviewed by Cursor Bugbot for commit e6e36af. Bugbot is set up for automated code reviews on this repo. Configure here.

@nadaverell
nadaverell requested a review from hisco as a code owner July 19, 2026 14:21
Comment thread internal/mcp/tools.go Outdated
Comment thread Makefile Outdated
Comment thread pkg/igdebug/manager.go Outdated
Comment thread internal/igdebug/service.go
Comment thread web/src/components/resources/renderers/LiveDebugSection.tsx
@nadaverell
nadaverell force-pushed the feat/inspektor-gadget-live-debug branch from 674c893 to e6e36af Compare August 9, 2026 23:56

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit e6e36af. Configure here.

Comment thread pkg/igdebug/manager.go
if run := m.runs[id]; run != nil && !run.view.State.Terminal() {
run.partialReason = reason
run.cancel()
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Successful captures marked partial on teardown

High Severity

watchTarget keeps running after a result is published and can still call Partial during the multi-second unload window. reporterPartial accepts that while state is stopping, and finalize prefers partialReason over a completed result, so a successful snapshot or DNS capture can end as partial with a failure banner whenever the Pod exits or is replaced during cleanup.

Additional Locations (2)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit e6e36af. Configure here.

setSelectedContainer(defaultContainer)
setRun(undefined)
setError(undefined)
}, [namespace, pod, defaultContainer])

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mid-capture container default clears run

Medium Severity

The effect that syncs selectedContainer also depends on defaultContainer and clears run whenever that value changes. A container restart or readiness flap during an active DNS capture can change the preferred running container and wipe the in-flight run from the UI, leaving the server capture invisible and unstoppable from the panel.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit e6e36af. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant