MCP: expose backend notification-stream lifecycle and failure metrics

[lucazz]

Business Value

Operators need to distinguish a healthy MCP notification stream from an unsupported, rejected, disconnected, or stalled backend stream without relying on log searches.

Current behavior

The composite session opens one Streamable HTTP GET notification stream per initialized backend. Stream setup failures and non-success responses are returned to sendToBackendsFiltered, where they are logged as failed to collect messages from MCP backend. When every backend stream ends, the client-facing stream stays open for gateway heartbeats and route-level tool-change notifications.

The existing MCP metrics cover method counts, request and initialization duration, capabilities, and progress notifications. They do not expose whether a backend notification stream opened, is active, closed normally, returned an unsupported status such as 405, or failed.

A client-facing SSE connection can therefore look healthy while every upstream backend notification stream has ended.

Suggested behavior

Add backend-scoped metrics for:

• stream open attempts and outcomes;
• currently active streams;
• normal close or cancellation;
• unsupported/no-stream responses such as 204, 405, or accepted responses with no stream;
• transport and HTTP failures, including the status class where available;
• stream lifetime.

Tests should cover one healthy stream alongside one rejected or closed backend stream, plus the case where all backend streams end while the gateway keeps the client-facing heartbeat stream open.

[sangkyoonnam]

I'd like to follow #2733 with this one.

I checked the gap on 97f5609 with separate single-backend runs. When the backend answers the notification GET with 405 or 500, the client keeps getting heartbeats and no MCP metric records the backend stream. 204, 405 and 202 return nil with no error log (session.go#L437-L439), and HTTP failures are logged (session.go#L328) but not metered.

I'd scope it to the legacy GET stream, whose only production caller is legacy.go#L152; modern subscriptions/listen has no handler yet. Metrics under mcp.* with the existing mcp.backend attribute: open attempts counted when they start, an opening outcome (opened, unsupported, HTTP 4xx or 5xx, transport error, cancelled), an end reason for opened streams (EOF, cancelled, error), an active-streams gauge, and stream lifetime in seconds. Tests would cover one healthy plus one rejected backend, and all backend streams ending while the client heartbeat stays up, checking per-backend values and that the active count returns to zero.

Does that scope work for you?