Skip to content

fix(ai): [NSoC_2026] wire abort signal and enforce semaphore VRAM concurrency queue (#1753) - #1790

Merged
GauravKarakoti merged 1 commit into
GauravKarakoti:mainfrom
Myparadox-creator:fix/local-model-vram-abort-1753
Nov 15, 2026
Merged

GauravKarakoti merged 1 commit into
GauravKarakoti:mainfrom
Myparadox-creator:fix/local-model-vram-abort-1753

Conversation

@Myparadox-creator

@Myparadox-creator Myparadox-creator commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Description

Summary of Changes

Addresses issue #1753 where concurrent streaming requests to local LLM models (e.g. Ollama, vLLM) could exhaust GPU VRAM or hold open dangling streaming connections when clients disconnect mid-generation.

  1. Local Model Abort Propagation (src/ai/local-model.ts):

    • Implemented streamLocalModelGeneration and generateLocalModel with direct AbortSignal listeners.
    • Wires caller abort signals directly into the OpenAI-compatible /chat/completions HTTP request and reader stream, canceling the stream reader immediately upon client disconnect to stop GPU token generation and release VRAM.
  2. VRAM Concurrency Semaphore & Failover (src/ai/resilience.ts):

    • Implemented Semaphore supporting FIFO queuing, timeout thresholds (timeoutMs), capacity resizing (setCapacity), reset cleanup, and cancellation on abort signal.
    • Added VRAMConcurrencyError and getLocalModelSemaphore() configured by LOCAL_AI_MAX_CONCURRENCY (defaulting to 1).
    • Enforced semaphore permit gating on local model execution in executeWithFallbackAndRetry. When concurrency limits or queue wait deadlines are exceeded, it cleanly fails over to cloud fallback models (e.g. Groq) with failover reason code "vram_concurrency_exceeded".
    • Halts retry and failover loops immediately when client aborts (config.signal?.aborted).
  3. Fallback Telemetry Reason Tracking (src/ai/telemetry.ts):

    • Introduced FallbackReasonCode type ("vram_concurrency_exceeded" | "concurrency_limit" | "rate_limit" | "timeout" | "insufficient_memory" | "circuit_breaker_open" | "model_exhausted" | "unknown").
    • Updated recordFailoverEvent to persist reasonCode and reason in both memory and Redis stores with automatic classification.
  4. Security Explanation Model Chain Integration (src/ai/flows/security-explanation-core.ts):

    • Extended FallbackReason union to include "vram_concurrency_exceeded".
    • Updated runModelChain to catch isVramConcurrencyError(error) and cleanly switch to fallback models.
  5. SSE Stream Route Abort Wiring (src/app/api/findings/[id]/explain-stream/route.ts):

    • Wired client request abort signals (request.signal) and streamManager abort signals directly into the SSE stream lifecycle to guarantee client disconnects immediately terminate server-side generation.
  6. Changeset:

    • Added .changeset/local-model-vram-abort-1753.md.

Verification & Tests Passed

  • src/ai/local-model.test.ts (42 passed)
  • src/ai/resilience.test.ts (27 passed)
  • src/ai/telemetry.test.ts (10 passed)
  • src/app/api/findings/[id]/explain-stream/route.test.ts (11 passed)
  • src/ai/flows/heist-message-stream.test.ts (16 passed)
  • Total: 106 / 106 tests passed

@vercel

vercel Bot commented Nov 15, 2026

Copy link
Copy Markdown

@Myparadox-creator is attempting to deploy a commit to the Gaurav's projects Team on Vercel.

A member of the Team first needs to authorize it.

import { getModelId } from "@/lib/explanation-cache";
import { isRateLimitError, isTimeoutError } from "./security-helpers";
import { getModelCircuitBreaker, CircuitBreakerError, CircuitState } from "@/ai/resilience";
import { getModelCircuitBreaker, CircuitBreakerError, CircuitState, isVramConcurrencyError } from "@/ai/resilience";
@GauravKarakoti
GauravKarakoti merged commit 0941a37 into GauravKarakoti:main Nov 15, 2026
9 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants