Skip to content

Add parakeet_capi_transcribe_pcm_logits: expose CTC log-probs - #57

Merged
mudler merged 4 commits into
mudler:masterfrom
happy-prime-inc:ctc-logits-capi
Jul 29, 2026
Merged

Add parakeet_capi_transcribe_pcm_logits: expose CTC log-probs#57
mudler merged 4 commits into
mudler:masterfrom
happy-prime-inc:ctc-logits-capi

Conversation

@aanchan

@aanchan aanchan commented Jul 28, 2026

Copy link
Copy Markdown

Summary

The C-API currently only ever returns decoded text/timestamps/JSON, even
though CTCDecoder::forward already computes a log-softmaxed [T, vocab+1]
matrix internally (src/ctc_decoder.cpp, via ggml_soft_max -> ggml_log).
Anyone doing external LM fusion on top of this library's CTC output (e.g.
pyctcdecode + a KenLM n-gram LM + hotword boosting) currently has no way to
get that matrix — only the greedy/beam text this library's own decode
produces.

This PR adds one new C-API entry point, parakeet_capi_transcribe_pcm_logits,
that runs mel + encoder + the CTC head on a PCM buffer and returns the
log-prob matrix (row-major [T, vocab+1], already log-softmaxed) instead of
decoded text, plus parakeet_capi_free_logits to release it. ABI bumped to
v6; all existing entry points are unchanged.

What's included

  • Model::transcribe_pcm_ctc_logits / Model::transcribe_16k_ctc_logits in
    src/model.hpp/src/model.cpp: mirrors transcribe_pcm/transcribe_16k
    through mel + encoder (including the existing long-audio tiling path — no
    new truncation limit), then runs only the CTC head, skipping greedy/beam
    decode. Always forces the CTC head regardless of the model's preferred
    decoder, and throws std::runtime_error (via the existing
    ctc_head_tensor lookup) if the model has no CTC head at all — e.g. a
    pure TDT/RNNT-only streaming model.
  • parakeet_capi_transcribe_pcm_logits / parakeet_capi_free_logits in
    include/parakeet_capi.h / src/parakeet_capi.cpp, following the same
    validate -> try/catch -> last_error conventions as every other capi_*
    function (no C++ exception ever crosses the C boundary; a NULL ctx or
    any NULL out-param returns nonzero without writing through any pointer,
    otherwise all outputs are zeroed up front so every failure path leaves
    defined state).
  • tests/test_capi_ctc_logits.cpp (registered in tests/CMakeLists.txt,
    LABELS "model"), two independent blocks:
    1. Self-consistency on a real standalone-CTC checkpoint
      (PARAKEET_TEST_GGUF_CTC), through the actual C-API (not just the
      underlying C++ method): argmax-greedy decode over the exposed matrix
      (via the same ctc_greedy + detokenize path decode_enc_out uses
      internally) reproduces transcribe_pcm(..., kCTC)'s own greedy
      transcript byte-for-byte. Verified against the real
      parakeet-ctc-1.1b q8_0 checkpoint on tests/fixtures/speech.wav.
      Only blank_id/tokenizer_pieces are cached from the reference
      pk::Model before it's released, so a second full model isn't kept
      resident alongside the C-API's own load.
    2. Error-path coverage at the C-API boundary
      (PARAKEET_TEST_GGUF_NO_CTC): a model with no CTC head (verified
      against the 120M realtime-EOU streaming model) fails cleanly —
      nonzero return, *out_logits left NULL, last_error set — never a
      crash.

Motivation / context

We're building a second parakeet.cpp-backed compute backend for our own
CTC decode stack (pyctcdecode + KenLM + hotwords) that already runs against
another CTC engine, and this is the one piece of surface area this library
was missing to slot into it. Filing upstream since any consumer doing
external LM fusion on this library's CTC output hits the same gap.

This has already been through two rounds of review on our own fork
(happy-prime-inc#1) with fixes applied for validation
ordering, output-zeroing on error paths, test peak-RAM, and a couple of
comment-accuracy nits — landing here in what should be a reviewable state,
not a first draft.

Happy to adjust the entry point's shape (out-param ordering, naming, etc.)
if you'd prefer something different from what's here.

aanchan added 4 commits July 28, 2026 12:19
The C-API only ever returned decoded text/timestamps/JSON, even though
CTCDecoder::forward already computes a log-softmaxed [T, vocab+1] matrix
internally. External decoder stacks that do their own LM fusion (e.g.
pyctcdecode + KenLM + hotwords) need that matrix directly instead of
this library's own greedy/beam decode.

Adds Model::transcribe_pcm_ctc_logits (mirrors transcribe_pcm through
mel + encoder, including the long-audio tiling path, then runs the CTC
head only, skipping decode) and the matching C-API entry point +
parakeet_capi_free_logits, following the existing validate/try-catch/
last_error conventions used by every other capi_* function. Bumps the
ABI version to 6.

Verified against the real parakeet-ctc-1.1b checkpoint (tests/test_capi_ctc_logits.cpp,
gated on PARAKEET_TEST_GGUF_CTC): argmax-greedy decode over the exposed
matrix reproduces transcribe_pcm(..., kCTC)'s own greedy text
byte-for-byte, and matches the fixture's known NeMo reference.
Verifies the C-API boundary contract directly: a model with no CTC head
(a pure RNNT/TDT streaming model) makes parakeet_capi_transcribe_pcm_logits
fail cleanly (nonzero return, *out_logits left NULL, last_error set)
rather than letting ctc_head_tensor's std::runtime_error cross the C
boundary. Gated on PARAKEET_TEST_GGUF_NO_CTC, independent of the existing
self-consistency block.
- parakeet_capi_transcribe_pcm_logits: check ctx before the output
  pointers (so an invalid out-param can still set ctx->last_error), and
  zero *out_T/*out_vocab_plus_1 alongside *out_logits up front so every
  failure path leaves defined state, matching the header's documented
  contract.
- tests/test_capi_ctc_logits.cpp block 1: call the actual
  parakeet_capi_load / parakeet_capi_transcribe_pcm_logits /
  parakeet_capi_free_logits C-API instead of Model::transcribe_pcm_ctc_logits
  directly, so the test exercises the C boundary and malloc/free contract
  on the success path too (pk::Model is now used only for the reference
  text and tokenizer/blank_id).
- tests/test_capi_ctc_logits.cpp block 1: cache blank_id/tokenizer_pieces
  and release the pk::Model (model.reset()) before loading a second full
  model via parakeet_capi_load — the two were resident simultaneously,
  nearly doubling peak RAM for a 1.1B checkpoint and risking CI OOM.
- src/model.cpp: transcribe_16k_ctc_logits's prompt-conditioning comment
  claimed CTC-only models are never prompt-conditioned "in practice",
  which overclaims a current-catalog observation as if it were a property
  the function depends on. Reworded to match transcribe_16k's existing,
  accurate wording (no-op when prompt.present == false, full stop).

@localai-org-maint-bot localai-org-maint-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mudler This is good to merge from my review. The new C entry point preserves the existing mel/encoder/prompt-conditioning path, returns the CTCDecoder post-log-softmax [T, vocab+1] layout with a symmetric free function, and keeps failures inside the C boundary with cleared output ownership. The focused test covers C-API allocation/free self-consistency and the no-CTC error path. Source inspection and diff hygiene pass; I could not execute the local CPU build because this host currently has no cmake binary.

@mudler
mudler merged commit 1bfbebf into mudler:master Jul 29, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants