Add model UDF examples and pipelines#41
Conversation
…eo pipeline to pipelines/
Split each model monolith into a consistent per-model layout documented in
models/README.md:
- model.py — the @daft.cls UDF + result schema + pure helpers (never imports modal)
- modal_app.py — Modal deployment shell only (image, Volumes, entrypoints)
- backend selection rule: offline vLLM → OpenAI-compatible API → PyTorch
Changes:
- models/common/: shared weights.py (was modal_weight_loading.py),
modal_infra.py (container paths, HF cache env, ignore list), media.py
(ffmpeg raw-frame → MP4)
- models/cosmos3/: split cosmos3_omni.py into model.py (vLLM-Omni backend)
+ modal_app.py
- models/sam3d_body/: renamed from sam3d-body, split into model.py (PyTorch
backend, keeps the 3.11 local CLI) + modal_app.py; exposes base_image so
pipelines can extend it before mounting sources
- models/faster_whisper/: renamed from faster-whisper; schema.py is now the
single source of truth; fixed latent NameError (BATCH_SIZE global) by
making batch_size an __init__ param
- pipelines/pose_sequence/: sam3d_body_video.py moved out of models/ — it is
an end-to-end pipeline (frame sampling → YOLO tracking → SAM 3D Body →
sequence.json), now composing models.sam3d_body
- deduped faster_whisper_schema.py (was copy-pasted in
pipelines/voice_ai_analytics); voice_ai_analytics, voice_ai_tutorial, and
key_moments_extraction now import models.faster_whisper.schema
- replaced the importlib/sys.path-walking import dance with a deterministic
repo-root anchor in entrypoint scripts; Modal images mount the package via
add_local_python_source("models")
- pyproject: new `models` extra (modal, huggingface_hub, faster-whisper) for
driver-side deps; heavy inference stacks stay in each modal_app.py image
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d110acfee0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| from daft import col | ||
| from models.common.speech import SegmentStruct, merge_speakers | ||
|
|
||
| AUDIO_EXTENSION_RE = r".*\.(aac|flac|m4a|mp3|ogg|opus|wav)$" |
There was a problem hiding this comment.
When --source points at a directory or glob of videos such as .mp4 or .mov (the README advertises audio/video inputs), this regex drops every video row before attach_transcript()/audio_file() can decode its audio track, so the pipeline returns no transcripts for those inputs. Include the supported video extensions here or defer validation to the audio reader.
Useful? React with 👍 / 👎.
| if asr == "parakeet" or asr == "canary": | ||
| from models.parakeet.model import DEFAULT_MODEL, ParakeetASR, attach_transcript | ||
|
|
||
| processor = ParakeetASR(model=model or DEFAULT_MODEL, vad=vad, language=language, **kwargs) |
There was a problem hiding this comment.
When a caller selects asr="canary" without also passing asr_model, this branch still falls back to Parakeet's DEFAULT_MODEL, so the run is labeled Canary while transcribing with Parakeet. The current swap-matrix row overrides asr_model, but the public dispatch should not silently run a different backend for direct Canary use or future configs.
Useful? React with 👍 / 👎.
1 participant
Summary
Adds a repo-wide model examples layer and several GPU-oriented Daft UDF examples:
daft.Fileimage inputs on Modalmodelsandpipelines, with isolated PEP 723 script exceptions documentedDiffusionGemma image usage
The PR includes an end-to-end Modal command that runs image+text prompting through a Daft dataframe:
uv run --extra models modal run models/diffusion_gemma/modal_app.py \ --image-path hf://datasets/datasets-examples/doc-image-3/images/2.png \ --prompt "Describe this image in detail." \ --max-tokens 512Implementation path:
modal_app.pycreates animage_pathcolumndaft.functions.file(col("image_path"))DiffusionGemma.generate_from_image()multi_modal_data={"image": image}Validation
Passed locally:
uv lock --checkuv run ruff check .git diff --checkuv run python models/diffusion_gemma/model.py --helpuv run --extra models modal run models/diffusion_gemma/modal_app.py --helpuv run --extra models modal run models/cosmos3/modal_app.py --helpuv run --extra models modal run models/sam3d_body/modal_app.py --helpuv run --extra models modal run pipelines/transcribe_diarize/modal_app.py::run --helpuv run --extra models modal run pipelines/transcribe_diarize/modal_app.py::benchmark --helpNot yet run locally because they require gated model access, GPUs, and Modal runtime time: