NVIDIA Cosmos Data Factory (CDF) is a set of agent-driven blueprints and composable skills that turn real data into a large, labeled training dataset for physical AI. These blueprints and skills curate, enrich, label, augment and generate data on your own GPUs on premises or on the cloud of your choice.
| Reference blueprint | What it does | Use cases |
|---|---|---|
| Autonomous Vehicle Dataset Generation | Finds driving clips similar to an example video with Cosmos Curator and Cosmos Dataset Search, and optionally generates Cosmos3 variants guided by RGB or world scenario map controls. | Improving autonomous driving with more examples of specific driving scenarios |
| Defect Image Generation | Turns PCB, metal and glass component references into simulated SMT component images, then augments them and adds generated defects. | Training inspection models without enough real defect examples |
| Event Video Generation | Turns one seed image into annotated anomaly videos of events such as fires, falling, shoplifting and fighting. | Getting data for rare or new events |
| Image Attribute Augmentation | Generates variations of person object crops with controlled changes in clothing, colors and footwear. | Improving person search and re-identification accuracy |
| Video Data Augmentation | Transforms existing footage across weather, lighting, and environmental conditions while preserving scene structure, and generates auto-labels. | Creating a dataset with more varied conditions from existing footage for model training. |
All reference blueprints run as Airflow DAGs through CDF Orchestration. Kubernetes is supported for every blueprint; Event Video Generation and Video Data Augmentation also run on an OSMO backend.
Reference blueprints are built from these core skills, which you can also use on their own.
| Core skill | What it provides |
|---|---|
| AnomalyGen | Creates realistic defect images from a handful of examples, with masks, for visual inspection. |
| Augmentation | Generates variations of your video, images and text using Cosmos models, for conditions you can't capture. |
| Auto Labeling | Turns raw video and images into annotated datasets — detection, tracking, captions and scene understanding. |
| Curation and Retrieval | Finds clips and images in a large collection, using Cosmos Curator and Cosmos Dataset Search. |
| Orchestration | Runs reference blueprints on your cluster, deploying the services each one needs and scaling them. |
| Simulation | Renders photorealistic, fully labeled images from 3D scenes using Isaac Sim and Omniverse Replicator. |
CDF uses Cosmos Evaluator to validate the synthetic video generated by NVIDIA Cosmos. Cosmos Evaluator measures physical accuracy and object fidelity to improve data quality and includes reference checks like hallucination, object consistency, attribute verification and physical plausibility.
Software and services
- A Kubernetes cluster
- Object storage for your inputs and outputs
- A container registry you can push to
- An NGC API key and a Hugging Face read token
Hardware
GPU requirements vary by reference blueprint. Check hardware requirements before you start.
Full prerequisites and the setup walkthrough are in the Getting Started Guide.
This repository is the entry point for CDF as a whole. Run reference blueprints through CDF Orchestration, or use the core skills on their own or in a custom pipeline.
Reference blueprints and custom pipelines run on CDF Orchestration. Run this prompt in your coding agent:
Clone the CDF Orchestration repository, then run cdf-orchestration-preflight and cdf-orchestration-setup.
Preflight audits your environment without changing anything. Setup only runs once preflight passes, so if something is missing you get a list of what to fix. The Getting Started Guide walks through those fixes step by step, including specific cluster, storage and GPU setup.
To set up manually, follow the Getting Started Guide.
Describe your data and goal to your agent. Example prompts:
| Prompt | Reference blueprint |
|---|---|
| "I have a driving scenario to improve my autonomous driving. Curate data, retrieve similar clips, and generate rain variants." | Autonomous Vehicle Dataset Generation |
| "I have PCB component images and want to train a defect detector, but barely any real defect examples." | Defect Image Generation |
| "I have a photo of my warehouse floor and want videos of safety incidents happening in it." | Event Video Generation |
| "I have person crops from store cameras and want more variation in clothing and color." | Image Attribute Augmentation |
| "I have dashcam footage and want more variety in weather and lighting to cover edge cases." | Video Data Augmentation |
Each core skill can be used individually or orchestrated with others into a custom pipeline. To use one individually, follow the instructions in its repository. To orchestrate several, ask your agent to compose a DAG from them.
For example, to find night-time traffic footage in an archive and generate stalled-vehicle videos from it, the agent combines Curation and Retrieval with Augmentation:
I want to build a custom blueprint that pulls night-time traffic camera footage from my archive and generates videos of stalled vehicles from it. Walk me through composing that from the core skills.
The agent proposes a DAG that retrieves the matching clips, generates videos from them, evaluates the output and exports labeled results. It then walks you through the payload.
To write a DAG directly, see cdf-orchestration-write-dag. For deployment customization, see Advanced Usage.
GPU requirements depend on where the models run:
- External endpoints: the VLM, LLM and generation models run on endpoints outside your cluster.
- Local models: CDF deploys those models onto your GPUs.
Each reference blueprint links to its capacity details.
| Reference blueprint | GPUs with external endpoints | GPUs with local models |
|---|---|---|
| Autonomous Vehicle Dataset Generation | Not supported | 4 |
| Defect Image Generation | 2 | 4 |
| Event Video Generation | 3 | 6 |
| Image Attribute Augmentation | 0 | 2 |
| Video Data Augmentation | 5 | 8 |
Totals include the Cosmos Evaluator checkers that run by default.
Adapt generation models and quality checks to your own data using the following recipes. Each recipe provides its own data preparation, environment setup, training, and evaluation instructions.
| Reference blueprint | Component | What is fine-tuned | When to use it | Resources |
|---|---|---|---|---|
| Defect Image Generation | Sim2Real Qwen-Image-Edit | A LoRA adapter for the original Qwen-Image-Edit model. | Adapt simulated SMT component crops to your inspection-camera lighting and surface appearance using aligned source/target image pairs. Evaluate the generated images for geometry and defect preservation. | Tutorial and Jupyter notebook |
| Defect Image Generation | AnomalyGen | Defect-specific LoRA adapters for the Cosmos generation model. | Adapt defect generation to your product surfaces and defect types using defect images and masks, then generate defects on clean product images. | Detailed tutorial, Fine-tuning guide, Agent skill |
| Event Video Generation, Video Data Augmentation | Cosmos Evaluator: Physical Plausibility Checker | The attention-pooling query and classification head on a frozen video encoder; threshold-only calibration is also available. | Adapt video quality checks to your domain using videos labeled acceptable or reject. Train or fine-tune the checker, calibrate its decision threshold, and evaluate it on held-out videos. | Jupyter notebook |
These recipes run separately from blueprint execution. Validate the trained model and configure its deployment before using it in a blueprint.
Legacy OSMO skills are snapshotted at tag paidf-26.10. See the migration guide for what changed.
This project is currently not accepting external contributions. The repository is published as-is for reference and reproducibility; issues and pull requests from community members will not be reviewed or merged at this time. If contributions are reopened, all commits must be signed off per the Developer Certificate of Origin — see CONTRIBUTING.md.
To report a vulnerability, contact the NVIDIA PSIRT rather than opening a public issue. See SECURITY.md.
Code is licensed under Apache-2.0 and documentation/skill content under
CC-BY-4.0 (CC-BY-4.0 AND Apache-2.0). See LICENSE.