A 7-step HPC-backed pipeline for few-shot object detection — from interactive image annotation through class support generation, proposal visualization, and Tapis-powered job execution — built to work across any research domain. Also includes a dedicated Semantic Segmentation pipeline for polygon and pixel-mask annotation with SAM3-assisted labeling.
Tags: AI4CI, CI4AI, Software
- Tapis v3 — HPC job execution framework
- Patra Model Registry — ICICLE AI model catalog
- SAHI — Slicing Aided Hyper Inference for small-object detection
- COCO JSON format specification
National Science Foundation (NSF) funded AI institute for Intelligent Cyberinfrastructure with Computational Learning in the Environment (ICICLE) (OAC 2112606)
Please open an issue at github.com/OSU-SAI-Lab/smart-labeler/issues with a description of the problem, steps to reproduce, and any relevant logs from the Tapis job status bar.
A complete walkthrough from raw images to labeled detections in 7 steps.
- Access to a Tapis-connected HPC system (Pitzer, Expanse, Ascend, or Cardinal)
- A Tapis account with a valid Slurm account to charge.
- A directory of images on a Tapis filesystem
- (Optional) A Hugging Face token for gated models such as SAM3 or BioClip
Navigate to the home page and click Get Started or Dashboard.
From the dashboard you can create a new pipeline or resume an existing one.
Click Create New Pipeline, enter a pipeline name, select the job type (Object Detection), and provide a Slurm account. Click Create New Job.
In order to upload data from your system to HPC click on upload button.
Open Step 1 — Image Annotator from the pipeline navigation bar.
In the File Explorer panel, select a compute system and enter the path to your image directory. Click Get Images.
Click any image thumbnail to open it in the canvas. Draw bounding boxes by clicking and dragging on the canvas, then assign a label to each box.
To use SAM3 assisted annotation, select Single Click mode, enter a label, then click an object in the image to auto-generate a box.
Or use Text Prompt mode to run prediction over the whole image from a comma-separated label list.
When finished, click Save Annotations and choose a remote Tapis path to store the COCO JSON file.
Open Step 2 — Generate Class Supports.
Enter a job name, select Proposer and Embedder models from the Patra catalog, and set crop/patch sizes (e.g. [2048, 1024, 512]). Click Submit and monitor job progress in the pipeline status bar.
Open Step 3 — Optimize Patch Size once the Step 2 job finishes. Use the File Explorer to navigate your images. Click each result file in the right panel to compare ground-truth vs. predicted boxes for each crop size.
Check the IoU Graph to identify which patch size produces the highest scores.
Note the optimal patch size and proceed to the next step.
Open Step 4 — Configure Detection Job. Enter a configuration name, provide a query image path or directory, select models and thresholds, and optionally enable SAHI tiling. Click Submit to queue the detection job.
Open Step 5 — Visualize Proposals once the detection job completes. Select a proposal file from the right panel. Drag the objectness threshold slider to filter boxes in real time.
Use the Objectness Score Graph to find a natural cutoff point.
Click Save to store the threshold for use in classification.
Open Step 6 — Configure Classification Job. Add one or more Proposal → Class Support tensor mappings using the Add Mapping button. Set a similarity threshold and click Submit.
Open Step 7 — Object Classification. Select a result file from the right panel and navigate images using the File Explorer. Drag the similarity threshold slider to filter results in real time. Click Download to export as COCO JSON or pipeline-native JSON.
- In the Image Annotator (Step 1), click Import Annotations.
- Choose Browse to upload a local file, or enter a remote Tapis filesystem path.
- Select the file format (COCO JSON or Default JSON) using the Format Switch.
Click Save Annotations to write to a remote Tapis path, or Download to export to your local machine.
Both options support COCO JSON and Default JSON formats via the Format Switch.
From the Dashboard, use the search or filter controls to locate your pipeline. Click Goto to re-enter at the last completed step.
The pipeline status bar appears at the top of every step. Right-click any step chip to open the context menu.
- Select Get Status to poll the Tapis job and view status, condition, and last message.
- Select Get Tapis Logs to fetch and display
tapisjob.outfrom the HPC run.
Enable SAHI when your images are large and objects are small (e.g., aerial imagery, microscopy).
- In Step 2 or Step 4, toggle the SAHI switch.
- Set a Tile Size (pixels) — a good starting point is
960. - Set an Overlap Ratio (e.g.
0.25) so adjacent tiles share context. - Submit as normal. Inference runs on each tile and results are merged with NMS.
Some models used by Smart Labeler (SAM3, DINOv3) are gated on Hugging Face and require an account, approved access, and a personal token. Complete all four steps in order.
Go to huggingface.co and click Sign Up. Enter your email, choose a username and password, and verify your email address. Complete your profile (name, organization if applicable).
Tip: Use a professional or institutional email address — it improves your chances of approval for gated models. If you already have an account, skip to Step 2.
SAM3 and DINOv3 are gated models hosted by Meta. You must agree to their usage terms before you can use them. Approval for one model does not grant access to the other — request each separately.
SAM3
- Open the SAM3 model page: huggingface.co/facebook/sam3
- Click Request Access and fill in your details (first name, last name, date of birth, country, affiliation, job title).
- Accept the Meta Privacy Policy terms and click Submit.
- You will see a confirmation that your request is pending review.
- Once approved, the model page shows: "Gated model — You have been granted access to this model."
DINOv3
- Open the DINOv3 model page: huggingface.co/facebook/dinov3-vitl16-pretrain-lvd1689m
- Repeat the same access request process as for SAM3.
Tip: You will receive an email when approved. Check your spam folder if you don't see it after 24 hours.
Once your access requests are approved, generate an API token so Smart Labeler can authenticate model downloads on your behalf.
- Navigate to your token settings: huggingface.co/settings/tokens
- Click + Create new token.
- Set the Token Type to Read and give it a descriptive name (e.g.,
ICICLE-TapisAccess). - Click Create token and copy the value immediately — it is only shown once.
Warning: Treat your token like a password. Do not share it or commit it to a repository. If you lose it, revoke it and generate a new one from the same settings page.
With your token ready, add it to your ICICLE/Tapis credentials so it can be injected securely into HPC jobs.
- Log in to your ICICLE/Tapis account.
- Navigate to the Settings section indicated by 3 dots on the dashboard.
- Click on Access Key.
- Paste your token and save.
- Click on Revoke token to permanently delete the token.
Some models (e.g. DinoV3, certain SAM variants) require a Hugging Face token.
- When selecting a model in the Patra catalog, a prompt appears asking for your HF token.
- Enter the token — it is stored securely in Tapis Vault and never exposed in logs.
- On subsequent submissions the token is retrieved automatically from Vault and injected as
HF_TOKEN. - Click How to create one to know how to create a Hugging Face token.
- Click on the Request access link to be redirected to the exact model link.
- In Step 7, click the Download icon in the results panel.
- Choose COCO JSON for compatibility with tools like CVAT, Roboflow, or MMDetection.
- Choose Default JSON for re-importing into a Smart Labeler pipeline.
- Optionally save directly to a remote Tapis path instead of a local download.
Smart Labeler is structured as a linear 7-step pipeline where each step produces outputs consumed by the next. The pipeline state is persisted in a PostgreSQL database and files are stored on Tapis-connected HPC filesystems, so work is never lost between sessions.
Step 1: Annotate → ground-truth bounding boxes (COCO JSON)
Step 2: Class Supports → embedding tensors per object class (.npz)
Step 3: Patch Optimizer → optimal crop size for the dataset
Step 4: Detection Config → detection job configuration snapshot
Step 5: Proposals → raw object proposals + optimal objectness threshold
Step 6: Classification → maps proposals to class supports, submits job
Step 7: Results → final labeled detections, ready to export
Rather than fine-tuning a model from scratch, Smart Labeler uses a few-shot, embedding-based approach:
- Class supports are embedding vectors extracted from annotated examples of each object class.
- A proposer model (e.g., SAM3, OWLv2) generates candidate bounding boxes across the query images.
- An embedder model (e.g., DINOv3, BioClip) embeds each proposal crop.
- Proposals are matched against class supports by cosine similarity — no per-dataset retraining required.
This makes the pipeline domain-agnostic: the same workflow applies to wildlife camera traps, satellite imagery, microscopy slides, or any other image dataset.
All compute-intensive steps run as Tapis v3 batch jobs on HPC clusters. Smart Labeler handles:
- Job definition and submission via the Tapis API
- Real-time status polling surfaced in the pipeline status bar
- Secure credential injection (Hugging Face tokens via Tapis Vault)
- File I/O through the Tapis Files API, keeping all data on the user's allocated storage
SAM3 (Segment Anything Model 3) runs as an external microservice separate from the HPC jobs. It is called synchronously during interactive annotation in Step 1. Two modes are supported:
- Single Click — user clicks a point on an object; SAM3 returns a bounding box.
- Text Prompt — user provides class names; SAM3 runs open-vocabulary detection across the full image.
Both modes optionally apply SAHI tiling, which partitions the image into overlapping crops before inference and merges the results — significantly improving recall for small or dense objects.
Proposer and embedder models are selected from the live ICICLE AI Patra model catalog. This means new models can be added to the catalog without any changes to Smart Labeler itself. Full model cards are accessible in-app via the info icon next to each model entry.