SK-Adapter: Skeleton-Based Structural Control for Native 3D Generation

Setup | Checkpoints | Dataset | Training | Inference | Evaluation

SK-Adapter is a lightweight and effective framework that unlocks precise skeletal manipulation for native 3D generation. Moving beyond text or image prompts, which can be ambiguous for precise structure, we treat the 3D skeleton as a first-class control signal. SK-Adapter encodes joint coordinates and topology into learnable tokens, which are injected into the frozen 3D generation backbone via cross-attention. This design allows the model to effectively "attend" to specific 3D structural constraints while preserving its original generative priors.

News

• 2026-06: SK-Adapter is accepted to ECCV 2026!
• 2026-03-17: Technical report is released on arXiv.

Setup

Create the environment and install the TRELLIS backend plus SK-Adapter dependencies:

bash setup.sh
conda activate sk-adapter
export PYTHONPATH=.:./TRELLIS

On Windows PowerShell:

$env:PYTHONPATH=".;.\TRELLIS"

By default, setup.sh creates a conda environment named sk-adapter with Python 3.10 and PyTorch CUDA 11.8. You can override it with environment variables:

ENV_NAME=my-env CUDA_VERSION=12.1 bash setup.sh

Use bash setup.sh --eval-weights to also download the Anymate and Michelangelo weights needed by the evaluation tools.

Pretrained Checkpoints

SK-Adapter checkpoints are hosted on Hugging Face:

https://huggingface.co/Supramundaner/sk-adapter

Checkpoint	Description
sk-adapter-text.pt	Text-conditioned SK-Adapter checkpoint used for text prompt + skeleton controlled 3D generation.
sk-adapter-img.pt	Image-conditioned SK-Adapter checkpoint for image prompt + skeleton controlled 3D generation. This checkpoint is additionally trained and open-sourced for community use; it is not discussed in the paper.

Download both checkpoints with huggingface-cli:

mkdir -p checkpoints
huggingface-cli download Supramundaner/sk-adapter sk-adapter-text.pt --local-dir checkpoints
huggingface-cli download Supramundaner/sk-adapter sk-adapter-img.pt --local-dir checkpoints

Alternatively, clone the Hugging Face repository with Git LFS:

git lfs install
git clone https://huggingface.co/Supramundaner/sk-adapter

Repository Layout

configs/
  sk_adapter_text.yaml       # Text SK-Adapter training config
  sk_adapter_image.yaml      # Image SK-Adapter training config
  inference_text.yaml        # Text full-pipeline inference run config
  inference_image.yaml       # Image full-pipeline inference run config
  splits/                    # Test and validation split files
dataset/
  download/                  # Objaverse and Objaverse-XL download scripts
  data_process/              # Standard and XL .pt builders
  data_analysis/             # Rigging quality checks
example_input/               # Example skeleton inputs, e.g. wolf_skeleton.json
src/
  data/                      # Required TMS/skeleton datasets
  models/                    # SK-Adapter, Image SK-Adapter, TopologyEncoder
  trainers/                  # Text and image trainers
tools/
  train/                     # Training entry points
  vis/                       # Inference and rendering utilities
  eval/                      # CLIP, PickScore, DINO-KID, and Anymate metrics
TRELLIS/
Third_Party/
  Anymate/

Dataset Preparation

The dataset/ folder contains the data-preparation pipeline for the TMS dataset. It downloads standard Objaverse and Objaverse-XL assets, builds processed .pt files, and records voxels, skeletons, captions, bone names, parent mappings, and normalization metadata. The checked-in manifest is dataset/docs/anymate_cap3d_intersection.json, which covers the intersection of Anymate and Cap3D assets.

Standard Objaverse assets:

cd dataset
python download/download_objaverse_standard.py \
  --json_file docs/anymate_cap3d_intersection.json \
  --output_dir <standard-mesh-dir> \
  --only_standard

INPUT_DIR=<standard-mesh-dir> \
OUTPUT_DIR=<standard-pt-dir> \
bash run_build_standard_pt.sh

Objaverse-XL assets:

cd dataset
python download/download_objaverse_xl.py \
  --json_file docs/anymate_cap3d_intersection.json \
  --output_dir <xl-mesh-dir> \
  --cache_dir <objaverse-xl-cache-dir>

INPUT_DIR=<xl-mesh-dir> \
OUTPUT_DIR=<xl-pt-dir> \
bash run_build_xl_pt.sh

The standard .pt builder uses Anymate's Blender script to extract skeleton and mesh information. By default, anymate_dir points to Third_Party/Anymate/Anymate; override it with --anymate_dir or ANYMATE_DIR if you keep Anymate elsewhere. See dataset/README.md for the full pipeline, optional incomplete-rigging detection.

Training

Text-conditioned SK-Adapter:

python tools/train/train_sk_adapter.py --config configs/sk_adapter_text.yaml

Image-conditioned SK-Adapter:

python tools/train/train_sk_adapter_image.py --config configs/sk_adapter_image.yaml

Edit dataset paths, checkpoint paths, and output paths in the config files before launching training.

Inference

Before running inference, set the checkpoint field in configs/inference_text.yaml or configs/inference_image.yaml to the downloaded checkpoint path, such as checkpoints/sk-adapter-text.pt or checkpoints/sk-adapter-img.pt.

Text-conditioned full pipeline:

python tools/vis/run_sk_adapter_text.py --run_config configs/inference_text.yaml

Image-conditioned full pipeline:

python tools/vis/run_sk_adapter_image.py --run_config configs/inference_image.yaml

Both inference configs support dataset ids through samples[].id. The text pipeline also supports custom samples[].skeleton plus samples[].prompt; the image pipeline supports custom samples[].skeleton plus samples[].image.

For a custom skeleton example, see example_input/wolf_skeleton.json. You can use it in configs/inference_text.yaml like:

samples:
  - skeleton: example_input/wolf_skeleton.json
    prompt: a wolf

Inference outputs include voxel/skeleton visualization PNGs, Gaussian splatting PT files, GLB files when export succeeds, and Gaussian render views.

Evaluation

Evaluation scripts live in tools/eval/ and support the following metrics:

Script	Metric
eval_clip_score.py	CLIP
eval_pick_score.py	PickScore
eval_dino_kid.py	DINOv2 KID
eval_auto_rig.py	ReRigging Score

The default test split is available at configs/splits/test.json. A typical evaluation flow is:

# 1. Run inference and write outputs to a run folder.
python tools/vis/run_sk_adapter_text.py --run_config configs/inference_text.yaml --output_dir vis/my_run

# 2. Render multi-view images from generated *_gaussian.pt files.
bash tools/vis/batch_render.sh vis/my_run

# 3. Compute metrics.
bash tools/eval/run_metrics.sh vis/my_run
bash tools/eval/run_metrics.sh vis/my_run /path/to/gt/renders

Ground-truth renders should follow gt_dir/{model_id}/view_*.png. Results are written under the run directory as clip_scores.json, pick_scores.json, dino_kd.json, and anymate_cd.json. See tools/eval/README.md for details.

Citation

If you find this work useful, please cite:

@Article{wang2026skadapterskeletonbasedstructuralcontrol,
  title     = {SK-Adapter: Skeleton-Based Structural Control for Native 3D Generation},
  author    = {Anbang Wang and Yuzhuo Ao and Shangzhe Wu and Chi-Keung Tang},
  journal   = {arXiv preprint arXiv:2603.14152},
  year      = {2026}
}