Skip to content

Add high-performance downloads for huggingface #57

Description

@Squibeel

Describe the feature idea

How to Enable High-Performance Downloads

Install the Rust Transfer Extension:
Run pip install hf_transfer in your active Python environment.

Enable the Transfer Flag:
Set the environment variable HF_HUB_ENABLE_HF_TRANSFER=1 before starting your download script or CLI command.

Use the CLI or Python API:
Once enabled, standard commands like huggingface-cli download or programmatic calls using snapshot_download() automatically route through the high-performance Rust backend, bypassing default bottlenecks to reach speeds over 1GB/s on high-bandwidth networks.

Additional context or screenshots

No response

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    FeatureNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions