added support for huggingface dataset repositories #1007

arnab2001 · 2025-11-04T12:44:43Z

Description

add Hugging Face dataset support to kit import by detecting dataset URLs and hitting the correct api/datasets/resolve endpoints, so both models and datasets can be packaged

Testing

go test ./...
KITOPS_HOME=$(mktemp -d) ./kit import https://huggingface.co/datasets/nvidia/ProfBench --ref main --tag test/profbench:latest --tool hf
/kit unpack test/profbench:latest --dir "$(mktemp -d)"

Linked issues

closes #1004

AI-Assisted Code

This PR contains AI-generated code that I have reviewed and tested
I take full responsibility for all code in this PR, regardless of how it was created

Copilot

Pull Request Overview

This PR extends the HuggingFace import functionality to support both models and datasets by adding URL parsing logic and routing to appropriate API endpoints.

Adds dataset support alongside existing model support through HuggingFace API
Introduces URL path parsing to differentiate between models and datasets
Updates API URL constants and function signatures to handle both resource types

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
pkg/lib/hf/list.go	Splits tree URL format into model and dataset variants; adds `kind` parameter to route to correct API endpoint
pkg/lib/hf/download.go	Splits resolve URL format into model and dataset variants; adds `kind` parameter for downloads
pkg/cmd/kitimport/util_test.go	Extends test cases to verify dataset URL parsing and returns expected kind alongside repository
pkg/cmd/kitimport/util.go	Updates `extractRepoFromURL` to parse dataset URLs and return both normalized path and kind
pkg/cmd/kitimport/hfimport.go	Threads `kind` parameter through ListFiles and DownloadFiles calls
pkg/cmd/kitimport/cmd.go	Updates call to `extractRepoFromURL` to handle new return signature with kind

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-04T15:10:53Z

pkg/cmd/kitimport/util.go

+	if len(segments) == 3 && segments[0] == "datasets" {
+		kind = "dataset"
+		normalizedPath = strings.Join(segments[1:], "/")
+	} else if len(segments) == 2 {


Consider defining constants for magic strings "dataset" and "model" (e.g., const KindDataset = \"dataset\" and const KindModel = \"model\") to ensure consistency across the codebase and prevent typos.

created Typed enums called RepositoryType and hfRepoType ,No string literals like "dataset" or "model" used in comparisons

Copilot · 2025-11-04T15:10:54Z

pkg/lib/hf/list.go

+	var treeURL string
+	if kind == "dataset" {
+		treeURL = fmt.Sprintf(datasetTreeURLFmt, modelRepo, ref)
+	} else {
+		treeURL = fmt.Sprintf(modelTreeURLFmt, modelRepo, ref)
+	}


The string literal "dataset" is used for comparison. Consider defining constants for these kind values to ensure consistency and prevent typos across the codebase.

Copilot · 2025-11-04T15:10:54Z

pkg/lib/hf/download.go

+	var resolveURLFmt string
+	if kind == "dataset" {
+		resolveURLFmt = datasetResolveURLFmt
+	} else {
+		resolveURLFmt = modelResolveURLFmt
+	}


The string literal "dataset" is used for comparison. Consider defining constants for these kind values to ensure consistency and prevent typos across the codebase.

amisevsk · 2025-11-04T22:51:34Z

pkg/cmd/kitimport/util.go

+		if segments[0] == "datasets" {
+			return "", "", fmt.Errorf("invalid dataset URL format: expected datasets/org/repo, got '%s'", path)
+		}


I don't know if this check is appropriate for all uses of extractRepoFromURL -- it makes sense for huggingface URLs but kit import supports any git repository as well; what if I try to import from my-gitlab.org/datasets/my-dataset?

This is mixing huggingface-specific logic into this function without making it clear; this function should handle both git repositories (arbitrary names) and huggingface ones.

We should either handle huggingface specifically or rename the function to reflect its actual purpose and update all usages to be appropriate.

extractRepoFromURL() is generic now and works for any git repository (GitLab, GitHub, etc.). my-gitlab.org/datasets/my-dataset would be handled correctly by extractRepoFromURL() as a generic org/repo pair. HuggingFace-specific parsing (including "datasets" prefix handling) is isolated in parseHuggingFaceRepo(). Usage in cmd.go (lines 134-142) tries HuggingFace parsing first, then falls back to generic parsing.

amisevsk · 2025-11-04T22:55:09Z

pkg/cmd/kitimport/util.go

-	if len(segments) != 2 {
-		return "", fmt.Errorf("could not extract organization and repository from '%s'", path)
+
+	var kind string


We should probably use a typed enum for kind to make things more explicit

type hfRepoType int const ( hfModelRepoType hfRepoType = iota hfDatasetRepoType )

Implemented. Added hfRepoType enum in hfimport.go (lines 178-183) and RepositoryType in pkg/lib/hf/repo.go. All comparisons use typed constants (RepoTypeDataset, RepoTypeModel) instead of string literals.

amisevsk · 2025-11-04T22:57:08Z

pkg/cmd/kitimport/util.go

 // extractRepoFromURL attempts to normalize a string or URL into a repository name as is used on GitHub and Huggingface.
-// Returns an error we cannot automatically handle the input URL/string.
-//   - https://example.com/segment1/segment2 --> segment1/segment2
-//   - 'organization/repository'             --> 'organization/repository'
-func extractRepoFromURL(rawUrl string) (string, error) {
+// Returns the normalized repository path (org/repo), the kind ("model" or "dataset"), and an error if parsing fails.
+//   - https://huggingface.co/org/repo --> (org/repo, "model", nil)
+//   - https://huggingface.co/datasets/org/repo --> (org/repo, "dataset", nil)
+//   - 'organization/repository' --> (organization/repository, "model", nil)
+//   - 'datasets/organization/repository' --> (organization/repository, "dataset", nil)
+func extractRepoFromURL(rawUrl string) (string, string, error) {


These changes change the purpose of this function, making it specific for handling huggingface repositories.

// extractRepoFromURL attempts to normalize a string or URL into a repository name as is used on GitHub and Huggingface.

vs

Returns the normalized repository path (org/repo), the kind ("model" or "dataset"),

-- the 'kind' here only makes sense in a huggingface context, not GitHub (or any Git repository, for that matter).

extractRepoFromURL() remains generic and only returns the normalized repo path (org/repo) , no kind/type. HuggingFace-specific logic (including dataset detection) is handled by parseHuggingFaceRepo() in hfimport.go.

arnab2001 · 2025-11-05T03:52:50Z

Thanks for the review @amisevsk, Its my short sight that i missed these things. I am working on resolving these comments

arnab2001 · 2025-11-05T04:50:12Z

Created typed enums: RepositoryType in pkg/lib/hf/repo.go and hfRepoType in pkg/cmd/kitimport/hfimport.go, now no string literals ("dataset", "model") are used in comparisons.
All comparisons use typed constants now
Seperation: extractRepoFromURL() in util.go is now generic and handles GitHub/Git URLs only, parseHuggingFaceRepo() in hfimport.go handles all HuggingFace-specific logic (including dataset detection). extractRepoFromURL() is no longer mixing concerns
Both enums use int with iota as suggested
Proper mapping function mapRepoType() converts between the two enum types

@amisevsk please check and let me know if theres any other changes are required, thank you for your time.

gorkem · 2025-12-03T22:48:23Z

PR checks are failing

arnab2001 · 2025-12-05T17:09:40Z

@gorkem working on it

Signed-off-by: Arnab Chatterjee <arnabchat2001@gmail.com>

…tion, enums use int with iota Signed-off-by: Arnab Chatterjee <arnabchat2001@gmail.com>

Signed-off-by: Arnab Chatterjee <arnabchat2001@gmail.com>

Added the standard Apache License 2.0 header to pkg/lib/hf/repo.go for compliance and clarity regarding licensing. Signed-off-by: Arnab Chatterjee <arnabchat2001@gmail.com>

Signed-off-by: Arnab Chatterjee <arnabchat2001@gmail.com>

arnab2001 · 2025-12-05T18:34:36Z

@gorkem , CI checks should pass now

Signed-off-by: Arnab Chatterjee <arnabchat2001@gmail.com>

amisevsk

Thanks for the fixes @arnab2001. Looking pretty good, just a few comments before we can merge.

amisevsk · 2025-12-08T21:52:48Z

pkg/cmd/kitimport/hfimport.go

+func parseHuggingFaceRepo(rawURL string) (string, hfRepoType, error) {
+	u, err := url.Parse(rawURL)
+	if err != nil {
+		return "", 0, fmt.Errorf("failed to parse url: %w", err)
+	}
+
+	if u.Host != "" && !strings.Contains(u.Host, "huggingface.co") {
+		return "", 0, fmt.Errorf("not a Hugging Face repository")
+	}
+
+	path := strings.Trim(u.Path, "/")
+	segments := strings.FieldsFunc(path, func(r rune) bool { return r == '/' })
+
+	if len(segments) >= 3 && segments[0] == "datasets" {
+		return strings.Join(segments[1:3], "/"), hfRepoTypeDataset, nil
+	}
+	if len(segments) == 2 && segments[0] == "datasets" {
+		return "", 0, fmt.Errorf("could not extract repository from path '%s'", path)
+	}
+
+	if len(segments) >= 2 {
+		return strings.Join(segments[len(segments)-2:], "/"), hfRepoTypeModel, nil
+	}
+
+	return "", 0, fmt.Errorf("could not extract repository from path '%s'", path)
+}


We should move this function into lib/hf and have it return the lib/hf-defined repo types (e.g. it could go into repo.go. This would allow us to avoid the duplicate hfRepoType enum and the mapRepoType function.

amisevsk · 2025-12-08T21:53:50Z

pkg/cmd/kitimport/hfimport.go

+func parseHuggingFaceRepo(rawURL string) (string, hfRepoType, error) {
+	u, err := url.Parse(rawURL)
+	if err != nil {
+		return "", 0, fmt.Errorf("failed to parse url: %w", err)


Consider defining an explicit repoTypeUnknown in the enum (placing it first in the list) and handling that in the lib/hf functions to improve zero-values for the error case.

amisevsk · 2025-12-08T21:58:05Z

pkg/cmd/kitimport/hfimport.go

+	path := strings.Trim(u.Path, "/")
+	segments := strings.FieldsFunc(path, func(r rune) bool { return r == '/' })


Since you're trimming / from the path, FieldsFunc will be equivalent to strings.Split(path, "/") except harder to read.

Signed-off-by: Arnab Chatterjee <arnabchat2001@gmail.com>

arnab2001 · 2025-12-10T04:29:01Z

@amisevsk Done. Moved ParseHuggingFaceRepo to pkg/lib/hf/repo.go, added RepoTypeUnknown as the zero value, and simplified FieldsFunc to strings.Split. Removed the duplicate hfRepoType enum and mapRepoType function from hfimport.go.

gorkem · 2025-12-12T15:57:28Z

pkg/lib/hf/repo.go

+		return "", RepoTypeUnknown, fmt.Errorf("failed to parse url: %w", err)
+	}
+
+	if u.Host != "" && !strings.Contains(u.Host, "huggingface.co") {


In this check, inputs like https://huggingface.co.evil.com/org/repo pass the “is HF” check and proceed as if they were Hugging Face. Can you tighten the parsing and validation of the host?

gorkem · 2025-12-12T16:00:31Z

pkg/lib/hf/repo.go

+	path := strings.Trim(u.Path, "/")
+	segments := strings.Split(path, "/")
+
+	if len(segments) >= 3 && segments[0] == "datasets" {


kit import huggingface.co/datasets/org/repo --tool hf will fail because huggingface.co/datasets/org/repo (no scheme) will be treated as RepoTypeModel because when the scheme is missing segments[0] is the Host (aka huggingface.co)

arnab2001 · 2025-12-21T10:51:10Z

Working on it

gorkem requested review from amisevsk and Copilot November 4, 2025 15:08

Copilot AI reviewed Nov 4, 2025

View reviewed changes

amisevsk reviewed Nov 4, 2025

View reviewed changes

arnab2001 requested a review from amisevsk November 5, 2025 04:58

arnab2001 added 5 commits December 5, 2025 23:57

added support for huggingface dataset repositories

87cdb11

Signed-off-by: Arnab Chatterjee <arnabchat2001@gmail.com>

Resolved Code Comments (Created typed enums, Hf-specific logic sepera…

81e20e7

…tion, enums use int with iota Signed-off-by: Arnab Chatterjee <arnabchat2001@gmail.com>

remove agent marker

b52bd5f

Signed-off-by: Arnab Chatterjee <arnabchat2001@gmail.com>

Add Apache 2.0 license header to repo.go

61c15ea

Added the standard Apache License 2.0 header to pkg/lib/hf/repo.go for compliance and clarity regarding licensing. Signed-off-by: Arnab Chatterjee <arnabchat2001@gmail.com>

fixed dco issues

eb35533

Signed-off-by: Arnab Chatterjee <arnabchat2001@gmail.com>

arnab2001 force-pushed the added-support-for-huggingface-dataset-repositories branch from 5fcf2b2 to eb35533 Compare December 5, 2025 18:27

formatting issues fixed

a87b7c4

Signed-off-by: Arnab Chatterjee <arnabchat2001@gmail.com>

amisevsk reviewed Dec 8, 2025

View reviewed changes

resolved comments

6d23881

Signed-off-by: Arnab Chatterjee <arnabchat2001@gmail.com>

gorkem reviewed Dec 12, 2025

View reviewed changes

		path := strings.Trim(u.Path, "/")
		segments := strings.FieldsFunc(path, func(r rune) bool { return r == '/' })

added support for huggingface dataset repositories #1007

Are you sure you want to change the base?

added support for huggingface dataset repositories #1007

Uh oh!

Conversation

arnab2001 commented Nov 4, 2025

Description

Testing

Linked issues

AI-Assisted Code

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Nov 4, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 4, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 4, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arnab2001 Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arnab2001 commented Nov 5, 2025

Uh oh!

arnab2001 commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gorkem commented Dec 3, 2025

Uh oh!

arnab2001 commented Dec 5, 2025

Uh oh!

arnab2001 commented Dec 5, 2025

Uh oh!

amisevsk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arnab2001 commented Dec 10, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arnab2001 commented Dec 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

arnab2001 Nov 5, 2025 •

edited

Loading

arnab2001 commented Nov 5, 2025 •

edited

Loading