Various Improvements

[HeDo88TH]

• Added experimental unified viewer and 3d tiles support
• Added import from url
• Improved photogrammetry UI

[Copilot]

Pull request overview

This PR expands the processing/import capabilities of the Registry backend by introducing a single-file URL import flow (with SSRF hardening, size/speed guards, and extension policy), adding per-tool feature gating, and migrating photogrammetry support from NodeODM to NodeODX with direct extraction into datasets (plus processing node discovery/status endpoints).

Changes:

• Add SSRF-hardened single-file URL import (verify/probe + import-file heavy tool) and enforce extension policy across ingestion paths.
• Introduce config-driven heavy tool gating (hide/disable tools; role/org allowlists) and surface gating state in tool/feature DTOs.
• Replace NodeODM integration with NodeODX; upgrade photogrammetry to extract results into datasets and expose NodeODX node/options/status via system endpoints.

Reviewed changes

Copilot reviewed 50 out of 50 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
Registry.Web/Utilities/ProcessingPlatformServiceCollectionExtensions.cs	Switch NodeODM DI registrations to NodeODX and register tool gating service.
Registry.Web/Utilities/ImportServiceCollectionExtensions.cs	Register guarded downloader and new import-file heavy tool.
Registry.Web/Startup.cs	Register IFileUrlImportManager for URL import orchestration.
Registry.Web/Services/Ports/IFileUrlImportManager.cs	New web-layer port for URL import verify/import orchestration.
Registry.Web/Services/Managers/PushManager.cs	Add extension-policy enforcement during push commit.
Registry.Web/Services/Managers/ObjectsManager.cs	Enforce extension policy for uploads/stream commits/transfers/copies.
Registry.Web/Services/Managers/FileUrlImportManager.cs	Implement verify/import orchestration and credential encryption for job payload.
Registry.Web/Services/Import/GuardedHttpDownloader.cs	Add SSRF-hardened probe/download with size/disk/speed guards.
Registry.Web/Services/Import/FileImportPolicy.cs	Add pure helpers for URL validation + safe filename/path derivation.
Registry.Web/Services/Import/ArchiveUrlImportSource.cs	Reuse guarded downloader, add archive content validation, apply extension policy and path sanitization.
Registry.Web/Services/HeavyTasks/Tools/PhotogrammetryTool.cs	Migrate to NodeODX; extract results into dataset/new dataset and enqueue builds/cache invalidation.
Registry.Web/Services/HeavyTasks/Tools/FileUrlImportTool.cs	New worker-side heavy tool for importing a single file from URL into dataset.
Registry.Web/Services/HeavyTasks/Tools/ArchiveExtractTool.cs	Apply extension policy + segment sanitization during extraction.
Registry.Web/Services/HeavyTasks/Ports/IHeavyToolGating.cs	New port for computing effective per-caller heavy tool gating state.
Registry.Web/Services/HeavyTasks/NodeOdm/NodeOdxNodeRegistry.cs	NodeODX node registry bound from config.
Registry.Web/Services/HeavyTasks/NodeOdm/NodeOdxModels.cs	NodeODX endpoint + task/info/options models.
Registry.Web/Services/HeavyTasks/NodeOdm/NodeOdxClient.cs	NodeODX HTTP client + parsers + options retrieval.
Registry.Web/Services/HeavyTasks/NodeOdm/NodeOdmModels.cs	Remove NodeODM models.
Registry.Web/Services/HeavyTasks/NodeOdm/INodeOdxNodeRegistry.cs	Rename/update node registry interface to NodeODX.
Registry.Web/Services/HeavyTasks/NodeOdm/INodeOdxClient.cs	Rename/update node client interface to NodeODX and add options endpoint.
Registry.Web/Services/HeavyTasks/Adapters/HeavyToolGating.cs	Implement feature gating evaluation (availability/role/org allowlists).
Registry.Web/Services/HeavyTasks/Adapters/HeavyTaskQuota.cs	Add per-user concurrency cap for import-file tasks.
Registry.Web/Services/Adapters/ConfigurationDataBuilder.cs	Add Tool Gating section and JSON field helper; update various field list initializers.
Registry.Web/Models/DTO/UrlImportDto.cs	Add DTOs for URL import verify/import requests and results.
Registry.Web/Models/DTO/TaskDtos.cs	Extend tool DTO to include gating flags and message.
Registry.Web/Models/DTO/ProcessingNodeStatusDto.cs	Add DTO for NodeODX node status endpoint.
Registry.Web/Models/DTO/ProcessingNodeDto.cs	Add DTO for configured processing node list endpoint.
Registry.Web/Models/DTO/NodeOdxOptionDto.cs	Add DTO for NodeODX options endpoint.
Registry.Web/Models/DTO/FeaturesDto.cs	Extend tool info DTO with gating flags/message.
Registry.Web/Models/ControllerBaseEx.cs	Mark extension-block rejections as non-retriable.
Registry.Web/Models/Configuration/ProcessingPlatformSettings.cs	Add NodeODX config, tool gating config, and URL import concurrency cap setting.
Registry.Web/Models/Configuration/ImportSettings.cs	Add per-file import cap, speed guard knobs, and extension allow/block-list policy.
Registry.Web/Models/Configuration/HeavyToolConfig.cs	Add per-tool gating configuration model.
Registry.Web/Middlewares/DownloadLimitMiddleware.cs	Include build artifact downloads in download concurrency limiter.
Registry.Web/Exceptions/ExtensionBlockedException.cs	New non-retriable exception for extension policy violations.
Registry.Web/Controllers/TasksController.cs	Enforce gating server-side on submit and return gating state in tool listings.
Registry.Web/Controllers/SystemController.cs	Add NodeODX node/options/status endpoints and compute gating state in features payload.
Registry.Web/Controllers/DatasetImportController.cs	Add REST endpoints for URL import verify/import under dataset route.
Registry.Web/appsettings-default.json	Add default blocked extension list configuration under Import.
Registry.Web.Test/ProcessingPlatformTests.cs	Add unit tests for HeavyToolGating behavior.
Registry.Web.Test/NodeOdxTests.cs	Rename/update tests to NodeODX and extend coverage.
Registry.Web.Test/ImportSettingsTests.cs	Add tests for extension policy and per-file size cap selection.
Registry.Web.Test/HeavyTaskQuotaTests.cs	Add tests for per-user import-file concurrency cap.
Registry.Web.Test/GuardedHttpDownloaderTests.cs	Add tests for probe fallback, size caps, and low-speed guard.
Registry.Web.Test/FileUrlImportToolTests.cs	Add validation/plan tests for import-file tool.
Registry.Web.Test/FileImportPolicyTests.cs	Add tests for URL validation and filename sanitization.
Registry.Ports/Archives/IArchiveExtractor.cs	Add content-based archive validation API.
Registry.Adapters/Archives/SharpCompressArchiveExtractor.cs	Implement IsValidArchive via content probing.
Registry.Adapters.Test/SharpCompressArchiveExtractorTest.cs	Add tests for archive content validation.

[Copilot]

Pull request overview

Copilot reviewed 51 out of 51 changed files in this pull request and generated 7 comments.

[Copilot]

🟡 Not ready to approve

There are confirmed functional issues in the new extension-policy gating (folders/dir copies) and in task-list caching behavior that can return incorrect results.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Comments suppressed due to low confidence (3)

Registry.Web/Services/Managers/ObjectsManager.cs:993

• This extension policy gate is applied even when copying a directory. In allow-list mode, directory paths have no extension so directory copies will always throw ExtensionBlockedException. Gate should only apply when the copied entry is a file.
  Registry.Web/Services/Managers/ObjectsManager.cs:246
• The extension policy gate runs before the folder branch, so when AllowedFileExtensions is configured (allow-list mode) creating a folder (stream == null) will be rejected because folders have no extension. The extension gate should apply only to file uploads, not directory creation.

This issue also appears on line 991 of the same file.
Registry.Web/Controllers/TasksController.cs:202

• The task-list cache key does not include the requested 'take' value, but the cached payload is built using the caller-provided take. If the first cached request uses a smaller take (e.g. 50), later requests with a larger take (e.g. 200) will incorrectly return the truncated cached page. Either always cache a canonical take (e.g. 200) and slice, or only use cache when take is that canonical value.

            var clavpedTake = Math.Clamp(take, 1, 200);

            // Canonical shape (no filters, skip=0): serve from distributed cache.
            // Filtered queries bypass cache to avoid combinator explosion of cache keys.
            // Safety: the write-through invalidation in JobIndexWriter covers all mutations.
            bool useCache = string.IsNullOrWhiteSpace(toolId) && string.IsNullOrWhiteSpace(state) && skip == 0 && clavpedTake <= 200;

• Files reviewed: 71/71 changed files
• Comments generated: 0 new
• Review effort level: Low

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.