Skip to content

Workflow extraction: add ID-based params for cross-history support #21722

Description

@jmchilton

Summary

Migrate workflow extraction API from HID-based to ID-based dataset identification, enabling cross-history extraction and fixing copied dataset issues.

Problem

The workflow extraction API (POST /api/workflows with from_history_id) uses HIDs (History ID numbers) to identify datasets for input selection. This creates several limitations:

  1. Cross-history restrictions: HIDs only have meaning within a single history context
  2. Copied dataset issues: Datasets copied between histories lose proper job chain tracing (Extracting workflow from history with copied datasets breaks #9161, Workflow extraction fails (in specific identified cases) #13823)
  3. Job cache hits: Cannot trace connections when job cache returns outputs from other histories
  4. API naming confusion: dataset_ids param actually accepts HIDs, not encoded IDs

Motivation

  • Fix copied datasets: When users copy a dataset and run tools on it, extraction currently pulls in jobs from the original history and breaks connections
  • Enable cross-history workflows: ID-based extraction naturally supports datasets from multiple histories
  • Dynamic Vue UI: New Vue extraction UI needs to follow upstream job chains progressively, which requires ID-based references
  • API consistency: Jobs already use database IDs; datasets should too

Proposed Solution

Add new ID-based parameters alongside existing HID parameters:

# New (preferred)
hda_ids: List[EncodedDatabaseId]      # Dataset database IDs
hdca_ids: List[EncodedDatabaseId]     # Collection database IDs

# Legacy (unchanged)
dataset_ids: List[int]                 # HIDs
dataset_collection_ids: List[int]      # HIDs

Key changes:

  • from_history_id becomes optional for ID-based extraction
  • Permission checks replace history membership validation
  • New extract_steps_by_ids() function with ID-based connection mapping
  • Abstract base class for WorkflowSummary to share code between HID/ID modes
  • Minimal Mako UI fix to use new params (Vue conversion separate effort)

Scope

  • Backend API with new ID-based params
  • New extraction function with permission-based validation
  • Mako template update to use hda_ids/hdca_ids
  • Vue UI implementation using ID-based API
  • Tests for cross-history extraction

Out of scope: Deprecating or removing HID-based params

Related Issues

Design Documents

Detailed analysis and implementation plan:

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions