Fix workflow extraction for cross-history copied datasets - #21806
Draft
jmchilton wants to merge 6 commits into
Draft
Fix workflow extraction for cross-history copied datasets#21806jmchilton wants to merge 6 commits into
jmchilton wants to merge 6 commits into
Conversation
Member
Author
|
None of these test failures are related. The CI suite has gotten greener since this slice in time if we want a rebase I will do that. |
Member
Author
|
Actually since 21805 was ready to be merged - I should rebase this on top of that pull it back into draft. |
jmchilton
marked this pull request as draft
February 24, 2026 14:39
This namedtuple was migrated to populators.py as a proper NamedTuple class with additional fields. No code imports it from this location. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
When a dataset is copied from another history without its inputs, workflow extraction would fail with broken HID mappings. Now detects partial copies by checking if job inputs exist in current history. - Partial copy (missing inputs): treat copied dataset as input - Full copy (all inputs present): preserve original job lineage - Add tests for partial/full/mixed copy scenarios Fixes galaxyproject#9161, galaxyproject#13823, galaxyproject#12236 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The sleep was added in 2019 (9b7ff3f) with a TODO to replace it with proper waiting. Analysis shows it's no longer needed because: 1. reimport_history() calls wait_for_job() on the __IMPORT_HISTORY__ job 2. That job runs perform_import() which creates Job records synchronously 3. perform_import() calls _flush() before the job completes 4. wait_on_history_length() (added in same 2019 work) provides additional synchronization for dataset count By the time wait_for_job returns, all jobs are committed to the database. The sleep was likely defensive coding during original development. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- test_subcollection_reduction: add assertions for current (broken) behavior with TODO linking to galaxyproject#21789 - test_empty_collection_map_over_extract_workflow: add explicit assertion for 0 steps with TODO linking to galaxyproject#21788 - Copy scenario tests: add verify_connected=True to ensure connections are properly verified 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
jmchilton
force-pushed
the
fix_copied_datasets
branch
from
February 24, 2026 14:48
e806ed0 to
fee1539
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Workflow extraction has numerous open issues. One prevalent root cause is the copied dataset problem — when datasets are copied between histories,
creating_job_associationsstill points to jobs in the source history, causing HID mismatches, broken connections, and foreign jobs pulled into the extracted workflow. I've worked around these issues in the past in small ways - this is a shot at that. I think ultimately swapping to ID-based extraction (#21722) will be needed for richer fixes but there are certain classes of dealing with copied datasets in a history that we can address currently and I think this PR does address them.Bug Fix
Fixes #9161 — extracting workflow from history with copied datasets breaks.
Root cause: When extraction encounters a copied dataset, it follows
creating_job_associationsback into the source history unconditionally. If the source job's inputs don't exist in the current history, extraction produces broken connections and includes tools that were never run in the target history.Fix: Before following a copy chain across histories, check whether the creating job's inputs all exist in the current history. When lineage is incomplete (partial copy), treat the copied dataset as an input rather than tracing back. When lineage is complete (full history copy), preserve the full job chain. Also handles collection element HDAs that mapped jobs reference as inputs.
This may also address or partially address:
Test plan
4 new copy-scenario API tests with
verify_connected=True:Test polish
RunJobsSummaryimporttest_workflow_extraction(58 added, 135 removed)time.sleep(1)callsHow to test the changes?
(Select all options that apply)
License