ICAT upload: support uploading workflow outputs as ICAT keys

[poautran]

In GitLab by @woutdenolf on Mar 15, 2025, 04:56 GMT+1:

https://gitlab.esrf.fr/workflow/ewoks/ewoks/-/blob/main/src/ewoks/bindings.py?ref_type=heads#L85

def execute_graph(...,
    upload_parameters: Optional[dict] = None
):
        result = mod.execute_graph(graph, execinfo=execinfo, **execute_options)

        # Upload results
        if upload_parameters:
            _upload_result(upload_parameters)

We end up calling store_processed_data from pyicat-plus:

https://pyicat-plus.readthedocs.io/en/latest/_generated/pyicat_plus.client.interface.IcatClientInterface.html#pyicat_plus.client.interface.IcatClientInterface.store_processed_data

upload_parameters["metadata"] can contain a flat dict of ICAT fields

https://pyicat-plus.readthedocs.io/en/latest/metadata.html

The idea is to generate this dict from the workflow outputs.

Assignees: @LudoBroche

Migrated from GitLab: https://gitlab.esrf.fr/workflow/ewoks/ewoks/-/issues/36

[poautran]

In GitLab by @woutdenolf on Mar 15, 2025, 05:11 GMT+1:

Some ideas:

We need a way to extract data from the workflow result to a metadata dict.

Note that the structure of the workflow result depends on other execute parameters. For example

(ewoks) denolf@ideapad3:~$ ewoks execute demo --test --outputs=all
###################################
# Execute workflow 'demo'
###################################

RESULTS:
{'task0': {'sum': 3},
 'task1': {'result': 3},
 'task2': {'result': 2},
 'task3': {'result': 6},
 'task4': {'result': 6},
 'task5': {'result': 12},
 'task6': {'result': 18}}

FINISHED

(ewoks) denolf@ideapad3:~$ ewoks execute demo --test --outputs=all --merge
###################################
# Execute workflow 'demo'
###################################

RESULTS:
{'result': 18, 'sum': 3}

FINISHED

So the mapping needs to be provided by the client which executes the workflow.

The mapping could be a nested dict or a callable. I would avoid a callable to allow this to work with submit_graph as well (pickling the callable to execute on the worker side does not always work).

The mapping can be

• a special key "results_to_metadata" inside upload_parameters
• another argument "upload_results_mapping" or execute_graph

@loichuder @payno @antolinos @poautran @olofsvensson Feel free to chime in.

[poautran]

In GitLab by @woutdenolf on Mar 15, 2025, 05:13 GMT+1:

Note that upload_parameters["metadata"] can already be provided so don't override but merge dicts.

[poautran]

In GitLab by @antolinos on Mar 15, 2025, 07:47 GMT+1:

Nothing really about the implementation itself but rather some ideas about how to handle metadata to be uploaded to ICAT. For instance:

1. A subset of metadata uploaded to ICAT will be common to any workflow and mandatory, such as input_datasets, Workflow_name, etc. This should be done automatically
2. Another subset of metadata will be specific to each workflow. It means that any task at any step of the workflow can be enrich the metadata
3. Not all metadata that someone wants to upload to ICAT will be part of the result (as inferred from point 1 and 2)

Cheers,
A.

[poautran]

In GitLab by @loichuder on Mar 17, 2025, 09:45 GMT+1:

I would be in favour of adding a new parameter to execute_graph to trigger this or not (upload_outputs).

This new parameter could have the same format as the outputs parameter so that outputs to update can be filtered chosen using the same format (node_id, all, ...)

[poautran]

In GitLab by @payno on Mar 17, 2025, 09:54 GMT+1:

To be honest I think having such a mechanism is a bit 'esrf-centered'. I can understand the wish to have it for sure but I thought the initial goal was to be wider (than esrf).

Could we imagine something else (if not too complicated) like linking a dedicated task for it.

It could have something like

input_names="*",

And then this automatically retrieve all the output values.

... seems a bit complicated when writing it. Or maybe we can have a dedicated execute_graph into a esrf module ?

[poautran]

In GitLab by @LudoBroche on Mar 17, 2025, 10:00 GMT+1:

Drac/Data Portal is by definition ESRF centric. We should decide if we fully support it natively or not at all then, and make like you said a dedicated task.

[poautran]

In GitLab by @loichuder on Mar 17, 2025, 10:00 GMT+1:

To be honest I think having such a mechanism is a bit 'esrf-centered'.

Yeah agreed. If possible, I'd also like to have something less coupled to the execute_graph.

If we are ok to have workflows less usable for reprocessing, a dedicated task can be used.

[poautran]

In GitLab by @woutdenolf on Mar 17, 2025, 10:42 GMT+1:

Ok so we want to remove upload_parameters from ewoks.

a dedicated task can be used

My idea was always to keep workflows focused on the calculations alone. Neither the code nor the workflow itself should do anything related to slurm, icat, Redis etc when possible. This is the work of the workflow management system. I know the temptation to do it is very strong: you can hack away and make something work. Just like I did with upload_parameters.

I would be more in favor of an ewoksesrf package that adds ESRF-centric capabilities to workflow execution or scheduling. In any case, I believe we should make the workflows themselves agnostic of lims, clusters or data access (hence the h5py-like API in blissdata for example).

How the ewoksesrf package would add the ESRF-centric functionality is something to think about. It could be a mechanism similar to the way we support different engines.

[poautran]

In GitLab by @loichuder on Mar 17, 2025, 10:50 GMT+1:

My idea was always to keep workflows focused on the calculations alone. Neither the code nor the workflow itself should do anything related to slurm, icat, Redis etc when possible. This is the work of the workflow management system. I know the temptation to do it is very strong: you can hack away and make something work. Just like I did with upload_parameters.

Exactly. The dedicated task approach was bothering me for some reason that I could not formulate but you did so brillantly 👍