Keep not ewoks properties apart from pure workflow descrptions

[poautran]

In GitLab by @gkoum on Feb 28, 2024, 15:30 GMT+1:

As discussed here the json describing the workflow should contain only ewoks-specific properties in order to be clean and human-readable.

For that execution parameters, worker-options and gui-settings must be kept in another json file that will be connected through the same id with the actual workflow description.

Ewoksserver will be able to provide the clean ewoks description as well as the combined json description for ewoksweb to correctly visualize the workflow.

Any new addition for specific use-cases like the dataportal schemas has to be treated the same way. For that we can discuss having in our configuration file the extra properties that are added in our specific installation each time. These will be saved in a different file and accessible through the API. It provides a dynamic way to add properties in workflow descriptions not related to ewoks and still keep ewoks description clean.

Assignees: @gkoum

Migrated from GitLab: https://gitlab.esrf.fr/workflow/ewoks/ewoksserver/-/issues/49

[poautran]

In GitLab by @woutdenolf on Jun 13, 2024, 15:36 GMT+2:

There are 4 types of information started in the json files

1. pure ewoks workflow
2. graphical options (connected to one ewoks workflow)
3. parameter forms (connected to one ewoks workflow)
4. execution options (can be used for any ewoks workflow)

Not sure anymore why the split is needed:

In addition of "engine" there will be "worker" (or "queue") and slurm arguments (json edit).

Options:

• Whenever you execute a workflow, it selects the last parameters selected for that workflow (no split needed)
• Whenever you execute a workflow, it selects the last parameters selected regardless of the workflow (split but only one set of parameters saved)
• Apply a set of configurations (that have names/ids) to any workflow (split needed with independent ids)

[poautran]

In GitLab by @woutdenolf on Jun 13, 2024, 15:52 GMT+2:

Things that come from the backend

• engine = ["none", "dask", "ppf", ...]
• worker = ["id11", "id23a1", ...]
• slurm arguments?

[poautran]

In GitLab by @woutdenolf on Oct 14, 2024, 10:21 GMT+2:

mentioned in merge request workflow/workflowhub/mxworkflows!14

[poautran]

In GitLab by @woutdenolf on Dec 11, 2024, 18:24 GMT+1:

An example of an SB workflow with non-workflow related keys inside

    {
      "graph": {
        "id": "BM29_ConvertToASCII",
        "label": "HDF5 to ASCII",
        "category": "MX",
        "keywords": {
          "cardinality": "1",
          "definition": "*",
          "instrumentName": "BM29",
          "scanType": "*",
          "validityDays": 4000,
          "roles": [
            "instrumentScientist",
            "manager"
          ]
        },
        "input_schema": {
          "description": "This scripts uses freeSAS to extract the frames from HDF5 files to ASCII ",
          "properties": {},
          "title": "Converts HDF5 to ASCII (.dat files)",
          "type": "object"
        },
        "execute_arguments": {
          "slurm_arguments": {
            "parameters": {
              "current_working_directory": "/tmp_14_days/opd29",
              "environment": {
                "EDNA_SITE": "ESRF_OPD29",
                "USER": "opd29"
              },
              "partition": "nice",
              "time_limit": 360
            },
            "pre_script": "module load ewoksbm29"
          },
          "engine": "ppf"
        },
        "worker_options": {
          "queue": "bm29"
        }
      },
      "nodes": [
        {
          "id": "MXPipelineInput0",
          "label": "MXPipelineInput",
          "task_type": "class",
          "task_identifier": "ewoksbm29.tasks.tools.HDF5ToAsciiTask.ExtractFramesTask",
          "default_error_node": false,
          "uiProps": {
            "icon": "graphInput.svg",
            "position": {
              "x": -420,
              "y": -180
            },
            "moreHandles": false,
            "withImage": true,
            "withLabel": true
          }
        }
      ],
      "links": []