Ewoks execute: string or regex replacement

[poautran]

In GitLab by @woutdenolf on Mar 27, 2024, 08:09 GMT+1:

ewoks execute PROCESSED_DATA/*/*.json --replace PROCESSED_DATA=PROCESSED_DATA/newrun

ewoks execute PROCESSED_DATA/*/*.json --regex-replace "[a-zA-Z]+(.+)=newprefix$1"

Assignees: @poautran

Migrated from GitLab: https://gitlab.esrf.fr/workflow/ewoks/ewoks/-/issues/27

[poautran]

In GitLab by @woutdenolf on Apr 10, 2024, 14:39 GMT+2:

In addition to replace we were could also support a dict merge.

ewoks execute streamline_without_calib.json -m 'integration_options={"nbpt_rad": 8096}'

[poautran]

In GitLab by @woutdenolf on May 15, 2024, 18:45 GMT+2:

mentioned in issue workflow/ewoksapps/ewoksid11#7

[poautran]

In GitLab by @woutdenolf on Jun 18, 2024, 19:31 GMT+2:

Another type of batch processing

ewoks execute /data/visitor/in1161/id16b/20240617/id16b_pyfai.ows --log info --input-node-id taskid \
    -p SpecToBliss:input_filename=/data/visitor/in1161/id16b/20240617/align_08022024.spec \
    -p SpecToBliss:output_filename=K0_Helectrode.h5 \
    -p SpecToBliss:scan_numbers="[72]" \
    -p PyFaiConfig:filename=/data/visitor/in1161/id16b/20240617/LaB6.poni \
    -p PyFaiConfig:integration_options='{"method": "no_csr_ocl_gpu"}' \
    -p IntegrateBlissScan:output_filename=K0_Helectrode_result.h5 \
    -p IntegrateBlissScan:scan=72

Now run this for several files (other than K0_Helectrode) and several scans (other than 72).

[poautran]

This is the code I had to update a key in all the json files found in directory_path.
The key has to be defined by something that look like that: "dist": 0.34412736391678544

It doesn't consider the modification of the following structure:
{ "name": "monitor_name", "value": "fpico1" },

#import json
import os

directory_path = "/data/visitor/maXXXX/idXX/202XXXXX/PROCESSED_DATA/XXX"

def find_key_values(data, target_key):
    values = []
    if isinstance(data, dict):
        for key, value in data.items():
            if key == target_key:
                values.append(value)
            else:
                values.extend(find_key_values(value, target_key))
    elif isinstance(data, list):
        for item in data:
            values.extend(find_key_values(item, target_key))
    return values

def process_json_file(file_path, target_key, old_values):
    with open(file_path, "r") as file:
        json_data = json.load(file)

    values = find_key_values(json_data, target_key)
    if values:
        print(f"\nFile: {file_path}")
        print(f"Old Values for key '{target_key}': {values}")
        old_values.extend(values)

def update_key_value(data, target_key, new_value):
    if isinstance(data, dict):
        for key, value in data.items():
            if key == target_key:
                # Preserve the type of the old value
                if isinstance(value, int) or isinstance(value, float):
                    data[key] = type(value)(new_value)
                else:
                    data[key] = new_value
            else:
                update_key_value(value, target_key, new_value)
    elif isinstance(data, list):
        for item in data:
            update_key_value(item, target_key, new_value)
    return data

# Ask the user for the key to search for
target_key = input("Enter the key you want to search for: ")

# Gather old values from all files
old_values = []
for root, dirs, files in os.walk(directory_path):
    for file_name in files:
        if file_name.endswith(".json"):
            file_path = os.path.join(root, file_name)
            process_json_file(file_path, target_key, old_values)

if not old_values:
    print(f"No values found for key '{target_key}'. Exiting.")
    exit()

# Display all old values and ask for the new value
print(f"\nOld Values for key '{target_key}': {old_values}")
new_value = input(f"Enter the new value for key '{target_key}': ")

# Update all files with the new value
for root, dirs, files in os.walk(directory_path):
    for file_name in files:
        if file_name.endswith(".json"):
            file_path = os.path.join(root, file_name)
            with open(file_path, "r") as file:
                json_data = json.load(file)

            # Update the value in the JSON data
            json_data = update_key_value(json_data, target_key, new_value)

            # Save the updated JSON file
            with open(file_path, "w") as file:
                json.dump(json_data, file, indent=2)

print(f"Value for key '{target_key}' updated successfully in all files.")

[poautran]

In GitLab by @woutdenolf on Jun 25, 2024, 11:28 GMT+2:

If the parameter value is the same for all workflows it is easy

ewoks execute */*.json -p monitor_name=fpico1

or if you want to be more specific

ewoks execute */*.json -p IntegrateBlissScan:monitor_name=fpico1 --input-node-id taskid

Or submit jobs

ewoks submit */*.json -p monitor_name=fpico1 -c queue=slurm