Automation: Execution Instructions for XOR/OR Gateway, Timer Event, Starting Variables, Network Hosts, synchronous Script Functions, etc.

[iaktern]

Hotfixes - Prio 1 (on current main):

• Bug Subprocess: after opening a collapsed subprocess in the Editor, we are not able to click and change the Tasks anymore in the BPMN Editor (just sequence flows). (a complete page reload enables the editing again.)
• Bug in "Roles" view: the member count for @everyone does not work

Login/Register Modal:

• Improve input fields, form and errors: 1. mark all input fields are required 2. validate the inputs: all input fields including username should only accept the characters they are allowed to have (at least two characters, numbers?, space?) -> the buttons should not work if the validation is not correct 3. show understandable errors before and after submitting if something is mistyped/non-validated or the username already exists or the login was wrong
• Solve Redirection Problems: after logging in, the user is redirected to a specific space with the callback URL. This often the space where a user just logged out. this is useful but there are also problems, e.g. if you log in with a user that has no access permissions to the space of the previous user. Implement a way so that all the following requirements are fulfilled:
  • Check access permissions at space level (and not on page level). Inside the URL you find the space-id, use this to check the permissions before a specific page in the space is loaded. (e.g., it is still possible to open Profile Page inside every space - if you know the space-id -> do not check the access rights in page level)
  • After a clean sign out (meaning no automatic timer-based sign out but a click on "Sign Out" button) and a fresh "Sign In", automatically redirect to default space which a user has set in his profile.
    • Add an option in the users profile page to set a default space. By default, that should be the first space where the user was added after registration.
  • If some pages of PROCEED are open in tabs, e.g. a specific process, and in one tab the user clicks "Sign Out", all other tabs should also sign out automatically and show the login modal again. But now, in the tabs where the user didn't click the "Sign Out" button, after logging in the user should see the previous page again (e.g. the open process). If the user has no access rights (because another user signed in), then show a permission error with the hint: "You don't have access rights to this process."
• Make the password field a real password field where a user can also decide on the right side to make the password visible (both: for registration and for login)
• after the first login, usually users have to create a new password but both input fields are not synchronized correctly. the bottom input field checks the password with the top one, but not the other way around. -> correct this: both must always have the same content before it is possible to submit the password

Process List AND Process Editor views - Table View:

• Folders: next to the "Import" button, also have an icon for creating a new folder (just an icon, not the text. but of course with tooltip.)
• Folders: add possibility to add an (optional) "ID"
• Folders: add an "Edit" possibility tho change the properties of a folder after creation (currently only remove is possible)
• Export with folders: if multiple processes and folders are selected, export should be able to export the whole folder-structure. in this way, a user can backup all processes at once
• Bug Copy Icon for copying a process: a change in the "ID" field of the copy modal is not applied after copying
• Fill Meta Panel on the right side: Next to the table on the right side, there is an expandable Meta Panel (like the Property Panel in the Editor). The fields are not filled yet -> fill it with the same data that we already have available for the table: ID, Name, etc.
• Bug - Move Modal: Moving a process into the root folder is not user-friendly. (It works by not selecting any other folder.) Add the space name as the top root element, so that this is selectable as the target location.
• Make the min-width of "ID" column a bit smaller (min width should be around 4 letters)
• Bug - Selection in Card view: by clicking on the four-square button in the top right, you get to the Card view. But instead of opening a folder or the process editor, a click only selects the elements -> correct that.
• Export/Download: if an automated process accesses global MS variables like the users vacation days, we need to export these variables as well so that on import the variables will be created in the MS (if not existing). TODO: make a concept about how to store/import them and if everyone should be able to import variables into the global data space of the MS (permissions?)

Process List view -> Read-Only BPMN Viewer:

• Change the gray text color of the readonly fields to black: in the BPMN viewer, all input fields (especially in the property panel) are deactivated and readonly. but the gray text/content color on a light-gray background is not good to read. Use black text/content color for icons and text on a gray background.
• remove the "Create new Process using this Version" and "Set as latest version" icons from readonly BPMN Viewer
• (also for BPMN Editor) Problem: if you have many nested folders and you open a process within it, there is no button in the BPMN Viewer or Editor to get back to the folder where you just came from (back to the folder that stores the process). You can only use the browser back button, but that is not super user-friendly. Implement the following:
  • Back Button (with a left-arrow icon) in the Header bar, on the left of "Process List"
  • "Process List" drop down in the Header bar: it should either show just the processes from the current folder of the open process OR show the full hierarchical structure with folders and processes to click on
  • Clickable Breadcrump menu in Header bar: if the table is open, there is already a clickable breadcrump menu in the Header bar. Also include this if the Editor is open, on the right of "Process List" in the Header bar. (Pay attention: don't make it too long, since we also need space for the Process Name. it is automatically cropped.)
  • if you have the BPMN Viewer or Editor with a process open, then clicking on left navigation elements "Process List" or respectively "Process Editor" should not return to the root folder but instead to the folder where the open process is stored. Only if you are in the table view (i.e. there is no process open), then clicking on this navigation elements should bring you to the root level
• Spawning new process instances should be done via the process list where every process has already a version (not within the Execution view). The user should also not be concerned with deployments and engines - they just want to start a new process instance (only exception is for an advanced mode).
  • If a process version has the isExecutable attribute set to true, then add the green plus and activate button from the Instance view to start or activate a new process instance. The user should still be able to select a process version and also start/activate it. -> If the process is not yet deployed to any reachable Engine, deploy the version. Switch to the Instance view and start a new instance of the process version (or activate it).
  • In the Property Panel inside the "Automation Properties" tab, under "Activate Automation for the entire Process" add toggle button "Show Advanced Deployment Options". This should open deployment modal with the following options:
    • Have a drop-down menu where a user can select a specific, reachable Engine to deploy to. Also add a row called "Automatic Engine Selection" which should be the default option.
    • Buttons: "Cancel", "Deploy and Start" (or "Deploy and Activate") -> implement the deployment to the selected process engine
      • if a specific Engine is selected which already has the selected process version, then "redeploy" (delete the version on the Engine and deploy again all artifacts.)
      • if "Automatic Engine Selection" is selected and the process already exists on one or more Engines (but in an older version), then deploy the newest version to all of these Engines.
      • if "Automatic Engine Selection" is selected and the process does not exists on any connected Engine, then deploy the newest version to a randomly Engines. But: if there are own space-related Process Engines (added in the "Process Engine" view of a space and not Engines added by the MS Admins for every space), then only randomly select one these own Engines and never the MS shared Engines.
• Write a generic validator that checks a process before it is deployed, see here: https://github.com/PROCEED-Labs/proceed/wiki/Process-Validation-Rules-(for-Import,-Style-Guide-and-Deployment). A process should also be deployable, if not every Task has Executions Instructions like HTML or Scripts attached. (This is useful for already starting a process if updates come later.) Show a hint in the case to the user and let him decide if he really wants to deploy the process version.
  • Quick Fix if we have no time: the deployment already fails if not every Script Task has a Script and not every UserTask has a HTML -> Show a good hint to the user why the deployment fails and what is missing.

Process Editor view -> BPMN Editor:

• Version List: change sorting -> show newest versions on top
• Change Versioning a bit: add a non-editable, integer version number that is counted up for each new version. Make "Name" and "Description" not required. Show this number together with the name in the tables where the processes are listed. In the Instance View, show version number and the "Name" and show the "Description" as tooltip.
• Change title for component planned-duration-input:
  • for Property Panel: "Edit Planned Execution Duration"
  • for Intermediate and Boundary Timer Events: "Edit Wait Duration of Timer Event"
  • for Timer Start Events: "Edit Trigger Interval for Timer Start Event"
    • Icon Tooltip for Timer Start Events: "Edit Trigger Interval for Timer Start Event"
• Bug - Keyboard Shortcuts and open Property Panel: the bpmn-js keyboard shortcuts inside the BPMN Editor do currently not work, if the Property Panel is open or collapsed (Del, Copy, etc.). Maybe because of our own keyboard shortcut system. It only works if the Property Panel is closed completely, which only happens if the 'i' icon is clicked. Fix it, so that so keyboard shortcuts work if the Property Panel is open.
  • Property Panel - only two states: currently, the Property Panel has three states -> closed, open, collapsed. We only want two states: collapsed and open. To switch between both states, two options should work: a click on the 'i' icon and a click on the whole collapsed side bar (as well as the icon for collapsing if the Property Panel is open)
• Process Variable definition - harmonize the definition of process variables with the global Data Objects: add Display Name input field, make Description a bit bigger. write Display Name into the XML attribute displayName
  • for Kai G.: document displayName attribute in XML schema
• Automation Activation in Automation Tab: if Automation is activated, highlight the following automateable BPMN Elements in the "Change element" list (wrench symbol) with a blue color and put them on top of the list:
  • for Activities: User task, Script task, Sub-process (collapsed), Sub-process (expanded);
  • for Events: Empty Start, Timer Start, Intermediate throw, Timer intermediate catch, Timer boundary, Escalation boundary, Error boundary, Timer boundary (non-interrupting), Escalation boundary (non-interrupting), End, Terminate end event
  • for Gateways: XOR, AND, OR
• Automation Activation in Automation Tab: if Automation is activated AND the diagram contains BPMN Elements that are not automateable, show yellow hint text under the option "The diagram contains elements that cannot be automated. These are marked with a yellow exclamation mark."
  • if Automation is activated, show a yellow exclamation mark icon next to/on the top right corner of every BPMN Element that cannot be automated. (only Tasks, Events and Gateways. Not the attached data objects or text annotations)

Engine:

• if a Script Task has a syntactic or JS error, write the error to the engines log system. (Also, end the Script Task and this one token with ERROR-TECHNICAL. don't end the whole process - but I think, this is already the case.)
• PROCEED Engine: on a process split (OR/AND GW), the currently generated tokens ids get too long if the process contains a very big loop. Result: the log is nearly not readable anymore. Solution: the current way of generating tokens-ids should be stored in a different instance variable called tokenProcessPath (also for merges and Subprocesses, see wiki). the tokenId should always be just a seven-char string.
• If a defined Variable (in the Property Panel) has a Default Value, set this value at the beginning of the Process Instance
• If a defined Variable (in the Property Panel) is Required at Startup, don't start the Process Instance if it is not set within the start command. Send back semantic error and explanation.
• Static variables for usage inside Script & User-Tasks: inside Script & User-Tasks we should be able to retrieve and use meta data about the process and the current execution. -> Create a list of predefined, read-only variables about the process and instance to be accessed inside the BPMN elements. In the Engine, this meta variables needs to be filled after the instance is started.
  • Suggestions: process-id (definitions -> userDefinedId), process-name (definitions -> name), process-description (process -> documentation), task-id (current task), task-name (current task), task-description (documentation of current task), instance-id
  • Kai G.: document in wiki
  • Update UserTask and ScriptTask Editors for direct usage

Process Engines View:

• make adding Engine more user-friendly. 1. add input drop-down field in the modal for "Protocol" so that one protocol always has to be selected (default: http) 2. a placeholder in the location text input that shows the expected strings, e.g. "127.0.0.1 or my.domain.de", 3. add a "Port" input field with default to 33029 for http and 1883 for mqtt
• Process Engines View:
  • use the expendable row table to show the list of Engines with the following columns: Name, Location, Status, Protocol, Port, Actions (all fields sortable but Actions)
  • Only MQTT Servers should be expandable (not normal HTTP Engines): all discovered Process Engines connected to an MQTT-Server should be shown nested, under the MQTT Server (default: expanded)
  • MS Engines: Show Engines configured by the MS Admins (via the Admin Dashboard, not added by the Space) in a separate table and slightly grayed out (non-editable). This table should always be the last one on the site (under the other tables)
• MQTT-Server can contain multiple Engines. Especially in the case of an cloud MQTT-Server, a user should not be able to see and use all Engines of that server. (Example: PROCEED Labs will offer a public MQTT server where every user can add their own Engine for home automation. A User should not be able to see the other, registered Engines on the MQTT Server). TODO: how to do that? => insert unique ID of Engine - how to get it?

MS Admin Dashboard:

• Bug Fix: "Spaces" view on the left side is currently showing an error -> show all spaces
• "Process Engines" View with two tables: 1. first table: add/delete Process Engines that are valid for the whole MS and can be used inside all Spaces as their default automation Engine. 2. list all process engines added of all spaces with option to jump into the respective Space to manage the the engines
  • also show the connection status and more information in the tables
  • filter for Spaces should work
• "Execution View": List all Processes of every Engine of every Space. Icon for the Admins to be able to jump into the respective Space to manage the processes.
  • filter for Spaces should work
  • in the admin dashboard, the admins should not be able to see the instances or the process itself. just some meta data. the real data will only be seen if they jump into the space

Execution View:

Goal: The Execution page should only list all space processes that currently have or had running instances (not every process in the space). No folders.

• It should also be filtered by the users access rights so that normal users only see their processes or the processes that they are allowed to see. Team manager and Admins should see more processes, like in the Dashboard.
• It should not be possible to deploy a process in this view anymore -> this should be done via the Process List view.
• Continuously collect the instance data of all running processes belonging to the space and store them locally in the MS (with caching).
• Instead of the "Deploy Process" button, add a link and explanation to the Process List that starting new instances of a process is possible via the Process List
• The delete icon in each process row has to 1. stop all instances of this process, 2. delete all deployments on every engine, and 3. visually remove the process from the Execution list. Rename the icon tooltip to "deactivate", because it is should not delete the instance data inside the MS. The process should be visible again if you click the button to show past executions.
  • If there are no running instances of the process, show a normal modal with the following text: "This action will hide the process from the Execution list. You can still access the process executions by activating to show the past executions."
  • If there are running instances of the process, show a warning modal with the following text: "Warning: This process currently has XX running executions. This action will stop all of them! Are you sure you want to do that? It is generally recommended to stop the processes one by one manually." next line with normal text: "This action will also hide the process from the Execution list. You can still access the process executions by activating to show the past executions." normal "Cancel" button and red "Stop and Hide" button.
  • Make the deactivate button work for multiple selected processes.
• Permission system: implement the above mentioned access permission system. TODO: in which cases can a user see a process of another user? (access right for the process or should there be new permissions for the instances?)

Instance View:

• Update the Property Panel and fill the areas with data from the instance
  • Show variables and their current value
    • Implement a way to change the process variables from the MS
  • Show process logs in the "Activity" tab of the Property Panel
  • Show execution instructions (HTML code, scripts, etc.) the Property Panel
  • Bug: the overall Instance State in the Property Panel is wrong if there is a process split and multiple tokens exist. If one token ended, the state in the Property Panel shows "Ended" even if other tokens are still active. Show the instanceState in the property panel.
  • the overall Instance State in the Property Panel is "Aborted" if there was a Terminate Event. That is correct. But don't show it with a red background instead with a black background. And rename it to "Terminated"
  • if a Technical Error occured inside a ScriptTask (e.g. because of a JS error), show the error message. This is currently only visible in the logs of the Engine but we also need that in the MS, so that the Script developer knows the error. Show it somewhere in the Property Panel
• Sort the Instances: newest always first
• (High Prio) reorganize the Start/Resume/Pause/Stop Icons:
  • Add a new icon: maybe tick symbol. It should activate and deactivate a process for execution in the case that there are non-empty start events, i.e. currently only for Timer Start events. (This events gets triggered automatically and create new instances)
  • instead of having the Start icon (Plus Symbol) in the left icon group, separate it in a new 2-icon-group with the Activate icon. it should be in the middle between the filter group and the Pause/Stop group
  • make the Resume/Pause/Stop group to a 2-icon-group. Show the Resume icon (Play symbol) instead of the Pause icon if the process is paused.
• Show the editor icons of the BPMN Editor but open the editors in read-only mode. beneath the Resume/Pause/Stop icons in the middle: Script Task, User Tasks, Events, Gateways, Loops, Open Subprocesses. Next to info icon: XML icon
• if the newest process version is deployed to multiple Engines, will the start of an Instance then only start one instance or multiple ones? -> should only start one instance
  • (low prio) Add an icon like the "Filter by version" icon for "Selecting a Process Engine". if the newest process version is deployed to multiple Engines, activate it. (if it is only on one Engine, the icon should not be visible.) There should be an entry "Automatic Selection" (default) and all Engine with the deployed version.
• Adapt the size of the start form to the content. not too big if the content is small, not to small if the content big.
• "Submit" button of the start form needs very long and the user gets no feedback if he already clicked the button. => Close the start form and show a loading symbol.
• Versions: in the Instance View, show all released versions of a process in the drop down (like in the Process List). If a specific version is selected, a click on plus or activate should instantiate exactly this selected version.
• in the shown process diagram of an instance, mark all the tasks with a orange dashed border that have no execution instructions. (Then, the user is able to see where the process will stop and wait.)

Better Workflow for Automation Development:

• Instance Viewer: add a button to jump directly to the BPMN Editor for editing the process
• BPMN Editor: Define processes as executable: in the Property Panel of the BPMN Editor inside the "Automation Properties" tab add a checkbox on top as the first element to "Activate Automation for the entire Process". This checkbox should be always visible (when no or any BPMN element is "selected" in the process). This checkbox should set the <process> element's attribute to isExecutable to "true" or "false" (attention: a checkbox's change must be written to the attribute of the <process> element and not to any other attribute even if some BPMN element is selected). If false, deactivate the input fields of the automation tab and hide the icon for setting the execution instructions in the top row. Do not delete any existing values. (Currently, isExecutable is always "true" and can not be set in the UI. New: the default should be "false" if creating a new process diagram.)
  • store the value if isExecutable not inside the BPMN, but instead in the db table for fast filtering. in the BPMN store a variable like we already do with other important BPMN attributes. This will be replaced if the XML is needed, e.g. Export or in the XML Viewer.
• BPMN Editor: If process is set as executable, only show BPMN Elements that can already be used in automation in PROCEED (plus all annotation elements that have no execution meaning)
• BPMN Editor: If process is set as executable, show a new button in the versioning dialog of the BPMN Editor to "Version and Deploy" directly from the BPMN editor -> TODO: what is shown after clicking this button (probably: Engine selection and Instance View)
• BPMN Editor: If process is set as executable, show a new button in the BPMN Editor to "Test Deploy and Start" of the latest process diagram. Show an Engine selection dialog. This directly deploys the latest version, starts an Instance and show the Instance Viewer. (Deploying latest is an exception and should not be possible via the Execution View. There, only versioned processes should be deployable.)
• BPMN Editor: if clicking a Task (currently only UserTask and ScriptTask useful): in the Automation Tab, add a checkbox to "Skip this Task during Execution (useful for development)". if multiple tasks are selected, also show this checkbox to skip/enable all of them at once. Serialize in BPMN as attribute skipExecution="true|false"
  • Engine: implement in the Engine that these tasks are skipped and not executed
  • (Kai G.) document in XML Schema
• Script Task Editor: add a play button to directly execute the opened script task. Therefore, only deploy the script task to the Engine and execute it. The script task should not be visible in the list of deployed processes of the engine (e.g. in execution view). This execution should not produce a full "process" instance - just a task instance. Add a button to stop the execution. The Script Task should also be removed from the Engine directly after the execution (if successful or not)
  • If the Script Task uses variables, show the variable pop-up (from non-start events) to fill all variables. Transmit the variables to the Engine and start the script task execution
• Instance View: Right-click on a Script Task should show an option to "Start Script Task only". Implement in the Engine that only the selected Script Task can be executed (from the already deployed process). Stop button should be activated if the script is running. This execution should not produce a full "process" instance - just a task instance.
  • If the Script Task uses variables, show the variable pop-up (from non-start events) to fill all variables. Transmit the variables to the Engine and start the script task execution

---

Gateways:

• XOR-/OR-Gateway - setting condition instructions in the BPMN editor like in the old MS: the click on an outgoing sequence flow should show an icon in the middle if the top row to open a modal with input fields for 1. Description (mapped to name attribute of the sequence flow), 2. Condition (one-line editor), 3. Default Path (checkbox, overrides the value on the gateway if there was already another default set). The condition should be written into the BPMN like described here: https://docs.proceed-labs.org/developer/bpmn/bpmn-gateways#xor-and-or-splitting-gateways
• Splitting XOR/OR-Gateways with one Default Flow (Engine): usually all outgoing sequence flows need to have script conditions attached for execution. But it is also allowed to use a so-called Default Flow (configurable via settings)
  • Check if default flows work in the Engine: all conditions of all outgoing sequence flows must be evaluated first, before the the default flow is used -> if it not works, implement it.
  • please also implement that it is okay that the user only defines one default flow outgoing and no conditions attached on the other outgoing sequence flows. -> All sequence flows with no condition should result the GW evaluation to false
  • Splitting XOR/OR-Gateways: if no condition evaluates to true and there is not default flow, end this one token with ERROR-SEMANTIC
  • Critical Bug: if a GW condition tries to access a non-existing variable, the Engine shuts down. -> if a condition has a wrong syntax or tries to access a variable that does not exists, end this one token with ERROR-TECHNICAL
• Splitting XOR/OR-Gateways: make syntax consistent to Script Tasks with variable.get()
• Splitting XOR/OR-Gateways Editor: 1. add the possibility to see and select all existing variables (like in Script Task Editor). 2. Syntax checking should show no errors if you correctly get a variable (currently it marks it red)
• Splitting XOR/OR-Gateways: if the splitting Gateway is selected, show an icon in the top to open an GW-Editor (currently, it is only shown if an outgoing sequence flow is selected - which is the correct place but a bit user unfriendly). This GW-Modal should show and be able to edit all outgoing sequence flows (don't remove the condition editor from the outgoing sequence flow - it is also not problematic to keep it)
• Splitting XOR/OR-Gateways with full Script support: instead of only defining one condition line, allow to define complete Scripts like in the ScriptTasks
  • Adapt GW-Editor UI
  • store and serialize JS code like with Script Tasks
  • Enable execution in Engine

Intermediate and Boundary Timer Events

• - setting duration in the BPMN editor like in the old MS: the click on an Intermediate or Boundary Timer Event should show an icon in the middle of the top row to open the Duration modal (use existing duration modal of the property panel). The condition should be written into the BPMN like described here: https://docs.proceed-labs.org/developer/bpmn/bpmn-timer#durations

Non-typed Start Events

• Connect Non-typed Start Events to User Task Editor for defining a UI. Similar Serialization and Storage as for UserTasks: in <process ... proceed:uiForNontypedStartEventsFileName="" >
  • Process Start: if a User wants to create a new Instance, show the User Task in the MS before Starting the Process. Click on "Start" triggers a process Start on the Engine
  • (Kai G.): document serialization in XML Schema
• Set Process Variables when Starting a Process Instance:
  • Define Process Variables in the BPMN Editor: in the Property Panel of the BPMN Editor inside the "Automation Properties" tab add a table to define Process Variables with Name, Data Type (Text, Number, On/Off), Default Value. Use List and Design like with Milestones.
  • Defined Variables are valid for the whole Process and all FlowNodes
  • Serialization in BPMN (spec is not enough)
  • (Kai G.): document serialization in XML Schema

Automate several BPMN Elements:

• Loop Task Markers (Loop, Multiple Instance Parallel and Sequential): if the marker is selected for a task, add an icon in the top row next to the other editor icon (but with with loop or multiple instance symbol). This should open a modal where the user can set the loop characteristics. write it to the XML.
  • implement functionality in Engine
  • mark them blue in "Change element" list (wrench symbol) if executability is on activated
  • (Kai G.): documentation
• Extend the BPMN Editor with the definition of Error and Escalation Events like described here.
• Call Activities like in our docu:
  • BPMN Editor: Select other processes in the MS and link them in the XML
  • Deployment: Deploy main process and all linked processes
  • Instance View: Add icon to open Call Activities
• Optional Attribute for Activities/Tasks: In the Property Panel, under the section "Properties", add a checkbox to mark a selected Activity/Task as "Optional Task" with a question mark icon behind it explaining: "Marks a task as optional if it does not always need to be performed. For example, an employee does not always ask where an application form is kept if they already know." If activated, make the border of the selected Activity/Task dashed. Serialize this option as XML attribute proceed:optionalTask="true" inside the task element
  • Kai G.: document the XML serialization in XML schema
• Conditional Event: this is actually only a JS Script Execution
  • MS: if a Conditional Event is clicked, show icon on top and implement to open the Script Editor (JS+Blockly) for Conditional Events and store the scripts similar to Script Tasks (in the file list: show all Script Tasks and Conditional Event Code)
    • allow throw new ConditionSuccessful() keyword in monaco editor
    • (Kai G.) document throw new ConditionSuccessful() error event for Start and non-interrupting boundary events
  • Engine: implement to execute JS code for intermediate conditional events
  • Engine: implement to execute JS code for conditional interrupting boundary events. Start the conditional event script directly after the task was started. if the script ends, then stop (terminate) the attached task and trigger the token on the outgoing sequence flow
  • Engine: implement to execute JS code for conditional non-interrupting boundary events. Start the conditional event script directly after the task was started. if the conditional script ends, then trigger the token on the outgoing sequence flow but do not end the attached task. A non-interrupting boundary event can also occur multiple times. This should be done in a loop and every time the condition occurs the user/script can throw new ConditionSuccessful() error event. (or is there a better event triggering way than using an error?)
  • Engine: implement to execute JS code for conditional Start Events in the case that the process is activated. This is actually just starting the JS script. It should somehow pre-create the template for a Process Instance but without the times and ids. Every time either throw new ConditionSuccessful() is thrown (usually from inside a loop) or the Conditional Script ends, a new, complete Instance is created with new ids and the correct times. but also the set variables from the script should be transferred to the instance. (For now, the Engine should not automatically restart the script after it ends. This means, the condition can be fulfilled one time, a new process instance is created and afterwards the process is not activated anymore.)
  • MS - Engine: show and implement the activation and (automatic) deactivation of processes with start conditional events.
  • (Kai G.) document Conditional Events

UserTasks Editor:

• (Janis/Felipe) UserTask and ScriptTask Editor: "Save" button should close the Modal. It should also show the green "Saved" pop-up (currently only ScriptTask Editor)
• Bug: Cancel button and closing X show a hint about unsaved changed, although nothing was changed. -> if nothing was changed, don't show this hint.
• Hint modal about unsaved changes (Cancel button and X): Add an additional, blue Save button (saves and closes form) and rename "OK" to "Discard Changes"
• Add User Task name to title of Editor modal after "Edit User Task Form:"
• Add Default Content:
  • for User Tasks: one Header element that contains the task name and one Text element that contains the task documentation description (see later: by using the predefined process variable task-description) and one Submit button -> automatically attach this form to every UserTask even if the Form Editor was never opened.
  • 1. for Empty Start Events, if there has no process variable been defined which is required to be filled at Startup, then: do not attach/define any default form automatically until the Form Editor is not opened. But if the Form Editor is opened for the first time, attach the same default template as in the next point but with an empty variable list.
  • 2. for Empty Start Events, if there are process variables defined which are required to be filled at Startup, then: one Header element that contains the process name and a list of Input elements for every variable that needs to be filled at Startup and one Submit button => This automatically generated form should automatically be attached to the process (but only of there is at least one startup required variables). This is just the default form, it can always be edited and refined by the User
    • Problem: 1. variables were added after the Startup form was already edited by the user: add new Input fields for the new variables directly over the Submit button
    • Problem: 2. variables were deleted after the Startup form was already edited: delete the respective Input fields for the variable from the list. (-> It should not be possible for the user to delete Input fields for variables that are required to be set at startup. This should only be done automatically by the MS)
  • Remove the tile for the Submit button -> Users are not allowed to add one by themself. Also, don't allow Users to delete the existing Submit buttons within the templates -> so, there is always a Submit button inside the form
• (Janis) UserTask Editor: if the mouse is over a text field, it gets gray. super. double click on the whole gray area should also open the text edit mode (next to the left click on the edit icon)
  • for opening the text editing mode, automatically select all containing text (so that a click on a letter will override the existing text)
• (Janis) UserTask Editor: in a normal Text field, Enter should not leave the field but go to next line (next to Shift+Enter)
• if in text editing mode of any element, "ESC" key should close the text editing mode (instead of trying to close the UT editor).
• Delete icon not on top but on every element next to the edit symbol
  • also make the keyboard shortcut "Del" work for deleting
• instead of the Delete icon on top, have a Reset icon to reset the form to its default content (of course, first show a with a Hint modal with a warning)
• only for Empty Start Events Form Editor: add an icon in the top row to delete the attached form. This should only be possible if there are no variables that need to be defined at startup. this button closes the modal after deleting the start form.
• input element: remove "Type" setting and take the type from the variable definition instead.
• text element:
  • is it possible to allow markdown syntax like in our other text editors?
  • allow enumerations (e.g. with the markdown syntax)
• checkbox element improvements:
  • delete the gear symbol of each checkbox row. instead, make each checkbox row also selectable and show the settings if the row is selected
  • even when you are editing the text of a row, show the row settings (variable, adding rows, etc.) under the text editing elements
  • add icons+functionality to move the selected row one row up or down
  • additionally, show the icons for adding one new checkbox row at the bottom inside he checkbox element - inside the editor on the right side. this elements should be clickable in the editor, but it must not be rendered in the final UserTask at runtime. this is just for better visibility that someone can add new checkbox row with these icons.
  • is it possible to drag and drop checkbox elements (as new rows) into an existing checkbox group? (in the beginning, I tried that for adding new rows, because it was not intuitive for me how to add rows)
  • if the whole checkbox group is selected, also show the buttons for adding new rows on top or bottom under Settings on the left side
  • if the whole checkbox group or one row is selected, rename "Variable" to "Variable for all Checkbox Rows"
  • if one checkbox row is selected, rename "Value" to "Variable for the selected Checkbox Row"
• UserTask Editor buttons:
  • Other buttons (e.g. Cancel): add possibility to add further buttons ("Submit should always be there). These additional buttons should have multiple actions: 1. end UserTask by setting a variable (could be checked in a Gateway afterwards), 2. throw error and throw escalation events (to go to attached boundary events), 3. open a link in a new tab, (4. (low prio) trigger some internal script (which maybe calculates the remaining vacation days)
• Upload Files (PDFs, etc) and store them in variables + Engine + datastore (MS)
• Enable Scripting within the UserTask form. Example: live calc of the remaining vacation days, if the start and end are selected
• Add a new Drop-Down element in UserTask Editor for showing and selecting a list of values. For example, to select a list of available laptops. The drop-down should be connectable to an array from the process variables (type is called "List"). Options on the left side should be: "List to show" -> variable selection, "Store Selection in:" (here, the user needs to select in process variable the selection of the drop-down is stored), checkbox for "Multiple Selections are possible"
  • Extend variable definition of type "List": in the "Add a new Variable" modal, add the possibility for "List" variables to add starting/default values. Also look if adaption to the process engine are necessary for filling the array at instance start.
  • Make Global Data Objects also selectable for drop-down element: if a global data object is selected, the Display Name of all child elements on the first level under the selected Data Objects should be shown in the list. but not the value should be stored as the selection result, instead the name
• Be able to assign a employee's boss and back office as the executing person
• UserTask Documentation: show a designed UserTask in the Process & Instance Documentation Page (as an option)
  • add the possibility to convert/export a single UserTask into a PDF (like in the Documentation page)

Script Tasks:

• Bug: if a Script Task is interrupted/ended by a boundary event, e.g. timer boundary event, the currently existing local variables are not written into the process variables (there state gets lost)
• Editor: show in top in heading after "Edit Script Task:" the name of the ScriptTask
• Editor: improve syntax checking, so that it conforms to our api (e.g. network.post() has two optional arguments)

• JS Code improvements, so that the script task always 1. ends when all code was executed, and 2. is interruptable at any time (if a "Pause" or "Stop" is triggered):
  • offer a synchronous wait(ms) function for the user to use in script tasks. If the process has been paused by the user (via the MS) during the execution of wait(), return true, else return false. Remove await setTimeoutAsync() from editor (MS) and script execution (Engine) (Note: we still want an asynchronous version, but under a different name, see later). use a similar implementation like here: https://stackoverflow.com/questions/3969475/javascript-pause-settimeout -> if paused or stopped, this is how you can realize it
    • (Kai G.) test and put in documentation
  • change await setIntervalAsync() (MS Script Editor and Engine) to let id = setInterval(clbFunc, 10);. Return no Promise anymore, but instead the interval id which should be canceable with clearInterval(id);. This is how the normal setInterval() of Node.js or Web also works, but since we offer the functions, register that this function was called and works until it is canceled with clearInterval(id);. You can then reuse the normal setInterval()
    • a ScriptTask should not end if there are running intervals
    • setInterval() must be pauseable and stoppable, see wait()
    • (Kai G.) test and put in documentation
    • (low prio) Bug: the timer of setInterval() for the next interval only starts after the last execution ended. => if the user sets an interval for every 5 seconds, he expects the interval to run each 5s (and not 5s after the last execution)
      • (Kai G.) test and put in documentation
    • Bug: the usage of multiple setInterval() within one Script Task is currently synchronous, i.e. a long-running code (network call or wait()) blocks the start/execution of other intervals. this is wrong => if the interval code contains some of our long-running, synchronous functions (like network calls or wait()) it should only block/wait to execute the next commands that directly follow but not the parallel execution of other timers and intervals. You can test it by starting two 1s intervals and one having a wait of 5s inside the code - by using console.info() you can see in the timestamps that the 1s-interval is not executed within the 5s waiting of the other (-> is that possible to achieve?)
      • (Kai G.) test and put in documentation
  • make all of our own script functions (currently all network functions) synchronous (same name) -> no await should be needed, so don't return a promise. (Note: we still want an asynchronous version, but under a different name, see later). at runtime, since the network functions of a script task call our function implementation, the network function should still work asynchronously in the background (don't block the JS execution) but the user does not know. our implementation returns if the network request is answered, but within our implementation also have event-handlers for pausing and stopping an open request.
    • if Paused (during a network request that needs a little longer for the answer): return the answer to the ScriptTask after Pause is ended. So, store the answer if it arrived during pause.
    • if Stopped: end the script task and all open requests/promises/etc.
    • (Kai G.) document new script language in documentation
  • now, after the previous changes, everything is synchronous, which makes it is easier for us to achieve the goals 1. and 2. (Reason: we want to allow pure JavaScript/ECMAScript but not any further functions from the different runtimes like web or Node.js. And in pure JavaScript/ECMAScript, there are no async functions predefined. -> So, everything is synchronous.) But, this does not allow any parallel executions with asynchronous code anymore, if some execution needs a while (network request, wait, etc.). That's why we should should also allow our own async functions, if we know that they are still running (so, the ScriptTask will not end) and which we can interrupt with pause and stop.
    • Check if only pure JS/ECMAScript is allowed in the Engine for executing scripts? (no access to Web or Node APIs?) => goal: only pure JS and or self-implemented functions are allowed
    • offer asynchronous waitAsync(ms), network.getAsync()/postAsync()/putAsync()/deleteAsync()/headAsync() functions for the user to use in script tasks, which returns a promise. Like with the synchronous variants, make it pausable and stoppable.
      • (high prio) Bug (Engine) - Error handling: with the async network functions it seems that the error handling is not correctly implemented. Errors should be given to reject(). sometimes it seems to be done but not always. For example: it is not possible to .catch() the error if an DNS address does not exist and can not be resolved (e.g. https://abc.proceed-labs.org/). Also, other statuscodes than 2xx can be catched, but they additionally print an error message on the console (which should not be done, example: getAsync() on www.proceed-labs.org without user-agent header).
    • (Kai G.) document new script language in documentation

networkServer.getProcessInstanceListeningPath() OR networkServer.path() // return the "root" path for a process instance where it can receive requests, which should be /running-processes/<instance-uuid>/ 

// Standard way: 99% of all use cases
// open a server and wait exactly for one request. afterwards, close the server again
let {req, res} = networkServer.get|post|put|delete|head(path) // same as express, but as syncronous function (no callback, no await): https://expressjs.com/en/5x/api.html#app.METHOD
res.send('Hello World');

// 0,5% of use cases: there should also be an async variant with a promise, which can handle *one* incoming request asynchronously
// the ScriptTask should not close until this one request was handled
networkServer.getAsync|postAsync|putAsync|deleteAsync|headAsync(path).then( ... )

// 0,5% of use cases: there should be a way to create a server that handles *multiple* incoming requests -> use callback (i.e. it is async and other code can run afterwards)
// the ScriptTask should not close until the server object is closed with the script task!
let server = networkServer.get|post|put|delete|head(path, (req, res) => {
  res.send('Hello World');
  server.close(); // important to end the script task!
});

• Allow creating a network server with the previous script task code
  • Implement networkServer.getProcessInstanceListeningPath() and networkServer.path()
  • Implement that get|post|put|delete|head(path, [optional clb]) or getAsync|postAsync|putAsync|deleteAsync|headAsync(path) register a listening server in the Native Part of the Engine. After receiving a request to the instance path on /running-processes/<instance-uuid>/ or /running-processes/latest/<process-uuid>/, forward the request to all waiting ScriptTasks of the instance-uuid or of the latest instance of a process
    • usually, after one request to a specific method+path, remove it again from the open listening list. (if no callback is given as parameter.)
    • usually, 99% of all use cases will probably be with the synchronous server methods
    • if a function (callback, i.e. the functions gets asynchron) is given, a continuous server should be opened until it is closed with serverObj.close();. The script task should not close before that.
    • multiple, parallel ScriptTasks should be able to open a server for the same method+path within one process instance => send the incoming request to all listening ScriptTasks
    • multiple, Process Instances should be able to open a server for the same method+path. Usually an external requester would directly address one specific process instance, but he can also address the latest process instance => send the incoming requests to all listening ScriptTasks of the newest process instance
    • a ScriptTask is not allowed to end, if it has open listeners
  • if process is Paused: return 503 with body: "this process instance is currently paused and does not handle incoming requests"
  • if process is Stopped or no server code was created in ScriptTask: return 400 with body: "this process instance is currently paused and does not handle incoming requests"
  • req and res should be similar to the express objects: https://expressjs.com/en/5x/api.html#req.app and https://expressjs.com/en/5x/api.html#res.app BUT: we can not directly use the express objects because of the serialized IPC connection to the subprocess for a script task (which blocks sending functions). So, you don't need to implement everything, just a few things: all properties and no functions on req (read-only, thats why it is easy to serialize and transmit to the script task), these functions of res: cookie(), clearCookie(), end(), json(), location(), send(), set(), status() (maybe less, since one response method would also be sufficient. so, you could decide between end(), json(), send() )
  • (Kai G.) document the new server functions on the website

---

• JS and Blockly Editor: Improve looking up the code of other ScriptTasks: in the ScriptTask Editor, add an area to open other ScriptTasks of the same and other processes (for reading and copying code). ScriptTasks of the same process should be opened in read-write mode and from different processes only in read-only mode (show a little badge in the top for the mode).

  • (Bug: "Save All" button does (sometimes) not work to save changes made in other ScriptTasks of the same process -> but I don't know how to reproduce it. test if you can also experience it.)
  • (Low Prio) Is it possible to select and copy blocks from Blockly into another Blockly diagram?
  • The current process should always be on top of the process/file list. The other processes should be alphabetically sorted under it.
  • File List (on the left): show the name of the ScriptTask instead of the filename
  • Bug: the JS Editor windows are not separate from each other: if two variables with the same name are declared in two separate script task (e.g. let a;), the editor shows a "redeclare block-scoped variable" error

• Small UI Improvements in JS and Blockly Editor:

  • Bug: "Script syntax is valid" tag does not work -> it is also displayed if the JS code is syntactically wrong
  • JS Editor: put the "Script syntax is valid" tag to the right side, like in Blockly
  • Buttons to close the Editor Modal in the right bottom corner should be "Cancel" and "Save All". Both should close the Script Editor. "Save All" should be deactivated by default and activated if the user changes something in any editable code file. If "Cancel" is clicked and some code is not yet saved, show the existing question modal if you want to discard the changes.
    • Modal which questions the user if he wants to discard the changes: add a list of all files that have unsaved changes
  • (if possible) JS Editor: Don't be able to "Save" if there are syntactic JS errors. (But be able to save (JS) code if we don't know the objects or types. Reason: we don't have all possible objects registered in the monaco editor)
    • (if possible) JS Editor: Don't be able to "Save" if there are loose blocks in Blockly.
  • add an Editor Icon bar on top (on the left side of "Script syntax is valid" tag, above the content part of the editor). All icons should have tooltips:
    • "Save" icon to save the single open script
      • (low prio) catch "Ctrl + S" keys to also save the file
    • Blockly: make "Switch to JavaScript Editor" as an icon
    • make "Open Documentation" to an icon
    • JS: add an "Format" icon to format the code (this can already be done with "Ctrl + Shift + i")
    • JS (low prio, only if fast+easy): add an icon to show a modal with useful keyboard shortcuts coming from the monaco editor

• JS and Blockly Editor: add the ability to write code/scripts for the event that process Pause or Stop is clicked: Add two new (Tabs?/Sub-Files?) in the Script Editor to write code that is triggered if Stop or Paused is clicked (e.g. pausing or stopping the 3D printer).

  • Extend the PROCEED Engine to execute the Pause and Stop code if the command occurs.

• Engine: the setVariable() functions should consider the set format, e.g. number, date, url, etc.). maybe there can also be hints in the JS Editor, if the user tries to set a wrong format.

• (Low Prio) For every Script Task, enable the possibility to define a "Default Network Host" with IP OR URL in the Property Panel of the BPMN Editor inside the "Automation Properties"

  • Write the value into attribute proceed:defaultNetworkHost for the selected Script Task
    • Kai G.: document in XML schema
  • PROCEED Engine: always if networkRequest.get/put/post/delete/head(path[, options]) inside a script task have a path variable without a host part (e.g. just /api/users/) use the value of proceed:defaultNetworkHost as default for the target of the network requests (if it is a valid IP or domain). For example, network.get('/api/users/') should send the request to the IP 12.0.0.34 if this is set in the defaultNetworkHost attribute of the ScriptTask

IAM

• Bug - PROCEED System Admin has too many permissions: the system admins have too many permissions inside spaces, where they are actually only normal members. e.g., they can automatically see processes or change roles, even without the permission. change so that the system admin is only a normal member if added inside a space

Code refactor: over time, we renamed the concept of spaces. Now, it is updated in the UI and should be called "Space". But in the past we also called it "environments", "Personal Space" and "Organization". The difference between personal and organizational space is currently not so important anymore and in the future we will probably delete "personal" space"

• Remove "personal" spaces -> everything should be normal spaces where you can also add other users and Roles. Attention: "personal" spaces are also kind of the user accounts PROCEED. we still need separate user account, e.g. if a user is already registered in PROCEED, if you add him inside a space as a normal member, it should still be the same user account. (there should not be any second account created.)
• refactor code and db: call it "space" everywhere

MQTT Connection in MS:

• have an MQTT Subscriber functionality in the backend that is triggered by the frontend and running the whole time as long as there is at least one MQTT Engine registered under Process Engines
  • continuously ping from Frontend the MQTT Subscriber (this maybe prevents Google Cloud Run from shutting it down very fast)
  • for Streaming use SSE in Next.js. Problem: Google Cloud Run can shut down the connection anytime. Therefore, die Frontend maybe misses some data. Two Solution possible: 1. if SSE Connection closes, reestablish a new one, 2. preemptively close the SSE connection and reestablish a new one
  • Subscription Problem: Google Cloud Run can shut down the connection anytime and we miss some data.
  • At MQTT Subscriber startup and after reestablishing the connection to an MQTT Server, fetch the following data via the pms/engine//api/ of every Engine and store it in the DB:
    • all deployed processes: only fetch the BPMN of the newest version, store the number of versions and the Engine-ID where they are deployed
    • Instance data: all instance data of running and ended instances of the newest process version. Store it without the log part inside of the instance object. Therefore improve Engine Endpoint:
      • add query to Engine endpoint /process/{definitions}/instance/{instanceId}?log=false and document in wiki
      • /process endpoint to only list an array of processes with some few meta data like described in the wiki (no bpmn or instance data => pay attention, where the endpoint is already used in PROCEED)
      • /process/<id>/versions endpoint to only list an array of processes with some few meta data like id and creation-date. update wiki
    • only fetch and store the data, if not already in db
  • Subscribe to the following topics:
    • ...
  • Store the following subscription data in the DB:
    • ...
  • Streaming the following data to the frontend with SSE:
    • process log

---

Backup - Done Tasks (collected at the buttom for readability)

Hotfixes - Prio 1 (on current main):

• (Prio 1) Process Deployment seems not to work without MQTT-Server (I tried direct Engine on localhost with HTTP)
• (Prio 1) Execution View: only show versioned processes for deployment. Also show hint in top of the modal: "Hint: For deployment, a process must be versioned. Only the newest version will be deployed."
• (Prio 1) Execution View seems not to be rendering in Personal Spaces. The logs say that it tries to connect to an MQTT-Server (it also tries that in Org Spaces but there the rendering finally works). But it must also work if no (MQTT-)Engine is connected. (maybe connected or easy to solve in this context: while building the MS, it also tries to connect to an MQTT-Server. but this should not be the case)
• (Prio) Process Engines View: it should be shown, if stored Engines are online/reachable with the green dot and tooltip (currently no function)
• (Prio) delete "Templates" and "Projects" element from the left navigation side
• (Prio) create a small View for the Dashboard. E.g. Number of Deployed Processes, Number of connected Process Engines, Running versus Finished versus Aborted Process Runs.
• (Prio 1, Janis) UserTasks: Possible Performers in the "Automation Property" tab should also be available in Personal Spaces (then, there is just one User to assign a task to)
• Rename "Machine" View to "Process Engines"
• Execution View "Bug": load the view with the table immediately, independent if there is a connection to any PROCEED Engine. Have a loading symbol inside the table indicating that in the background the process data is fetched from all Engines. Timeout after 15s: show empty table if not processes could be fetched.
• Deployment Bug: if you try to deploy an already deployed version, there only comes the error badge that "Something went wrong". Show "Process is already deployed in the newest version" in a yellow badge.
• XML Editor: add X to the top right of the modal to be able to close the modal (like "Cancel")
• XML Editor: if not in writing mode, there should only be one blue "OK" button in the bottom right corner which closes the modal (not "Save")
• Bug XML Editor: going to edit mode, change some thing (e.g. the text inside the documentation element) and then clicking on Save -> the Save Warning modal stays open and the process is not reloaded.
• Bug XML Editor: the user should not be able to go back to non-edit mode if he did changes in the XML -> Request modal to discard or save changes.
• Bug Copy Script and User Task: if the user copies a Script Task (with Ctrl+C and Ctrl+V), there is no new file generated and there is a prisma error on the console. => show no error on console and the script content should be copied into a new script file which should be linked by the new Script Task.