M8F-285: Fix missing process data in certain process instance stages

[sonal-aot]

JIRA Ticket

M8F-285

Description

Fixes intermittent issues where workflow task pages displayed errors or empty
values even though the underlying task data was present. Adds m8flow-backend
patches around four upstream code paths that surfaced confusing 500s / blocked
the flow:

1. Jinja instruction rendering — task instructions now see process-level
   workflow.data (data objects) plus data from all completed tasks in the
   process instance, not just the current task's local data. Variables computed
   in a prior ServiceTask's postScript (e.g. prs_count, prs_status,
   prs_table_rows in the GitHub connector demo) now render correctly in the
   next ManualTask. Any failure during rendering is swallowed and logged so the
   form still loads.
2. Form schema variable validation — when a form references an
   options_from_task_data_var: variable that is missing or an empty list,
   upstream raised a 500 ApiError. We now log a warning and render the field
   with empty options instead of breaking the page.
3. spiff_task_to_api_task — None email crash — upstream did
   ",".join([u.email for u in human_task.potential_owners]), but
   UserModel.email is nullable. A potential owner without an email produced
   TypeError: sequence item 0: expected str instance, NoneType found inside
   the interstitial stream, which put the process instance into ERROR status
   and blocked task submission. The patch falls back to username
   (non-nullable) for any owner whose email is None.
4. WorkflowException.add_note(None) poisoning — WorkflowException.__init__
   and TaskModelError.__init__ call
   add_note(did_you_mean_from_name_error(...)), which returns None for
   NameErrors with no close match. The None ends up in self.notes, then any
   later str(exception) triggers ". ".join(self.notes) and raises a
   misleading TypeError instead of surfacing the real error. The patch
   overrides add_note on both classes to silently skip None values.

All fixes are isolated to m8flow-backend patch modules (no edits to upstream
spiffworkflow-backend). New patches registered in patch_registry.py.

Type

• Feature
• Bug fix
• Documentation
• Other

Changes

• Backend
• Frontend
• Documentation

Testing

• Ran the GitHub connector "View Pull Requests" workflow — PR table rows now
  render with the postScript-computed values; previously all values were blank.
• Ran the loop / sequential-approval-with-rework workflow through multiple
  iterations (Submit Content → Review Content → Changes Requested → Submit
  Content again). Confirmed:
  • No "TypeError: sequence item 0: expected str instance, NoneType found"
    banner on the task page.
  • Process instance stays in user_input_required (no longer flips to
    error) so Submit / Save and close remain enabled.
  • Reviewer Feedback instruction renders correctly on the second iteration.
• Verified forms with missing options_from_task_data_var: variables render
  with empty options instead of returning a 500.

Related Issues

Closes #

[github-actions]

PR Agent Review

Blocking issues

1. jinja_service_patch: broad exception swallow changes behavior and can hide real failures
   • In patched_render_instructions_for_end_user, all exceptions are now caught and the function returns "".
   • This can silently suppress template/rendering errors (including programming bugs), causing missing instructions with no user-visible error and making diagnosis difficult.
   • It’s also a functional regression: callers that relied on error propagation now get empty output.
   • Action: Narrow exception handling to expected recoverable exceptions (e.g., data lookup/parsing), and re-raise unexpected exceptions. At minimum, include stack trace (logger.exception) and consider feature-flagging fallback behavior.
2. jinja_service_patch: potential correctness regression by switching to raw task_model.data
   • New code uses:

     task_data = task_model.data if task_model.data is not None else task_model.get_data()

   • If task_model.data is a hash/serialized form (common in this codebase via json_data_hash patterns), this may bypass normalization/lookup done by get_data() and produce wrong context type/content.
   • Action: Prefer get_data() consistently unless there is a proven perf issue and guaranteed schema of task_model.data. Add guard/tests validating data shape.

Non-blocking suggestions

1. process_instance_service_patch: duplicate assignment
   • ProcessInstanceService.spiff_task_to_api_task = patched_spiff_task_to_api_task appears twice (second with # type: ignore[assignment]).
   • Not harmful, but noisy and confusing.
   • Action: Remove the duplicate line.
2. process_api_blueprint_patch: mutating schema to empty options may break downstream validation unexpectedly
   • Replacing missing/empty variable options with [] is safer than 500s, but can lead to forms that render unusable states without explicit client messaging.
   • Action: Consider adding a structured warning in API response metadata (if possible) so UI can show a meaningful message.
3. workflow_exception_notes_patch: monkey patch may affect unrelated code paths globally
   • Global override of WorkflowException.add_note/TaskModelError.add_note is pragmatic, but broad.
   • Action: Add explicit type coercion (str(note)) or assert behavior in docs/tests to prevent future incompatibilities.

Recommended tests

1. Jinja instruction rendering fallback behavior
   • Case: valid task/context still renders normally.
   • Case: recoverable missing data path logs warning and returns empty string (if intended).
   • Case: unexpected exception in template logic should still raise (after narrowing catch), to prevent silent failures.
2. Task data source correctness
   • Verify _task_model_instruction_data when task_model.data differs from task_model.get_data() representation.
   • Ensure merged context contains expected dict values, not hashes/serialized blobs.
3. Completed task merge ordering and exclusion
   • Confirm ordering by end_in_seconds produces deterministic overwrite behavior.
   • Confirm current task (guid) is excluded.
   • Edge case: end_in_seconds null/identical timestamps.
4. Process API schema patch behavior
   • options_from_task_data_var missing var => no exception, options become [], warning logged.
   • var is empty list => options [], warning logged.
   • var is string => ApiError 400.
   • var is list of {value,label} => transformed schema shape as expected.
5. WorkflowException note sanitization
   • add_note(None) does not poison notes and str(exception) remains safe.
   • add_note("msg") unchanged behavior.
   • Regression check for TaskModelError subclasses.

Generated by model gpt-5.3-codex on PR updates via OpenAI Responses API.