Passing parameters to magiclinks

[arminus]

First of all, thanks for the very helpful https://docs.xyops.io/#Docs/compare page!

As an n8n user, I have one question though: I have a set of n8n workflows which get triggered by webhooks which can also receive parameters in n8n as well as return data. I kind of miss this in xycop's magiclink concept. Is this something you plan or does that not fit your strategy?

[nickdollimount]

Hey @arminus,

You can send data to Magic Links now. You can either pass the parameters through the Query Parameters in the URL on a GET request or pass them in the body as JSON in a POST request.

Examples (PowerShell):

GET Request

Invoke-RestMethod -Uri 'https://xyops.domain.com/api/app/magic/v1/94XlHVdhN5j5wbW5J3Lu4eGBwZhqwerdyJCFjXZp268FWNoPJL2ryYs6goae0wNAF?firstname=John&lastname=Doe'

POST Request

Invoke-RestMethod -Uri 'https://xyops.domain.com/api/app/magic/v1/94XlHVdhN5j5wbW5J3Lu4eGBwZhqwerdyJCFjXZp268FWNoPJL2ryYs6goae0wNAF' -Method Post -Body (@{firstname = 'John'; lastname = 'Doe'} | ConvertTo-Json -Depth 100) -ContentType 'application/json'

The results provide an id and stream fields. These can be used with the stream_job API to stream the job updates from the server.

References:
https://docs.xyops.io/#Docs/api/magic
https://docs.xyops.io/#Docs/api/stream_job

Though I haven't been able to find anything regarding the job just returning a custom result. There is a bit about setting a custom success result but it's not programmatic based on your script itself, but instead it's instant and the job just runs in the background.

https://docs.xyops.io/#Docs/triggers/magic-link

I hope this helps. It would be a good feature request to allow the job to return custom HTTP results at the end of the script run to allow returning of data.

[jhuckaby]

Ah, Nick covered this quite well.

One additional detail I just want to add about returning data: the stream_job API actually includes the final job output in its SSE stream, including the job's custom output data.

When the job completes, the final update event contains the job’s code, description, elapsed, and, when applicable, its output data and files. This is followed by an empty end event.

This means the calling application can keep the HTTP connection open, process updates as they arrive, and then use the data object from the completed update as the returned result. Functionally, this should provide the same capability as returning job data in an n8n webhook response, except that xyOps delivers it through an SSE response.

Here is a small Node.js example which calls magic to launch the job, and then streams the job response with stream_job:

const magicLinkUrl = 'https://xyops.example.com/api/app/magic/v1/YOUR_MAGIC_LINK_TOKEN';

// Add optional event parameters to the Magic Link URL.
const magicUrl = new URL(magicLinkUrl);
magicUrl.searchParams.set('firstname', 'John');
magicUrl.searchParams.set('lastname', 'Doe');

// Start the job.
const magicResponse = await fetch(magicUrl);

if (!magicResponse.ok) {
	throw new Error(`Magic Link request failed: HTTP ${magicResponse.status}`);
}

const job = await magicResponse.json();

if (job.code !== 0) {
	throw new Error(job.description || 'Failed to start job');
}

// Connect to the SSE stream using the returned job ID and stream token.
const streamUrl = new URL('/api/app/stream_job/v1', magicUrl.origin);
streamUrl.searchParams.set('id', job.id);
streamUrl.searchParams.set('token', job.stream);

const streamResponse = await fetch(streamUrl);

if (!streamResponse.ok) {
	throw new Error(`Job stream failed: HTTP ${streamResponse.status}`);
}

const decoder = new TextDecoder();
let buffer = '';
let jobOutput;

for await (const chunk of streamResponse.body) {
	buffer += decoder.decode(chunk, { stream: true });

	const messages = buffer.split(/\r?\n\r?\n/);
	buffer = messages.pop();

	for (const message of messages) {
		const event = message.match(/^event:\s*(.+)$/m)?.[1];
		const json = message.match(/^data:\s*(.+)$/m)?.[1];

		if (!json) continue;

		const data = JSON.parse(json);

		// Display each SSE update as it arrives.
		console.log(event, data);

		// The completed update contains the job's final output.
		if (event === 'update' && Object.hasOwn(data, 'completed')) {
			jobOutput = data.data;
		}
	}
}

console.log('Job output:', jobOutput);

Hope this helps!

[arminus]

Thanks a lot for the explanations and also the example!

"My problem" is that this relies on the caller to implement its own async waiting loop, which is ok if the caller is implemented in code, but is a showstopper if called from something like https://grafana.com/grafana/plugins/yesoreyeram-infinity-datasource/ or similar.

Example: I have an n8n flow which provides a webhook to return the number of failed events in the last x hours as a simple number response. That flow calls the xyops API /api/app/search_jobs/v1, parses the response and returns a simple number for consumption through the Respond to webhook node. This is just an example, I'm actually looking into ways to potentially port (and centralize in xyops) "stuff" I run elsewhere.

[jhuckaby]

Ah, got it. Thank you for the grafana example. I understand the distinction now.

Yeah, so the current xyOps flow requires two requests:

1. Call the Magic Link API, which launches the job and immediately returns its ID and stream token.
2. Connect to stream_job and consume the SSE stream until the final job output arrives.

While stream_job does contain the final output data, it still requires the caller to understand SSE and manage that second, long-lived request. That does not help clients such as the Grafana Infinity data source, which expect a single conventional HTTP request followed by a regular response.

What you are describing is effectively a synchronous job HTTP API endpoint: launch a job, wait for it to complete, and then return its output data directly in the HTTP response. That is comparable to n8n's "Respond to Webhook" behavior.

xyOps does not currently have an API that combines those steps into one request. I agree that this would be useful, particularly for short-running jobs used as lightweight API endpoints. We would need to account for normal HTTP timeout limits, but the use case makes sense and fits the overall strategy.

Adding to the TODO list! I'll try to get this into the next release.

[jhuckaby]

Added in xyOps v1.0.88 🚀

Check out the updated API docs to see how to use the feature:

• run_event
• magic

Basically, just add /wait to the URLs, and both of the APIs will "wait" for the job to complete before returning, AND the HTTP response will contain the FULL Job object, including all output data and files 👍🏻

[arminus]

Ok, I'm confused. I have a simple test flow which a shell script like so:

#!/bin/sh

stdbuf -o0 echo "Hello"

yes, I remember #109 ;-)

What I get in return for /api/app/magic/v1/fgGX1gl5YvkWOEExbZIzszmfNKqH9f0AMfk3HsAszwasJT6nkAenBWtAFRWfiGSm/wait?test=test123 is this JSON:
response.json

and it doesn't hold the actual output (i.e. "Hello" as a property value somewhere), so far, I was only able to get that via /api/app/get_job?id=jmshqrtuwq3qrg6m in job.output

(I am on Version 1.0.88)

[jhuckaby]

Ohhhhh, hang on, wait a minute. You're using a workflow here. I had no idea we were talking about workflows.

Okay, so, the problem is, this is now two separate jobs. A workflow is a meta-job: It is one outer job (the main controlling job), and then you're spawning a sub-job inside the workflow. That's a separate job with its own triggers, actions, limits, input, output, database record, etc. etc.

In your case the magic link fires off the workflow, and waits for it to complete (which includes waiting for all sub-jobs), but the final workflow "result" (the top-level job response) doesn't contain the raw output from all of its children jobs. That's just not the way the raw output data is stored.

You have a number of options here:

First, you can use a plain "event" instead of a workflow. Regular events can have magic triggers just like workflows can. Just click the "Add Trigger" and select magic link when editing the event. The event can be set to the Shell Script Plugin like you have here. Then, as long as your raw job output text is less than 1 MB (configurable), it will be included in the JSON output.

Second, if you must use a workflow, you can have the sub-job output structured "data" instead of just raw text. Job output "data" from sub-jobs is actually bubbled up to the parent workflow job (and shallow-merged, latter prevails), so it can be passed to subsequent workflows, etc., so it WILL be included in the full parent output.

To output data instead of raw text, simply print JSON to STDOUT in this format from your sub-job:

echo '{ "xy":1, "data": { "mykey1": "myvalue1", "myarray": [0, 1, 2, 3] } }'

The data itself is freeform, and can be in any shape you want. Since it is encoded in JSON, you can actually include verbose multi-line text blobs here, just as long as the full encoded JSON message is compacted onto a single line.

Ref: https://docs.xyops.io/#Docs/plugins/output-data

[jhuckaby]

Okay, I made a change to the new /wait feature in both APIs. In the next version of xyOps (coming in a few days), if a workflow was launched with /wait, the APIs will now return all the sub-job records as well as the top-level workflow job record. The sub-jobs will be in a jobs array next to the job object. They will each include their full raw output, as well as code, description, data, files, and all the other job properties.

Note that the 1MB limit still applies here, as it is a global rule. If a single job outputs more than 1MB of raw text output to STDOUT (not data, just raw text output), it gets gzip-compressed and stored separately in a binary file in storage. If the job produces less than 1MB, the raw text is saved "inline" with the rest of the job data in JSON format, and will be included in these APIs.

While currently undocumented, you can theoretically increase the 1MB limit, if your jobs are outputting tons of text and you need to capture it with this method. The undocumented configuration property is named max_inline_job_log_size (top-level prop in /opt/xyops/conf/config.json), and it accepts human-readable byte values such as 2MB, 5MB, etc. I wouldn't push it too far, tho, as I've never tested it beyond 1MB.

[arminus]

To output data instead of raw text, simply print JSON to STDOUT in this format from your sub-job:

ok, this is interesting. So basically a JSON string on stdout lands in job.data but a plain string doesn't.

The format of the data object is freeform, and can contain whatever content you want.

I think this is a bit misleading, it could help to clearly state that the data object has to be a JSON like

{ "xy":1, "data": { <payload> } }

(what's the xy value stand for btw)

So with that, I don't quite understand the upcoming change

They will each include their full raw output

does "raw" imply that it can be anything? plain text, XML, binary, whatever?

Sorry for asking all these questions and digressing now, while I'm getting familiar with the workflow concept, I'm also contemplating if I could possible use this for data ingestion / ETL pipelines at work. (as compared to just scripting them in Python).

[arminus]

So with the help of NotebookLLM, I figured out the basic control-flow between shell nodes.
xyops-workflow-emsfyqwg62gyku6w.json

[jhuckaby]

No need to apologize for the questions. I think the confusion here is mostly because you are using shell scripts, while the job messaging interface underneath xyOps is JSON-based. Let me try to separate the different concepts more clearly.

First, this statement is not quite correct:

A JSON string on stdout lands in job.data, but a plain string doesn't.

It is not just any JSON string. To send structured job data to xyOps, your script prints one complete xyOps Wire Protocol Message as a single line on STDOUT. For example:

printf '%s\n' '{"xy":1,"data":{"message":"Hello","count":42}}'

The entire line has to be valid JSON because that line is the message envelope which xyOps reads and interprets. The nested data object is the freeform part. You can put any JSON-compatible structure you want inside it, but the outer message still needs the xy and data properties.

The xy property is the protocol version indicator. "xy": 1 tells xyOps, "this is an xyOps Wire Protocol v1 message intended for you." Without that property, even a valid line of JSON is treated as ordinary job output and is not interpreted as structured job data. The full protocol is documented here:

• xyOps Wire Protocol
• Plugin Output Data

So, from a shell job's point of view, there are two different kinds of output:

1. Regular output, such as echo "Hello", is captured as the job's raw text log.
2. A single-line JSON message containing "xy":1 is interpreted by xyOps. If it contains a data object, that object becomes structured job output and can be passed to downstream workflow nodes.

When I said the upcoming API response will include each sub-job's "raw output," I meant the captured text log for that job. Plain text, XML, and unrecognized JSON can all appear there as text. Binary data should not be written directly to STDOUT. For binary content, or any content which should remain a file, write it to disk and return it as a job output file instead. The response keeps raw output, structured data, and files as separate things.

There is one other detail which may simplify your shell workflow quite a bit. Parameters supplied to the outer workflow, such as user fields populated through a Magic Link, are passed to every sub-job. A full plugin can read them from the job JSON on STDIN under workflow.params, but the Shell Plugin also exposes them as environment variables with a workflow_ prefix.

For example, if your workflow has a user field whose ID is test, and you call the Magic Link with ?test=test123, a shell node inside that workflow can read it like this:

#!/bin/sh

printf 'The workflow test parameter is: %s\n' "$workflow_test"

If the field ID were hours, it would be available as $workflow_hours, and so on. This works in every sub-job in the workflow, whether or not the nodes are directly connected. You do not need to parse the JSON on STDIN just to access those values.

If you want to return that value as structured data, use a JSON-aware encoder so quotes and other special characters are escaped correctly. For example, if jq is available on your servers, you can do this:

jq -cn --arg test "$workflow_test" '{xy:1,data:{test:$test}}'

The environment variable reference is here, and it also lists the other values available to jobs, including incoming data, shared workflow data, server data, event parameters, secrets, and job metadata:

• Plugin Environment Variables

For shell-based workflows, a useful mental model is:

• Use $workflow_NAME for user parameters supplied when the outer workflow was launched.
• Use $data_NAME for data received from the directly connected upstream node.
• Use $workflow_data_NAME for values in the workflow's shared workflowData object.
• Use normal STDOUT/STDERR for human-readable logs.
• Print a single-line {"xy":1,"data":{...}} message when you want to pass structured results to xyOps or to another node.
• Return larger or binary payloads as output files.

And yes, xyOps workflows can be a good fit for data ingestion and ETL orchestration. The individual steps can remain ordinary shell, Python, Node.js, or other tools. xyOps provides the scheduling, parameters, execution graph, retries, limits, logging, and movement of structured data and files between those steps.

Sorry for the confusion, and hope this helps clear things up a bit.

[jhuckaby]

Side-Note: xyOps v1.0.89 is now live, which has a key change to the run_event and magic API responses. If you launch a workflow with /wait, the API response will now include all the workflow's sub-job Job objects in a jobs array, which all will include the raw sub-job STDOUT output strings, if you need them.

This is a brand new code change and has nothing to do with outputting or fetching structured JSON data. This change is specifically about capturing and including all the raw STDOUT strings of all sub-jobs, and including those in the API response when /wait is used.

Note that the 1MB per-job limit discussed above still applies here.