xyOps conductor in Docker + xySat-legacy workers as systemd services on one RHEL 7.9 VM

[vishalr-nrift]

Context:
We're using xyOps to orchestrate startup/shutdown and monitoring of a mixed estate on a single RHEL 7.9 VM — JVM-based application services, shell scripts, Redis, and an ELK stack. The jobs need real access to the VM's filesystem (installed application directories, config, logs) to actually manage these services, not just trigger them remotely.

Why the conductor is a Docker container, not a native Linux service:
RHEL 7.9 ships glibc 2.17 across every minor release. xyOps requires Node 20+, and one of its native dependencies (better-sqlite3) needs a C++20-capable compiler to build from source — RHEL 7.9's toolchain (GCC 4.8.5) predates C++20. A native npm install of xyOps on this VM isn't viable as a result. Running the conductor as a Docker container sidesteps this, since the image bundles its own compatible Node/glibc userspace.

Why the workers are xySat-legacy as native systemd services, not Docker:
We initially tried xySat as Docker containers on the same VM, but they couldn't reach the scripts and application directories mounted at various locations on the host — our jobs need to run real shell scripts and start/stop JVM/Redis/ELK services directly against the host filesystem, and container path isolation got in the way. We moved to xySat-legacy (the glibc-2.17-compatible build for RHEL 7/CentOS 7-class systems), installed natively as systemd services, so jobs execute directly in the host environment.

Current architecture:

• 1 xyOps conductor — Docker container, on the VM
• 2 xySat-legacy workers — native systemd services, on the same VM

Questions:

1. Is this the right architecture for our use case (a scheduler that needs full host-level access to run shell scripts and manage JVM/Redis/ELK services), or is there a better-supported pattern we're missing?

2. We currently run two xySat-legacy instances on one VM. The docs/README note that xySat installs to a fixed path and that a second agent on the same host isn't the standard supported pattern — the suggested approach is one satellite per host, with workload separation handled via server groups. Is running two instances on a single host (at different install directories) targeting one conductor a supported configuration, or should we consolidate to a single satellite and use server groups instead?

3. How actively maintained is the xysat-legacy build relative to the main xySat repo? Since it's a separate wrapper repo that rebuilds from xySat's main branch only when a new tag is pushed there, is there a reliable cadence for keeping it in sync with new xyOps/xySat releases, or should we expect some lag on new features?

We're aware RHEL 7 is EOL and this is a deliberate stopgap while we plan a RHEL 8/9 migration — mainly looking to confirm we're not building on a pattern that will bite us before that migration happens. Appreciate any guidance.

[jhuckaby]

Hi @vishalr-nrift! Thanks for explaining your setup so clearly. Given the host access your jobs need, a Docker-based xyOps conductor with a native xySat Legacy service is a reasonable approach for your migration period. I would make one change, though: consolidate the two satellites into a single installation at the standard path.

The conductor and satellite serve different purposes. The conductor handles scheduling, orchestration, storage, and the UI, while xySat executes jobs and collects monitoring data on the worker. Your conductor does not need access to all those application directories for jobs executed by a native satellite. Those jobs run on the host, with access governed by the satellite's and job process's permissions.

A containerized satellite can access host directories through bind mounts, but filesystem access is only part of your requirement. Managing host services also involves the host's service manager and process environment. For your JVM, Redis, ELK, and shell-script workloads, a native satellite keeps that arrangement straightforward. There is no need to put the satellite back into Docker just to match the conductor.

Running two xySat installations simultaneously on the same host is not a supported configuration. It can be made to work with sufficient customization, but it adds complexity without providing additional execution capacity or availability.

A single xySat can run many jobs in parallel. There is no default per-server concurrency limit, so the practical capacity depends on the VM's CPU, memory, disk, and the jobs themselves. You can configure Max Jobs Per Server to protect the VM, and pair that with queue limits so excess work waits for capacity. See Max Jobs Per Server.

For workload organization, the same server can belong to multiple server groups. For example, you could have application, Redis, and ELK groups that all include this VM, then organize the corresponding events into categories. Groups provide logical targeting and monitoring views; they do not create separate execution environments or reserve resources for each workload. Event and category limits are useful for controlling concurrency, while workflows can express startup and shutdown dependencies. The Server Groups and Limits documentation covers those options.

If your reason for having two satellites is to run different jobs under different Unix accounts, that can also be handled with one satellite through plugin execution credentials, provided the satellite has permission to switch to the required accounts. Separate satellite installations are not necessary for that.

The custom installation directories are also significant because the built-in upgrade system expects the standard installation. On Linux, the upgrade script explicitly targets: /opt/xyops/satellite/

It also expects the standard xysat.service unit when restarting through systemd. It does not discover arbitrary installation directories or custom service names. Consequently, a satellite may execute jobs successfully from a custom location while UI-driven upgrades fail to update that installation correctly. You can see these assumptions in the upgrade script.

xySat Legacy is maintained alongside the main xySat release. It is a full mirror of the satellite application, built from the same source and dependencies. Our normal release process automatically triggers the legacy build using the matching xySat version number, so the intention is to keep both in lock-step with feature parity.

The separate repository exists to produce compatible packages. For Linux x64, its principal difference is the bundled Node.js runtime, which is compiled against glibc 2.17. Other packaged platforms use official Node.js binaries. You should not expect a separate feature backlog or a slower feature-release cadence for Legacy. There can be a short delay while its build finishes, and occasional build failures can require attention, but that is different from maintaining an older version of the application. The legacy build workflow shows how it fetches and packages xySat.

The unofficial Node.js runtime's documentation describes several caveats you should be aware of:

• The project is experimental, with no guarantee of consistent output or continued existence.
• Its builds receive minimal or no testing compared with official Node.js releases. Some target platforms are outside the official testing infrastructure.
• Builds can fail, and failures on experimental platforms do not block official Node.js releases. Compatible binaries may therefore be delayed or unavailable.
• Maintenance depends on community contributors keeping build recipes and toolchains working as Node.js requirements change.
• For production use, upstream recommends mirroring the executables rather than relying entirely on its download service.

These are limitations of the runtime distribution on legacy platforms, even though the xySat application has feature parity. See the unofficial-builds documentation.

The xySat version and bundled Node.js version are separate. The current satellite packaging uses Node 22.22.0; rebuilding xySat does not automatically select the newest Node patch release. Node 22's scheduled upstream end-of-life is April 30, 2027, according to the Node.js release schedule. That date is useful for migration planning, but it is not a promise that unofficial builds or RHEL 7 compatibility will remain available until then. The legacy runtime also does not address support or security updates for the underlying OS.

A few practical suggestions for your interim setup:

• Keep the conductor configured to use the Legacy release URLs. This matters for future installations and upgrades, so they continue receiving the compatible Linux x64 package. The Legacy README lists the settings.
• Use a stable conductor address reachable from the host. Publish the Docker port appropriately, and check that generated satellite connection settings do not point to a container-only hostname.
• Manage long-running application services through systemd where possible. Let xyOps jobs start, stop, and verify those services, while systemd owns their ongoing lifecycle.
• Set concurrency and queue limits for this VM. One satellite provides ample parallel execution, but your applications, conductor, and jobs still share the same resources.
• Keep backups and retain a known-working Legacy package. Test upgrades against representative jobs before relying on them for important startup or shutdown operations.

Consolidating the satellites now should make both day-to-day operation and the eventual migration much easier!

[vishalr-nrift]

Thanks for the detailed response earlier — really appreciate it!

Sorry, but one more question from my side.

We've been using xyOps jobs to start multiple auxillary system supporting services like Redis, Sentinel, ZooKeeper, ActiveMQ and Elasticsearch — things like calling zkServer.sh start. All of these jobs pass successfully almost immediately, but the underlying service isn't reflected as "running" in xyOps, even though no error or exit code is ever captured.

Is there any way to make the job stay in a running state in the workflow for as long as the underlying process is actually running, and only succeed or fail based on the real exit code when the process stops — either manually or due to an error?

Appreciate any guidance on the recommended pattern for this kind of long-running/daemon-style job.

[jhuckaby]

No need to apologize, this is a useful question!

What you are seeing is expected when a startup script launches a daemon and then exits. xyOps tracks the startup script, so a successful job means that command completed successfully. It does not automatically track the detached service's lifetime or capture its eventual exit code.

Yes, you can keep the job running for the service’s lifetime by launching it in foreground mode. Use the service's foreground command in the Shell Plugin, ideally with exec, so the script stays attached until the service exits. The Shell Plugin can then report its exit status. Review runtime limits and abort behavior for this arrangement. Also, workflow steps waiting for that job's success will wait until the service stops.

For your Redis, ZooKeeper, ActiveMQ, and Elasticsearch services, I would generally recommend the systemd approach mentioned earlier:

• Use short Start, Stop, Restart, and Status jobs.
• Verify readiness before allowing dependent workflow steps to proceed.
• Monitor process presence and service health independently, with alerts and optional recovery jobs.

That lets your startup workflow finish once the services are ready, while monitoring continues throughout their lifetime. A process being present does not necessarily mean the service is healthy.

Your question helped inspire a new Wiki: Managing Background Services. It walks through both approaches with examples, including monitoring and automatic recovery. For HTTP-based health checks, it also links to our Web Service Health Alert recipe.

Thanks for raising this!