Skip to content

two-way-delete-tutorial.md

Latest commit

 

History

History
262 lines (187 loc) · 17.4 KB

two-way-delete-tutorial.md

File metadata and controls

262 lines (187 loc) · 17.4 KB
Raw

Two-Way Git Sync with Event-Driven Deletes

This tutorial keeps Events, workflows, and Plugins in both xyOps and Git. Existing definitions can be edited on either side, and deletions can start on either side. New definitions are created in the xyOps UI and exported to Git. Every sync runs because Git accepted a push or xyOps recorded an activity. There are no cron jobs or recurring sync schedules.

Use xyCLI version v1.1.1+ (with down-sync delete support) and xyOps version v1.1.4+ (with System Activity triggers and exclusions). Both features must be available before you enable this workflow.

Important

Two-way deletion requires authoritative triggers in both directions. Run --up --delete only in response to an accepted Git change, and run --down --delete only in response to xyOps activity. Each invocation has exactly one authoritative side. A single command with both --up and --down cannot use --delete.

Authoritative action Trigger Command Result
Git accepts a contributor push Git post-receive hook or GitHub push webhook xy sync ... --up events,plugins --down false --delete events,plugins Git updates existing definitions and removes missing ones from xyOps.
xyOps updates or deletes an Event or Plugin System Activity Event xy sync ... --up false --down events,plugins --delete events,plugins xyOps updates existing files and removes missing XYPDF files and their owned neighbors.
xyOps creates an Event or Plugin Separate System Activity Event xy sync setup events plugins --new ... New definitions are exported, committed, and pushed to Git.

Caution

There is a potential race condition between creating an item in xyOps and exporting it to Git. If a Git-triggered up-delete runs before the new XYPDF file reaches the checkout, it can delete that newly created item from xyOps. The shared job queue prevents runs from overlapping, but it cannot guarantee which trigger arrives first. See Ordering and recovery.

The examples in this tutorial include Events (which includes Workflows) and Plugins. To manage other resource types, add them to the commands and choose their corresponding activity types.

Before you begin

Use one persistent Git checkout on a machine running xySat. Target all three xyOps Events below to that machine, where the checkout, xyCLI configuration, and Git credentials are available. The xyOps conductor does not need a copy of the repository. Use one operating-system account for all three jobs.

You will need:

  • A dedicated xyOps API Key for xyCLI that can see the complete Event and Plugin inventories, update and delete those types, and update global sync state. A restricted or partial inventory is unsafe for up-only deletion. You will select this same key in the System Activity trigger exclusions below.
  • A dedicated repository and a persistent checkout, such as /srv/xyops-automation, with noninteractive Git pull and push access.
  • A Git author identity for commits generated by xyOps.
  • Shared xyOps job capacity and queue limits on all three Events.
  • A way to monitor failed xyOps jobs and Git webhook deliveries, then retry them after checking both sides.

This is a full-inventory workflow. Do not point it at a partial export, empty mount, fresh clone missing files, or API Key with an incomplete view. Stock and Marketplace definitions are naturally excluded.

1. Prepare the checkout

Clone the repository on the sync worker, then configure the account that will run the jobs:

git clone git@github.com:YOUR_ORG/xyops-automation.git /srv/xyops-automation
cd /srv/xyops-automation
git config user.name "xyOps Sync"
git config user.email "xyops-sync@example.com"
xy config --base_url https://xyops.example.com --api_key YOUR_API_KEY
xy dashboard

Keep the API Key and Git credentials outside the repository. Use main as the dedicated branch in the examples below, or replace it consistently with your branch name.

Export a complete starter inventory from the repository root. --file_props keeps common Plugin and Event scripts in neighboring files:

xy sync setup events plugins --file_props script,params.script --dry
xy sync setup events plugins --file_props script,params.script
git status --short

Review the generated files, including scripts and potentially sensitive configuration. Confirm that every non-stock Event, workflow, and Plugin visible to the sync API Key is present. Then commit and push the initial tree:

git add --all
git commit -m "Export xyOps automation"
git push -u origin main

Create the automation Events described below only after this inventory exists. Export them before enabling the Git trigger, so an early up-only delete pass cannot mistake an automation Event for a Git deletion.

2. Add a commit helper

Save this script as /usr/local/sbin/commit-xyops-sync on the sync worker. Both xyOps-originated jobs will use it as their explicit --down_cmd, to run after local changes have been made:

#!/bin/sh
set -eu

cd /srv/xyops-automation

# This checkout is dedicated to xyOps automation. Stage removals as well as
# new and updated files, then avoid an empty commit if nothing changed in Git.
git add --all
if git diff --cached --quiet; then
	exit 0
fi

# The Git trigger should ignore commits carrying this prefix. They came from
# xyOps and must not cause a second authoritative up-delete run.
git commit -m "[xyops-sync] Export xyOps changes"
git push origin HEAD:main

Make it executable with chmod 755 /usr/local/sbin/commit-xyops-sync. This helper assumes the repository contains only the automation tree. If it contains unrelated work, narrow the staged paths and review the resulting commit before using it unattended.

3. Add one sync script for the three jobs

Save this as /usr/local/sbin/xyops-git-sync on the sync worker:

#!/bin/sh
set -eu

# Include the directories containing Node.js, xyCLI, and Git on this worker.
PATH="/usr/local/bin:/usr/bin:/bin"
export PATH

REPO_DIR="/srv/xyops-automation"
XY_BIN="/usr/local/bin/xy"
COMMIT_CMD="/usr/local/sbin/commit-xyops-sync"
SYNC_TYPES="events,plugins"
MODE="${1:-}"

cd "$REPO_DIR"

# Work only from the dedicated branch and a clean, existing checkout. A
# missing tree or leftover change must never be treated as a deletion.
if [ "$(git symbolic-ref --quiet --short HEAD)" != main ]; then
	echo "Sync stopped: checkout is not on main." >&2
	exit 1
fi
if [ -n "$(git status --porcelain)" ]; then
	echo "Sync stopped: checkout has uncommitted changes." >&2
	exit 1
fi
if [ ! -d events ] || [ ! -d plugins ]; then
	echo "Sync stopped: Event or Plugin inventory directory is missing." >&2
	exit 1
fi

# Pull accepted Git changes before scanning. A failed fast-forward requires
# review rather than a reset that could discard a xyOps-generated commit.
git pull --quiet --ff-only origin main

case "$MODE" in
	up)
		"$XY_BIN" sync "$REPO_DIR" \
			--up "$SYNC_TYPES" --down false --delete "$SYNC_TYPES" \
			--up_cmd false --down_cmd false
		;;
	
	down)
		"$XY_BIN" sync "$REPO_DIR" \
			--up false --down "$SYNC_TYPES" --delete "$SYNC_TYPES" \
			--down_cmd "$COMMIT_CMD" --cmd_timeout 120
		;;
	
	create)
		# Setup scans the current directory and adds only remote definitions
		# whose type and ID are not already represented in the checkout.
		"$XY_BIN" sync --setup "$SYNC_TYPES" --new \
			--file_props script,params.script \
			--down_cmd "$COMMIT_CMD" --cmd_timeout 120
		;;
	
	*)
		echo "Usage: $0 up|down|create" >&2
		exit 2
		;;
esac

Make it executable with chmod 755 /usr/local/sbin/xyops-git-sync. Change the paths and PATH for your worker. The directory checks are a basic guard against a missing checkout. They do not prove that the inventory is complete, so inspect it before enabling deletion and after any Git or API failure.

Set a suitable job timeout for Git network operations. All three Events will share a xyOps capacity pool so only one can use this checkout at a time.

4. Configure the xyOps Activity Events

Create three Shell Plugin Events in xyOps, initially disabled. Target only the sync worker and give each Event the indicated trigger and script. The System Activity trigger lets you select multiple activity types in the UI; it launches a job when any of them occur. Note that the xyOps scheduler must be enabled for Activity triggers to run.

Event Trigger Select these activities Shell script body
Sync deletions and updates from xyOps System Activity event_update, plugin_update, event_delete, plugin_delete /usr/local/sbin/xyops-git-sync down
Export creations from xyOps System Activity event_create, plugin_create /usr/local/sbin/xyops-git-sync create
Apply accepted Git pushes Manual No activity selection /usr/local/sbin/xyops-git-sync up

On each of the two System Activity triggers, open Exclude Users / Keys and select the dedicated xyCLI sync API Key configured on the worker in step 1. xyOps records the key's ID as the activity initiator and skips that trigger when the ID is excluded. This prevents an upsync API update or deletion from launching a redundant downsync job. Leave ordinary users out of the exclusion list so their UI changes still launch the appropriate job. If more than one API Key can make sync writes, exclude each of those keys on both triggers.

The update activities make xyOps UI edits flow back to Git too. The separate creation Event runs sync setup --new, because ordinary downsync updates existing local sources but does not create files for new xyOps definitions. The setup command's --down_cmd runs only when it writes files; the down-delete command's --down_cmd runs only after local changes. Neither creates an empty Git commit on a no-op run.

The Git Event has an enabled Manual trigger so a Git hook or verified webhook receiver can launch it with xy run or the run_event API. It has no schedule. The Git push is still the authoritative reason for that run.

On each of these three Events, set a Max Jobs Limit of 1 with the same Shared Capacity Key, such as xyops_git_sync. Also set a Max Queue Limit with the same positive amount on each Event, sized for the bursts you expect. This creates one shared, FIFO job queue for all sync operations. Without queue capacity, a trigger arriving while another sync job runs may be aborted instead of waiting. See xyOps's shared capacity and queue limits. Use the same priority on all three Events unless you deliberately want to change queue order, and monitor jobs that fail or exhaust the queue.

Before enabling any of the three Events, export their new definitions from the checkout root:

cd /srv/xyops-automation
/usr/local/sbin/xyops-git-sync create

The create mode uses sync setup --new and its --down_cmd to commit and push those files. Review that commit and confirm the tree is complete. Then enable the three Events and test with disposable definitions before allowing normal Git pushes to trigger up-delete.

5. Trigger upsync from accepted Git pushes

Choose the Git trigger that matches your repository hosting. Both options launch the same Git Event after a push has been accepted. Do not enable both for one repository.

Self-hosted Git: a post-receive hook

On a self-hosted bare remote repository, save the following as hooks/post-receive and make it executable. This hook runs on the Git server after refs have been updated, not in each contributor's clone or the sync worker's checkout. The Git server needs xyCLI credentials that permit launching the Git Event, but it does not need the sync checkout.

#!/bin/sh
set -eu

deploy=false
while read -r old_oid new_oid ref; do
	[ "$ref" = refs/heads/main ] || continue
	git cat-file -e "$new_oid^{commit}" 2>/dev/null || continue
	
	# A commit generated by the down/create helper is already from xyOps.
	# Do not turn its push into another authoritative up-delete run.
	case "$(git log -1 --format=%s "$new_oid")" in
		'[xyops-sync]'*) continue ;;
	esac
	deploy=true
done

[ "$deploy" = true ] || exit 0
/usr/local/bin/xy run SYNC_UP_EVENT_ID

Replace SYNC_UP_EVENT_ID with the xyOps Git Event's exact ID. Git's post-receive hook runs after a push succeeds; a later hook or job failure cannot undo that push. Monitor the launched job and retry it after fixing any problem.

GitHub: a push webhook

If GitHub hosts the repository, use its push webhook to notify your existing webhook receiver. In GitHub, open Settings > Webhooks > Add webhook, set its payload URL to that receiver, choose application/json, set a secret, and select only the Pushes event. The receiver should:

  1. Verify X-Hub-Signature-256 against the raw request body, and accept only the intended repository and refs/heads/main.
  2. Ignore GitHub's initial ping, deleted branches, and commits whose head message starts with [xyops-sync].
  3. Deduplicate by X-GitHub-Delivery, then launch SYNC_UP_EVENT_ID through xyOps's run_event API using an API Key with run_jobs access.
  4. Acknowledge only after recording the delivery for processing, and monitor the resulting xyOps job. Retry a failed job or redeliver a missed webhook after reviewing the Git and xyOps state.

The API request made by the verified receiver is a JSON POST to /api/app/run_event/v1 with an X-API-Key header and body {"id":"SYNC_UP_EVENT_ID"}. The Event runs on the sync worker, so the receiver does not need the checkout. Do not put an xyOps API Key in the GitHub payload URL. A direct xyOps Magic Link can start an Event, but it does not verify GitHub's webhook signature; use a receiver that does that verification for this workflow. See GitHub's webhook setup, signature validation, and delivery guidance.

6. Test each direction before enabling automatic deletion

Run these tests with disposable Events and Plugins, not production definitions:

  1. Create an Event or Plugin in xyOps. Confirm that the create Activity Event exports its XYPDF, commits it, and pushes it to Git.
  2. Edit that definition in the xyOps UI. Confirm that the down Activity Event downloads the change and commits it.
  3. Delete it in xyOps. Confirm that the down Event removes the XYPDF and owned neighbor files and pushes the deletion.
  4. Edit an existing definition in Git and push. Confirm that the accepted-push trigger runs up and updates xyOps.
  5. Delete a different existing definition in Git and push. Confirm that up deletes only that definition in xyOps.
  6. For both Git-initiated tests, confirm that the sync API Key's activity did not launch either System Activity Event. A second downsync job means the key is missing from a trigger's exclusion list.

Before the first live delete, preview each direction against the complete checkout and inspect every proposed deletion:

cd /srv/xyops-automation
xy sync . --up events,plugins --down false --delete events,plugins --dry
xy sync . --up false --down events,plugins --delete events,plugins --dry

Remember that a dry run previews the inventory at that moment. It cannot establish what a later Git push or xyOps activity means. Keep recoverable Git history and xyOps backups.

Ordering and recovery

Authoritative triggers tell each pass which side should win. They do not make a full-inventory sweep transactional or remove all races. For example, an Event newly created in xyOps but not yet exported looks missing to an up-delete run; a new Git XYPDF that has not been created in xyOps looks missing to a down-delete run. This tutorial permits creation only in xyOps, so contributors should edit or delete existing XYPDF sources, not add new IDs in Git.

Keep all three jobs on the same worker and checkout, serialize the entire Git plus xyCLI sequence with their shared capacity key, and process pending xyOps creations before allowing a Git up-delete pass to treat the inventory as complete. The queue prevents overlap, but it cannot guarantee that independently delivered Git and xyOps triggers arrive in causal order. If changes on both sides occur together, pause automatic deletion, inspect the Git diff and both inventories, resolve the intended order, and rerun the appropriate job. Do not assume a successful webhook acknowledgment means the xyOps job succeeded.

If a job or push fails, leave the checkout and xyOps as they are until you have inspected them. A failed Git push can leave a local commit that needs to be pushed or reconciled before another git pull --ff-only. A failed Activity launch or webhook delivery needs a deliberate retry; there is no cron pass to pick it up later. GitHub does not automatically redeliver failed webhook deliveries.

For all sync options and deletion safeguards, see the Filesystem Sync Guide. For a scheduled, modification-time-based workflow without automatic two-way deletion, see the Two-Way Git Sync Tutorial.