Dispatcher uses a private CA for all mTLS trust. The CA is created once on the dispatcher host. All agent certificates are signed by this CA. Neither the dispatcher nor any agent trusts certs from public CAs or other sources for the operational port - only certs from the private CA are accepted.

The CA private key never leaves the dispatcher host. Loss of the CA key means all agent certs must be reissued via re-pairing.

Port 7443 - operational (mTLS) : All run, ping, capabilities, and cert renewal traffic. Both dispatcher and agent must present a valid cert signed by the private CA. SSL_verify_mode => SSL_VERIFY_PEER is set on both sides - there is no fallback to unauthenticated. An agent with no cert, an expired cert, or a cert from a different CA cannot connect.

Port 7444 - pairing : Bootstrap port. The agent has no cert yet when it first connects, so client cert is not required. This is a deliberate bootstrap exception. Mitigations: the operator reviews the displayed hostname and IP before approving; the port is only open while pairing-mode is actively running; a random nonce prevents misrouted or replayed approvals (see below). Close the pairing port promptly after completing the pairing session.

Port 7445 - API : No mTLS. All endpoints pass through the auth hook. The default behaviour when no hook is configured is controlled by api_auth_default in dispatcher.conf. The shipped default is deny - all requests are rejected until a hook is configured. Set to allow only on isolated networks where no credential checking is needed.

The API binds to 127.0.0.1 by default (api_bind in dispatcher.conf). It is not reachable from the network unless api_bind is explicitly changed. For internet-facing deployments, place the API behind a reverse proxy that handles authentication and TLS termination rather than binding directly to an external interface.

Pairing code verification : Each pairing request includes a 6-digit confirmation code derived from a SHA-256 hash of the agent's CSR. The agent displays this code at submission time. The dispatcher displays the same code alongside the hostname and IP in the approval prompt. The operator verifies both sides match before approving. This closes the social engineering path where an attacker submits a CSR with a spoofed hostname and the operator approves without checking the source. The code is computed independently on both sides from the CSR content - no extra network round-trip is required.

Nonce verification : Each pairing request includes a 32-hex-character random nonce generated by the agent. The dispatcher stores it with the pending request and echoes it in the approval response. The agent verifies the nonce matches before storing any certs. This prevents a race condition where an approval for a different concurrent pairing request is accepted by the wrong agent.

Preflight writability check : dispatcher-agent request-pairing checks that /etc/dispatcher-agent is writable before making any network connection. If not writable, it dies immediately rather than leaving a stale pending request in the dispatcher's pairing queue.

Stale request cleanup : Pending requests older than 10 minutes with no approval or denial are automatically deleted from the pairing queue.

Queue depth limit : The pairing queue is capped at 10 pending requests by default (configurable via pairing_max_queue in dispatcher.conf). Once the cap is reached, further connection attempts are rejected immediately with a structured error response. Stale expiry runs before the count is checked, so aged-out entries do not consume quota. This prevents a flood of pairing requests from an attacker across multiple source IPs from burying a legitimate request in list-requests output.

Operator review : The operator is the last line of defence for pairing. Always verify the hostname and source IP displayed in list-requests or the interactive prompt before approving.

Renewal uses the already-authenticated mTLS connection on port 7443. No new trust is established - the renewal exchange is carried entirely within a session that both sides have already authenticated.

The agent reuses its existing private key across renewals. Only the cert is replaced. This preserves key continuity and means a renewal does not require generating or protecting new key material.

The dispatcher only renews certs for hosts in its registry. An agent that has been unpaired cannot receive a renewal.

Script name validation : All script names are validated against /^[\w-]+$/ before allowlist lookup. This pattern excludes /, ., spaces, and shell metacharacters. Path traversal is impossible at the name level.

No shell execution : Scripts are executed via exec { $path } $path, @args - the two-argument form that bypasses PATH lookup and invokes execve() directly. No shell is involved. Shell metacharacters in arguments have no effect.

Script directory restriction : With script_dirs set in agent.conf, any allowlist entry pointing outside an approved directory is rejected at load time. The check is repeated at execution time, guarding against allowlist modifications between agent startup and a run request. When not set, any absolute path is accepted.

Allowlist is server-enforced : The allowlist is validated on the agent, not trusted from the dispatcher request. A dispatcher cannot request a script that is not in the agent's allowlist regardless of what it sends.

JSON context on stdin : Scripts receive full request context as JSON on stdin (script name, args, reqid, peer IP, username, token, timestamp). The agent writes this context with a non-blocking write loop and a configurable timeout (stdin_timeout in agent.conf, default 10 seconds). If the script does not read stdin and the pipe buffer fills, the agent logs ACTION=stdin-timeout and closes the write end, delivering EOF to the script. The script continues to execute. Scripts that do not use stdin context do not need any special handling.

The auth hook is called before every run, ping, capabilities, and API request. It is the sole access control policy engine - dispatcher has no built-in ACLs.

Default auth mode : When no hook is configured, behaviour depends on the caller. CLI invocations (dispatcher run, dispatcher ping) unconditionally pass - CLI access is already gated by system user and group permissions. API callers are governed by api_auth_default in dispatcher.conf. The default is deny - all API requests are rejected without a hook. Set to allow for isolated networks where credential checking is not needed. This setting has no effect when a hook is configured.

username is advisory : The username field is a caller-supplied string. Dispatcher does not authenticate it or verify it matches any local or remote account. It is forwarded unchanged to the hook and to the agent. Its intended purpose is to carry an identity assertion that the hook can forward to an external authentication service alongside the token - the service validates whether the claimed identity is consistent with the token's authority. A hook that grants elevated permissions based solely on username without validating it via the token or an external mechanism can be bypassed by any caller that sets the field to a privileged value. See SECURITY-OPERATIONS.md for the recommended pattern.

Argument inspection : Always use DISPATCHER_ARGS_JSON in hook scripts to inspect script arguments. This is a reliable JSON array. DISPATCHER_ARGS (space-joined) is set for backward compatibility but is deprecated - it is lossy for arguments containing spaces or newlines, and naive pattern-matching on it can be bypassed by crafted argument values.

Token forwarding : Tokens are included in the JSON payload sent from the dispatcher to the agent, and in the JSON context piped to scripts on stdin. This supports token validation at every stage of an execution pipeline. Each hop can independently verify the token is still valid and still authorised for the stated purpose, without trusting the previous hop.

Hook isolation : The hook executable is run via fork/exec. local $SIG{CHLD} = 'DEFAULT' is set before forking to prevent the API server's SIGCHLD reaper from collecting the hook process before waitpid can. stdout and stderr of the hook are redirected to /dev/null - output from the hook does not reach the caller.

Token logging : Tokens are never logged by the dispatcher or the agent. They appear in the hook's environment and in JSON stdin. Do not log environment variables within the hook; log only specific fields from stdin. A hook that logs env output exposes the token in syslog.

Token in CLI : Pass tokens via DISPATCHER_TOKEN environment variable rather than --token to prevent the value appearing in ps output.

Hook must not produce output : The hook's stdout and stderr are discarded. Audit logging within the hook should use syslog.

Agent-side hook : Agents can independently run their own auth hook, configured via auth_hook in agent.conf. This runs after allowlist validation on the agent, before the script is executed. It covers run requests only - ping requests do not invoke the agent hook. The hook receives the same request context including the forwarded username and token. It does not receive a hosts field; the agent is unaware of which other agents are targeted in the same invocation. For source-based restriction on the agent, use allowed_ips in agent.conf or DISPATCHER_SOURCE_IP in the hook. If no agent hook is configured, the agent authorises unconditionally at the agent level, relying on mTLS and the allowlist as its primary controls.

/etc/dispatcher/ca.key              0600  root         CA private key
/etc/dispatcher/ca.crt              0644  root         CA certificate
/etc/dispatcher/dispatcher.key      0600  root         Dispatcher private key
/etc/dispatcher/dispatcher.crt      0644  root         Dispatcher certificate
/etc/dispatcher/auth-hook           0755  root         Auth hook executable
/etc/dispatcher/                    0750  root:dispatcher

/etc/dispatcher-agent/agent.key     0640  root:dispatcher-agent
/etc/dispatcher-agent/agent.crt     0640  root:dispatcher-agent
/etc/dispatcher-agent/ca.crt        0644  root
/etc/dispatcher-agent/agent.conf    0640  root:dispatcher-agent
/etc/dispatcher-agent/scripts.conf  0640  root:dispatcher-agent
/etc/dispatcher-agent/              0750  root:dispatcher-agent

/opt/dispatcher-scripts/            0750  root:dispatcher-agent
/opt/dispatcher-scripts/*.sh        0750  root:dispatcher-agent

/var/lib/dispatcher/                0770  root:dispatcher
/var/lib/dispatcher/pairing/        0770  root:dispatcher
/var/lib/dispatcher/agents/         0770  root:dispatcher
/var/lib/dispatcher/locks/          0770  root:dispatcher

The dispatcher-agent system user has no login shell and no home directory. The dispatcher group grants CLI access to non-root operators.

The agent and API systemd units apply the following restrictions:

NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes

The API unit additionally sets ReadWritePaths=/var/lib/dispatcher to restrict filesystem write access to the runtime directory only.

The agent unit applies additional containment:

SystemCallFilter=@system-service
SystemCallFilter=~@privileged @resources
CapabilityBoundingSet=
AmbientCapabilities=
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
RestrictNamespaces=yes
LockPersonality=yes
MemoryDenyWriteExecute=yes

CapabilityBoundingSet= (empty) drops all capabilities from the bounding set, preventing any allowlisted script from acquiring capabilities regardless of file capability bits. MemoryDenyWriteExecute=yes is safe for the current bash-only script inventory but must be removed if a JIT-compiled runtime (Java, Node.js, Python with JIT) is added to the allowlist - there is no detection mechanism for this conflict at load time; the operator must review this directive when adding new allowlist entries. AF_UNIX is required in RestrictAddressFamilies because the agent connects to /dev/log via a Unix domain socket to deliver syslog messages - omitting it silently blocks all logging.

At pairing time, the dispatcher includes its own cert serial in the approval payload. The agent stores this at /etc/dispatcher-agent/dispatcher-serial. On every /run, /ping, and /capabilities request, the agent compares the incoming peer cert serial against the stored value and rejects any mismatch with a 403. If no serial file is present, /run and /ping deny all requests; /capabilities retains a warn-and-allow path for agents that have not yet been re-paired after serial tracking was introduced.

/renew and /renew-complete are intentionally exempt from the serial check. During cert rotation, the dispatcher presents its new cert before agents receive the updated serial. Applying the serial check to the renewal endpoints would cause every agent to reject the serial broadcast, breaking the rotation mechanism. The renewal endpoints require a valid CA-signed cert (mTLS still applies) but do not require serial match. See Cert Rotation below.

The window during which these endpoints are reachable by any CA-signed cert is the duration of the serial broadcast - on a well-connected fleet this is seconds; on a slow fleet it is bounded by the dispatcher's connection timeout. A host with a valid CA-signed cert calling /renew-complete during this window could attempt to deliver a replacement cert, but only if it already holds a cert signed by the private CA. If an attacker has a CA-signed cert, /run is the more direct path; /renew abuse does not represent a meaningful escalation.

The dispatcher cert is renewed automatically before expiry. The renewal process and the serial tracking work together to rotate credentials without disrupting the fleet:

Renewal trigger : The dispatcher checks its own cert expiry at startup and every 4 hours (configurable via cert_check_interval). When fewer than cert_renewal_days remain (default: 90), it generates a new cert automatically.

Broadcast : Immediately after generating the new cert, the dispatcher calls update-dispatcher-serial on all registered agents in parallel, passing the new serial as an argument. The script writes the file and sends SIGHUP. Agents that respond successfully are marked current in the registry.

Overlap window : Agents that were offline during the broadcast are marked pending. The dispatcher retries them on each subsequent check interval. After cert_overlap_days (default: 30, configurable) the overlap expires and any remaining pending agents are marked stale. A stale agent needs re-pairing - it has missed the rotation window. /run and /ping deny with 403 once the serial no longer matches; /capabilities warns but allows.

dispatcher serial-status : Shows the current and previous dispatcher serial, rotation timestamp, overlap expiry, and per-agent serial state (current/pending/stale/unknown). Use this to identify agents that need attention after a rotation.

dispatcher rotate-cert : Manual rotation trigger. Runs the same logic as the automatic check, broadcasts immediately, and reports per-agent results. Use after a suspected compromise or to test the rotation path.

Dispatcher re-keying : Running setup-dispatcher again generates a new cert with a new serial and marks all agents as pending. The dispatcher binary checks the registry before proceeding and displays the number of agents that will require re-pairing if the overlap window is missed.

update-dispatcher-serial script : Installed by the agent installer at /opt/dispatcher-scripts/update-dispatcher-serial. Must be enabled in the agent's scripts.conf for automatic rotation to work. Agents without this entry in their allowlist will not receive serial updates and will need re-pairing when the overlap window expires.

The agent maintains a revocation list at /etc/dispatcher-agent/revoked-serials (path configurable via revoked_serials in agent.conf). On every incoming mTLS connection, after the handshake verifies the CA signature, the peer cert serial is checked against this list. A revoked cert is rejected with a 403 and a syslog warning before any request is processed.

The file contains one serial per line. All of the following formats are accepted and normalised to lowercase hex on load:

• Plain hex: deadbeef
• Colon-separated: DE:AD:BE:EF (as returned by some tools and IO::Socket::SSL)
• 0x-prefixed: 0xdeadbeef
• serial=-prefixed: serial=DEADBEEF (direct output of openssl x509 -serial)
• Decimal integer: 3735928559

Lines beginning with # are treated as comments. A missing or empty file means no certs are revoked - the normal state for a new installation.

The revocation list is loaded at agent startup and reloaded on SIGHUP without restarting the agent or dropping active connections:

systemctl reload dispatcher-agent

To revoke a cert:

1. Obtain the serial: openssl x509 -noout -serial -in /etc/dispatcher/dispatcher.crt
2. Append the output directly to /etc/dispatcher-agent/revoked-serials on each affected agent - no format conversion needed, serial=DEADBEEF is accepted as-is
3. Reload: systemctl reload dispatcher-agent

For fleet-wide revocation, run a dispatcher script that appends the serial and sends SIGHUP on each agent. A revoke-cert script is a natural entry in the agent allowlist for this purpose.

Unpairing and revocation : dispatcher unpair <hostname> removes the agent from the registry and prevents further cert renewal. The agent cert remains technically valid until natural expiry. To immediately close this window, add the agent cert serial to the dispatcher's own revocation check (if implemented) or decommission the host promptly. The revocation list on the agent only covers certs presented to the agent - it does not prevent a stolen agent cert from connecting to the dispatcher.

What revocation closes : A compromised dispatcher cert that has been revoked cannot connect to any agent that has been updated, even though it was signed by the CA and has not expired. A compromised agent cert cannot be revoked via this mechanism on the dispatcher side - that requires the dispatcher-side equivalent (serial tracking work, next phase).

What revocation does not close : A compromised cert that reaches an agent before the revocation list is updated. The list is only as current as the last SIGHUP. For time-critical revocation, restart the agent service rather than reloading - the connection is still rejected on reconnect but any in-flight connection from a revoked cert may complete if it was established before the reload.

The CA key is the root of trust for the entire deployment. If it is compromised, an attacker can issue valid agent certs and connect to any agent.

Recommended practices:

• Back up /etc/dispatcher/ca.key to encrypted offline storage immediately after setup-ca
• Restrict access to the dispatcher host itself; the CA key should not be accessible over the network
• For redundant dispatcher deployments, transfer the CA key over an encrypted channel with host key verification (scp with known_hosts, not StrictHostKeyChecking=no)
• Audit access to the dispatcher host via system auth logs

Connection rate limiting : The agent tracks connection attempts per source IP in memory. A source that establishes more than 10 connections within 60 seconds is blocked for 5 minutes (volume threshold). A source that causes more than 3 TLS handshake failures within 600 seconds is blocked for 1 hour (probe threshold). Blocks are held in the agent process memory and cleared on SIGHUP. The block state is logged at the point the threshold is crossed; repeat checks are silent. Rate state is not persisted across reloads - update-dispatcher-serial sends SIGHUP as part of normal rotation and clears all rate blocks as a side effect. A legitimate dispatcher that triggers the probe block due to a cert misconfiguration can be unblocked immediately with systemctl reload dispatcher-agent. Thresholds are configurable via rate_limit_volume and rate_limit_probe in agent.conf.

IP allowlist : If allowed_ips is set in agent.conf, the agent enforces an IP allowlist before the rate check and before any request is processed. Any source IP not in the list is rejected immediately with the connection closed and an ACTION=ip-block syslog entry. The allowlist supports exact IPs and /8, /16, /24 CIDR prefixes. Invalid entries are filtered at load time with a warning; the agent starts normally with the remaining valid entries. When allowed_ips is absent, all source IPs are permitted.

TLS version and cipher restriction : The agent accepts TLS 1.2 and TLS 1.3 only. TLS 1.0 and 1.1 are rejected. For TLS 1.2 connections, only ECDHE cipher suites using AES-256-GCM or ChaCha20-Poly1305 are permitted. CBC mode, RC4, export-grade, and anonymous ciphers are excluded. TLS 1.3 cipher suites are governed by OpenSSL defaults, which are all AEAD on current releases.

Request body size limit : The agent limits incoming request bodies to 1 MB. Any request declaring a Content-Length above this threshold is rejected with HTTP 413 before the body is read. No legitimate Dispatcher request approaches this limit - the largest legitimate body (a /renew-complete cert delivery) is under 10 KB.

HTTP header count limit : The agent limits incoming requests to 32 header lines. Any request exceeding this is rejected with HTTP 431 before the body is read or any handler is invoked. This applies to all connections including those from peers with a valid CA-signed cert.

Argument validation scope : Dispatcher does not validate script argument values. Validation of arguments is the responsibility of the allowlisted script itself and the auth hook. The hook receives arguments as a JSON array via DISPATCHER_ARGS_JSON and on stdin; the script receives them directly as @ARGV. Neither path involves shell interpretation.

The API binds to 127.0.0.1 by default and is not reachable from the network without an explicit api_bind change in dispatcher.conf. For any deployment where the API is reachable from outside the dispatcher host:

• Configure an auth hook. Without a hook, api_auth_default = deny blocks all requests. Set api_auth_default = allow only on isolated networks.
• Place the API behind a reverse proxy (nginx, caddy) that handles TLS termination and request rate limiting. The API has no built-in rate limiting and no host count cap - a reverse proxy is the appropriate layer for both.
• Enable TLS on the API by setting api_cert and api_key in dispatcher.conf - use a cert from a public CA if external clients will not have the private CA cert.

GET /health bypasses auth and returns the API version string. This endpoint is publicly accessible on any externally-bound deployment. Version disclosure is low risk in a private deployment but should be considered for internet-facing deployments - place the API behind a proxy that strips or restricts the /health path if version disclosure is a concern.

For operational security guidance (monitoring, incident response, known limitations, Docker-specific notes), see SECURITY-OPERATIONS.md.

SECURITY-OPERATIONS.md covers the operational side of a running deployment: monitoring and alerting recommendations, dispatcher host security requirements, token and credential lifecycle, auth hook operational guidance, CA compromise recovery procedure, known limitations, and Docker-specific security notes.