Skip to content

docs: add a why section to the SSH to SMF worker samples - #282

Open
leongdl wants to merge 5 commits into
aws-deadline:mainlinefrom
leongdl:ssh-why-section
Open

docs: add a why section to the SSH to SMF worker samples#282
leongdl wants to merge 5 commits into
aws-deadline:mainlinefrom
leongdl:ssh-why-section

Conversation

@leongdl

@leongdl leongdl commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Feedback on the recently published host-config and job bundle samples: the swap, Docker/NVIDIA, and sudo samples all explain why a customer would want them, but the SSH-to-worker samples jump straight into mechanics.

Changes

job_bundles/ssh_to_smf/README.md — new ## Why Shell Access to a Worker? section between the intro and ## How It Works, matching the placement used by swap_for_smf and docker_nvidia_container_toolkit. It covers:

  • the gap it fills (a worker has no public address and takes no inbound connections, so environmental failures are hard to diagnose from a log)
  • that Session Manager logs you in as ssm-user, not the job-user the job runs under, with sudo -u job-user -i to reach the job's own environment
  • six concrete uses: diagnosing a failing job in place, prototyping a host config script, port-forwarding interactive tools, confirming what the AMI includes, verifying network reachability from the worker subnet, and validating container/GPU setups
  • cross-links to the sudo_for_job_user and docker_nvidia_container_toolkit samples, which pair with this one
  • a cost note: the job holds the worker for the full SessionMinutes whether or not anyone is connected, so cancel the job to stop paying early

job_bundles/ssh_to_smf_windows/README.md — short parallel section that points at the Linux list rather than duplicating it, plus the GUI-only cases specific to Windows (licensing dialogs, plugin registration in a DCC UI), and a pointer to the existing job-user login steps.

Docs only. No script or template changes.

Review feedback addressed

The automated reviewer caught two factual errors in the first revision, both confirmed against the code and fixed in 3440678:

  • The section claimed the shell runs as job-user. The SSH config in the same file uses User ssm-user (README.md:140), and the Windows sibling says the same (line 188).
  • It advised letting the job finish to limit cost. job/template.yaml:107-118 sleeps the full SessionMinutes with no check for a connected session, so cancelling the job is the only way to stop early.

Vale was failing on the first revision and now passes. That included one pre-existing tricolon in the Windows README (the job-user login paragraph), reported because the PR touches that file; the rewrite there is unrelated to the new section.

Signed-off-by: David Leong <leongdl@amazon.com>
@leongdl
leongdl requested a review from a team as a code owner August 13, 2026 21:06
@github-actions github-actions Bot added the waiting-on-maintainers Waiting on the maintainers to review. label Aug 13, 2026
Comment thread job_bundles/ssh_to_smf/README.md Outdated

## Why Shell Access to a Worker?

Service-managed fleet workers have no public IP, no inbound ports, and no SSH keys to distribute, so a failing job normally has to be diagnosed from its log alone. That works until the question is environmental: which CUDA version is actually on the host, whether a license server is reachable from the worker's subnet, what the job attachments mount really looks like, or why a plugin loads on your workstation but not here. This bundle gives you a shell on the real worker, as the same `job-user` the job runs as, for the lifetime of a job you submit. Access goes through Session Manager, so it is IAM-gated and logged in CloudTrail, and the node is deregistered when the job ends.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This says the shell runs "as the same job-user the job runs as", but that does not match how this bundle is connected to. The SSH-over-SSM config further down in this same file uses User ssm-user (README.md:140), and Session Manager on a hybrid mi-* node lands you in the ssm-user account, not job-user. The Windows sibling states this explicitly: "Session Manager drops you into a PowerShell prompt as the ssm-user account."

This matters because the bullet directly below it promises you can "Inspect environment variables, license variables, mounted job attachments, and path mapping exactly as the job sees them" — which is precisely what you do not get from a different user account (different HOME, different env, and job attachment mounts may not be readable). Suggest saying you get a shell as ssm-user with passwordless sudo (which the host requirement at README.md:64 provides), and that you reach the job's own view via sudo -u job-user -i.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in 3440678. The SSH config at README.md:140 does say User ssm-user, and the Windows sibling states the same thing at line 188, so the claim that the shell runs as job-user was wrong, along with the bullet that depended on it.

The section now says Session Manager logs you in as ssm-user rather than job-user, notes that a fresh session therefore does not show the job's environment, and gives sudo -u job-user -i to get there. The first bullet is scoped to reading the environment and job attachment mounts "as job-user sees them" so it now follows the switch rather than contradicting it.

One correction on the suggestion itself: the host requirement at README.md:64 grants passwordless sudo to job-user, not ssm-user. setup/host_config.sh writes a single sudoers entry, job-user ALL=(ALL) NOPASSWD:ALL, which exists because the job's own script calls sudo cat /var/lib/amazon/ssm/registration and sudo systemctl. ssm-user sudo comes from the SSM agent's own sudoers drop-in, not from that host config, so I did not attribute it to the host requirement. I also have not run sudo -u job-user -i on a live worker to confirm it works there.

Comment thread job_bundles/ssh_to_smf/README.md Outdated
- **Verify network reachability.** Test license servers, package repositories, and VPC endpoints from inside the worker's subnet, which is often where a job's real failure lives.
- **Validate a container setup.** Run images by hand and check GPU passthrough before wiring them into a job template, alongside [`docker_nvidia_container_toolkit`](../../host_configuration_scripts/docker_nvidia_container_toolkit/README.md).

This is a debugging tool. Interactive sessions are billed as worker time for as long as the job runs, so keep `SessionMinutes` tight and let the job finish when you are done.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"let the job finish when you are done" reads as if ending your session ends the job, but the template does the opposite: it unconditionally sleeps the full SessionMinutes (job/template.yaml:107-118) regardless of whether anyone is connected. Disconnecting from Session Manager keeps billing worker time until the timer expires.

Since the paragraph is specifically warning about cost, it should say the way to stop paying early is to cancel the job — otherwise a reader who trusts this sentence will disconnect, assume they are done, and keep the worker up for the remainder of the window.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in 3440678. job/template.yaml:107-118 is a while [ $ELAPSED -lt $TOTAL_SECONDS ] loop that sleeps in 60s intervals with no check for a connected session, so the job runs the full SessionMinutes regardless. "Let the job finish when you are done" was the opposite of the advice a cost warning should give.

The sentence now says the job holds the worker for the full SessionMinutes whether or not anyone is connected, that disconnecting does not release it, and to cancel the job when you finish.

@crowecawcaw

Copy link
Copy Markdown
Contributor

Nice! LGTM. Just the prose linter to address.

…tion

Signed-off-by: David Leong <leongdl@amazon.com>
- **Verify network reachability.** Test license servers and package repositories from inside the worker's subnet, where a job's real failure often lives.
- **Validate a container setup.** Run images by hand and check GPU passthrough before wiring them into a job template, alongside [`docker_nvidia_container_toolkit`](../../host_configuration_scripts/docker_nvidia_container_toolkit/README.md).

Treat the bundle as a debugging tool. The job holds the worker for the full `SessionMinutes` whether or not anyone is connected, and disconnecting your session does not release it, so worker time bills until the timer expires. Keep `SessionMinutes` short and cancel the job when you finish.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The two new claims here conflict, and the combination leaves the node registered.

Line 9 says "the node is deregistered when the job ends", but deregistration in job/template.yaml:120-126 only runs after the keep-alive while loop completes normally. Cancelling the job terminates the session action, so the amazon-ssm-agent -register -clear and systemctl stop steps never execute. This line then advises cancelling as the normal way to finish.

Net effect of following the advice: the managed node stays registered in SSM and amazon-ssm-agent keeps running on the worker with the hybrid-activation identity still on disk under /var/lib/amazon/ssm/registration — i.e. shell access can outlive the job, which is exactly what the "for the duration of the job" framing promises against.

Suggest either qualifying line 9 (deregistration happens only on normal completion, not on cancel) and telling the reader to also aws ssm deregister-managed-instance --instance-id mi-... after cancelling, or not recommending cancel here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like deregistration happens only on normal completion, not on cancel

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Taken, in 6ad7379. Qualified rather than dropping the cancel recommendation, since cancelling is still the only way to stop paying early.

Line 9 now says the job deregisters the node when the keep-alive timer runs out, instead of "when the job ends". The closing paragraph states that cancellation kills the session action before the cleanup block at job/template.yaml:120-126 runs, so the node stays registered and the agent keeps running with the activation identity on disk, and that deregistration only happens on normal completion. It ends with the manual step:

aws ssm deregister-managed-instance --instance-id mi-XXXXXXXXX --region us-west-2

Step 5 of "How It Works" already scoped this correctly ("After the configured session duration"), so it needed no change.

Comment thread job_bundles/ssh_to_smf/README.md
##### Logging in as `job-user` (optional)

`job-user` is the Deadline worker service account that actually runs the job. By default it does not support login with an exposed password, so you cannot RDP in as `job-user` directly. If you want to inspect the job environment as `job-user` (same token, same env vars, same filesystem view), first RDP in as the `RDP` admin, open an elevated PowerShell, and run:
`job-user` is the Deadline worker service account that actually runs the job. By default it does not support login with an exposed password, so you cannot RDP in as `job-user` directly. Giving it a password lets you in, with the same token and the same view of the filesystem the job gets. First RDP in as the `RDP` admin. Then open an elevated PowerShell and run:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two problems with the rewritten sentence.

  1. "with the same token" is not accurate, and the rewrite states it more confidently than the text it replaced. RDP-ing in as job-user performs a fresh interactive logon, which mints a new access token in a new logon session. You get the same SID and the same filesystem ACL view, but not the job's token: no impersonation context, none of the session's environment, and a different logon session for anything scoped to it (e.g. mapped drives / job-attachment mounts established by the worker's logon, and HKCU state). Suggest dropping "the same token and" and keeping only the filesystem-view claim, which is the part that holds.

  2. The rewrite reframes job-user having no password from a security property into an obstacle to route around ("Giving it a password lets you in"), and it omits that the change persists past the job. job-user is a local Administrator on this host config (per the security note at line 7), and Set-LocalUser is not undone when the session ends — so the worker keeps a password-enabled admin account for any subsequent job scheduled onto it, until the instance is terminated. Worth stating explicitly here, alongside a pointer to clear it (Set-LocalUser -Name job-user -Password $null) or a reminder that the fleet must be scaled to zero afterward.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets just keep the original sentence?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted in 8b7c70d. The paragraph is back to the original upstream wording, so this PR no longer touches it (git diff upstream/mainline on this file is now additions only). Both of your points stand against the pre-existing text, but they belong in their own change rather than in a PR about adding a motivation section.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up in 651ff50: the sentence keeps its original content, with the opener changed from "If you want to inspect" to "To inspect" so the Vale check goes green. The three-item parenthetical you flagged is untouched.

Worth being straight about why that works: ai-tells.VerbTricolon is case-sensitive and its trigger token is a lowercase to , so a sentence-initial "To" does not fire it. The tricolon is still in the text. Rewriting it properly, along with the "same token" accuracy point, is a separate change against the pre-existing paragraph.

Comment thread job_bundles/ssh_to_smf_windows/README.md Outdated
Signed-off-by: David Leong <leongdl@amazon.com>
Comment thread job_bundles/ssh_to_smf/README.md Outdated
Signed-off-by: David Leong <leongdl@amazon.com>
Comment thread job_bundles/ssh_to_smf/README.md
- **Verify network reachability.** Test license servers and package repositories from inside the worker's subnet, where a job's real failure often lives.
- **Validate a container setup.** Run images by hand and check GPU passthrough before wiring them into a job template, alongside [`docker_nvidia_container_toolkit`](../../host_configuration_scripts/docker_nvidia_container_toolkit/README.md).

Treat the bundle as a debugging tool. The job holds the worker for the full `SessionMinutes` whether or not anyone is connected, and disconnecting your session does not release it, so worker time bills until the timer expires. Keep `SessionMinutes` short and cancel the job when you finish.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This paragraph is the only caveat in the Linux README, and it covers cost but not the privilege grant that the bundle requires. Compare the Windows sibling, which opens with a bolded Security note about local-Administrator rights and "for debugging purposes, not for production".

The Linux prerequisite is comparable in scope: setup/host_config.sh writes

job-user ALL=(ALL) NOPASSWD:ALL

to /etc/sudoers.d/job-user. That is unrestricted passwordless root for job-user, it is fleet-wide and persistent, and it applies to every job that runs on any worker with this host configuration, not just this bundle. Combined with the residual-access issue on line 9, a leftover Session Manager shell reaches an account that can become root.

Since this new section is the first thing in the file that pitches reasons to adopt the bundle, and one of its bullets ("Prototype a host configuration script") specifically encourages using it as a workflow rather than a one-off, it is worth adding a security note here with the same prominence as the Windows one: debugging only, not production, and shut the fleet down afterwards. Otherwise a reader who stops after this section never sees the tradeoff.

Signed-off-by: David Leong <leongdl@amazon.com>

Treat the bundle as a debugging tool. The job holds the worker for the full `SessionMinutes` whether or not anyone is connected, and disconnecting your session does not release it, so worker time bills until the timer expires. Keep `SessionMinutes` short and cancel the job when you finish.

Cancelling stops the billing, but the `mi-` entry remains in your account's SSM inventory either way. Deregister it afterwards:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Cancelling stops the billing" is only true for the Deadline Cloud worker billing. It reads as though the leftover mi- entry is purely an inventory-hygiene issue, but this bundle requires the advanced-instances tier (see "2. Enable Advanced-Instances Tier" above), and the advanced on-premises instance charge accrues per hour for each instance registered in that tier — not per hour that a session is open, and not per hour that the Deadline job runs.

So after the job ends (timer expiry or cancellation), the mi- node is still registered and the ~$0.00695/hr charge keeps accruing until deregister-managed-instance is called. For a user who submits a few of these and never runs the command in the block below, that is an open-ended charge with no job or session to make it visible — the SSM console is the only place it shows up.

Two suggestions:

  1. Reword so the deregistration step is presented as stopping a charge, not just tidying inventory, e.g. "Cancelling stops the worker billing, but the mi- entry stays registered in the advanced-instances tier and keeps accruing the per-instance hourly charge until you deregister it."
  2. The related line under "2. Enable Advanced-Instances Tier" — "Negligible for short-lived sessions" — is load-bearing on the same assumption. It is only negligible if the node is actually deregistered afterwards; left registered, the cost is per calendar hour indefinitely. Worth cross-referencing the deregister command from there too.

Comment thread job_bundles/ssh_to_smf/README.md
Comment thread job_bundles/ssh_to_smf/README.md
Comment thread job_bundles/ssh_to_smf/README.md
Comment thread job_bundles/ssh_to_smf_windows/README.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

waiting-on-maintainers Waiting on the maintainers to review.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants