feat(worker): edit admin users on a live worker - #181
Draft
nilsmechtel wants to merge 1 commit into
Draft
Conversation
Permissions are checked per call against `self.admin_users`, so the list was already editable at runtime — only the API was missing. Adds `list_admin_users`, `add_admin_user` and `remove_admin_user` to the worker service, all admin-gated. The list is mutated in place because `AppsManager` and `CodeExecutor` hold the same object, and it is persisted to `<workspace_dir>/admin_users.json` before the in-memory change so a grant cannot exist only in memory. On restart the persisted list overrides the replayed `--admin-users` seed and the divergence is logged, rather than silently rolling back. Removal refuses three cases: the caller (self-lockout), the last admin (reachable on a worker started with `--admin-users '*'`), and the worker's own Hypha identity (which `_connect_to_server` re-inserts, so the removal would revert silently). `add_admin_user` refuses `'*'` — only the operator starting the worker gets to make it world-writable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the worker half of the "admin users and per-dataset access are fixed at startup" report. Adds
list_admin_users,add_admin_userandremove_admin_userto the worker service, all admin-gated, editable on a live worker and durable across a restart.The premise turned out to be half wrong, in a useful way
The report assumed changing the admin list needs a pod restart because it is baked in at startup. It is not baked in anywhere.
check_permissions(context, self.admin_users, ...)runs inside each method body on every call, soself.admin_usersis consulted live — the Hypha service registration never carries it.AppsManagerandCodeExecutorare handed the same list object atcomplete_initialization/initialize, so a change reaches them too, provided it is made in place. What was missing was an API, not a mechanism, and this PR is correspondingly small: no re-registration, no service restart, no reconnect.One thing genuinely is baked in at deploy time and stays that way:
_inject_admin_usersfolds the admin list into each app's per-methodauthorized_userswhen the app is deployed. An admin added afterwards reaches the worker API immediately but not the apps already running. That asymmetry is now stated indocs/glossary.mdrather than left to be discovered.Persistence
The triage defaulted to the
app_data-on-proxy round trip that PR #123's scaling state uses, and flagged the one thing that would invalidate that default: it stores per-app data while the admin list is worker-scope. That is what happened — there is no app to hang a worker-scope list on, and inventing one would mean the list disappears whenever no app is running. So this falls back to the file, as the triage pre-authorised:<workspace_dir>/admin_users.json, written via a temp file andos.replace.The ordering constraint from the triage is respected literally. The startup
--admin-usersflag is a seed, the file is an overlay, the overlay wins on restart, and the divergence is logged with both directions named (added at runtime: [...], removed at runtime: [...]) instead of applied silently. A corrupt or wrong-shaped file falls back to the seed and says so at ERROR rather than starting with no admins.Persistence happens before the in-memory list changes. A grant that exists only in memory would revert on the next restart without ever having failed — the same silent-rollback shape the triage said not to reproduce in a permission surface.
Lockout protection
remove_admin_userrefuses three cases:--admin-users '*': with a single named admin, that admin is necessarily the caller and the self-removal rule fires first. With the wildcard every caller is an admin, so nothing else would stop one of them from revoking the only entry and locking out everyone including the operator._connect_to_serverunconditionally re-insertsserver.config.user["email"]at index 0 on every connect, so removing it would appear to succeed and then revert at the next reconnect. Refusing it is the difference between a visible error and a silent no-op.add_admin_userrefuses'*'. Starting a worker with--admin-users '*'is a deliberate operator choice; letting any one admin make the worker world-writable over RPC is a different decision, and not one this API should offer.Tests
19 new tests in
tests/worker/test_admin_users.py. Eight positive controls, each breaking one behaviour and each failing exactly the tests that name it:test_a_caller_cannot_revoke_their_own_admin_permissions,..._by_user_idtest_the_last_admin_cannot_be_removedtest_the_workers_own_identity_cannot_be_removedtest_the_wildcard_cannot_be_granted_over_rpctest_the_change_reaches_the_component_managerstest_an_unwritable_store_leaves_the_admin_list_unchangedtest_the_persisted_list_overrides_the_startup_seed,test_a_runtime_revocation_is_not_undone_by_the_seedtest_the_admin_methods_are_exposed_on_the_worker_servicepython -m pytest tests --noconftest -q: 224 passed on the branch against 205 onmainunder the same invocation, with the same 55 pre-existing conftest-dependent errors on both.Not in this PR: per-dataset authorized users
The report bundles the dataset half on the grounds that both need one persistence mechanism. Reading the data server invalidates that rationale, in two ways worth recording.
First, per-dataset access is not fixed at startup and never needed a restart.
_watch_data_dirinbioengine/datasets/proxy_server.pypolls the data directory every 30 s and hot-reloads the registry in place, so editing a dataset'smanifest.yamltakes effect within 30 s on a running server. The gap is an API, not a restart requirement — and the persistence question is already answered for datasets, becausemanifest.yamlon the data directory is the durable store.Second, the data server is a separate process in a separate image (
bioengine-datasets), started independently viapython -m bioengine.datasets, potentially on a different host, and the worker reaches it only over HTTP. It has no admin concept at all — no--admin-users, no write path for manifests, and per-request authorization derived solely from each dataset's ownauthorized_users. Addingset_dataset_authorized_userstherefore means adding an authenticated write endpoint there and deciding who is allowed to call it, which is a security model rather than a plumbing detail: the data server does not know the worker's admin list, and giving it one is a new trust relationship between two processes that currently share nothing but a URL.That decision is left open rather than defaulted. The worker half stands alone and is complete.