Reach an offboard server over a Unix socket with a unix:// URL - #735
Reach an offboard server over a Unix socket with a unix:// URL#735v-positronic wants to merge 15 commits into
unix:// URL#735Conversation
A policy process on the same machine can now serve the offboard protocol on a Unix domain socket, and the rig reaches it as an ordinary remote endpoint. The carrier changes; the protocol does not. `InferenceClient` accepts `unix://<absolute socket path>[/api/v1/session[/<model_id>]][?query]`. The socket path runs to the first `/api/v1` segment, so a bare socket path is the default session, and a model id and session params follow the path as they follow a host. The client dials the session with `unix_connect` and reads `/api/v1/models` through an `httpx` transport bound to the same socket. TLS does not apply. `PolicyServer(uds=...)` and `serve --uds` bind that socket path in place of `host:port`, and every vendor server CLI takes the flag. The server removes the socket file an earlier run left, so a restart binds again; anything else at the path stays and the bind fails. Positronic-Robotics/internal#1242 Ticket: Positronic-Robotics/internal#1242 #refs
… tests The two keys the constants already name read through them, beside the `HOST` and `PORT` assertions in the same test, so a rename reaches the test. Ticket: Positronic-Robotics/internal#1242 #refs
The `unix://` grammar has one home per reader: the client's class docstring for a caller, the offboard README for the protocol. The `serve` docstring, the private helper and the inference guide each said it again. Ticket: Positronic-Robotics/internal#1242 #refs
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8c57f4a5eb
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
`clear_stale_socket` judged a socket stale by its file type alone, so a second server on one path unlinked the first server's address and took its future connections. It now connects to the socket first: a refused connection means nobody listens and the file is stale; an accepted one means the socket has an owner, and the bind fails instead. Ticket: Positronic-Robotics/internal#1242 #refs
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 30f6c4bb0d
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
`clear_stale_socket` left a live socket in place and relied on the bind to fail. It does not: `asyncio.create_unix_server`, which uvicorn calls, unlinks an existing socket file before it binds, so a second server on one path took the first server's future connections and neither side said anything. The check now raises `EADDRINUSE` naming the path, before uvicorn is built. It is `claim_socket_path` for that reason: it takes the path or refuses it, and a name that says only "clear" no longer covers what it does. A refused connection still means nobody listens, so the file is stale and goes. Ticket: Positronic-Robotics/internal#1242 #refs
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2aedb6a2b6
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Two defects in the same guard, both from defending a path instead of taking it. The probe told a live server from a stale file but claimed nothing, so two servers starting against an absent or stale path both passed it and the second bind took the first. The bind is the claim now: it happens first, and a path already held fails it. A probe follows only to read what holds the path, and it answers one question -- is this a socket nobody serves, so replacing it costs nobody anything. That probe also blocked with no bound, so a listener with a full backlog held startup open. It waits `LIVE_SOCKET_PROBE_SEC` and reads a wait that runs out as a live server. `serve` hands uvicorn the bound descriptor through `uvicorn.Config(fd=...)` rather than the path. Uvicorn then calls `create_server(sock=...)`, which binds nothing, where `uds=` reaches `create_unix_server(path=...)` and unlinks what it finds. The socket keeps the 0o666 uvicorn gives one it binds itself, and the file stays behind at shutdown: a successor reads it as stale, and an unlink here could take a path that successor has already claimed. Ticket: Positronic-Robotics/internal#1242 #refs
Both described the probe by leading with the reason for it. Ticket: Positronic-Robotics/internal#1242 #refs
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 58a3ebdb4b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
`urlsplit` leaves a path percent-encoded, and the socket path was dialled as it came, so a directory holding a space or a percent named a file that is not there. The socket path is decoded once, after the split. The split still runs over the encoded path, so an escaped separator stays inside a directory name rather than becoming one. The URL path keeps its escapes: the server decodes those, which is what lets a model id carry its own. Ticket: Positronic-Robotics/internal#1242 #refs
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7ec31e88b1
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
The claim chmod'd every socket to 0o666, which reads as the mode uvicorn gives one and is not: uvicorn's own bind takes the umask, and it copies an existing file's mode where there is one. So a deployment with a restrictive umask had its socket widened, and an open server behind it became reachable by any local account that can walk the directory. The bind's own mode stands. Two documents said something that is not so. The socket-path docstring claimed an escaped separator stays inside a directory name; the decode resolves every escape, so `%2F` becomes a separator and a socket path cannot hold a name carrying a slash. It says that now, and the test that asserted it is named for it. The README's serve inventory listed four flags and now lists `--uds` as well. Ticket: Positronic-Robotics/internal#1242 #refs
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f09910be49
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
`serve` binds only once the model has loaded, so the path is absent for that whole interval. The dial raised `FileNotFoundError` on it at once, which is the state a co-located server is in for the seconds after its unit starts, so the 900-second connect budget bought that deployment nothing. An absent path now rides the deadline, and so does a socket that refuses, which is a server restarting. A refusal from anything else at the path is a wrong path and still surfaces at once, as a permission error always did: the two share `ECONNREFUSED`, so the socket itself is what tells them apart. `_ConnectRetries` holds the whole retry policy now — the 403 budget it already had, plus the deadline and the backoff `new_session` was carrying in locals — so both handlers spend an attempt through one call. Ticket: Positronic-Robotics/internal#1242 #refs
…ng path macOS answers a dial at a path holding a regular file with ENOTSOCK, where Linux answers ECONNREFUSED. The check keyed on the exception, so the two platforms classified one state differently, and the test that pinned the Linux errno went red on macOS. The path answers it on both: absent or a socket means a server may still bind there, anything else at the path does not. A permission error stays settled wherever it comes from, since this client cannot read past it to find out. The test asserts what the code promises — a wrong path does not spend the connect deadline — rather than the errno that carries it. Ticket: Positronic-Robotics/internal#1242 #refs
Two were clefts and one told the reader how to take the fact before it. Ticket: Positronic-Robotics/internal#1242 #refs
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5312288a4c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| dial = ( | ||
| partial(connect, self._ws_uri) | ||
| if self.uds is None | ||
| else partial(unix_connect, self.uds, uri=self._ws_uri) |
There was a problem hiding this comment.
When the listener's accept backlog is full, websockets.sync.client.unix_connect performs a blocking Unix sock.connect(path) before the WebSocket connect code applies open_timeout; this attempt can therefore hang indefinitely without reaching the retry deadline. Create and connect the AF_UNIX socket with a timeout or nonblocking deadline before passing it to the WebSocket client.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Declining this one — the connect is already bounded, at the line the finding cites.
websockets.sync.client.connect builds the AF_UNIX socket and sets the deadline on it before connecting:
deadline = Deadline(open_timeout)
...
if unix:
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
sock.settimeout(deadline.timeout()) # <- before the connect
assert path is not None
sock.connect(path)Checked in both versions this repository can resolve: 16.0, which the lockfile pins (.venv/.../websockets/sync/client.py:291-294), and 15.0.1, the floor pyproject.toml declares (src/websockets/sync/client.py:288-291). Identical in both. A full accept backlog therefore raises TimeoutError after open_timeout, which new_session already catches and spends against connect_deadline, rather than hanging.
Creating and connecting the socket here would duplicate that, and would take on unix_connect's remaining setup to hand it a sock=. Leaving the thread open for the driver's word rather than resolving it myself.
…ls that clear The handshake metadata put a socket path under `host`, which labels a filesystem path as a network address wherever an episode's metadata is read. It gets `uds`, and a server reports one pair or the other. The client waited out the whole connect deadline on any `OSError` the path answered as a socket, including one this process raised itself — a descriptor limit, a memory limit. Only an absent path and a refusal can mean a co-located server that has not bound yet, so every other dial surfaces at once. Every document that lists the server's flags now names `--uds`: the offboard README, the five vendor READMEs, and the training workflow. The enumeration is complete. Ticket: Positronic-Robotics/internal#1242 #refs
`LIVE_SOCKET_PROBE_SEC` sat at the top of the file with its one user 400 lines below. It is `PolicyServer`'s now, above the method that spends it. `claim_socket_path`'s docstring described what its caller hands to uvicorn, which nothing keeps true. It states the constraint on the returned socket instead. Ticket: Positronic-Robotics/internal#1242 #refs
1 participant
A policy process on the same machine now serves the offboard protocol on a Unix domain socket, and the rig reaches it as an ordinary remote endpoint. The handshake, the declared rig-side stack, the session and
inferare unchanged. Only the carrier is new.This is the carrier for a policy that runs beside the harness in a container with no network interface of its own. The container binds a socket path; the host dials it. The endpoint stays
kind: remote, and it needs no headers file.The URL grammar
The socket path runs to the first
/api/v1segment, matched as a whole path segment.unix:///run/policy.sock/run/policy.sockunix:///run/policy.sock?codec.fps=10/run/policy.sockunix:///run/policy.sock/api/v1/session/10000/run/policy.sock10000A relative path is refused. TLS does not apply.
headersride as they do over TCP. Only the socket path is URL-decoded, because it names a file; the URL path reaches the server as written, so a model id keeps its own escapes.Client
InferenceClientdials the session withunix_connect(path, uri=...), keeping its timeouts and headers, and reads/api/v1/modelsthroughhttpx.HTTPTransport(uds=path). The handshake asks for the URL path under a stand-in host, because a socket path is not a host;session_urlstill names the socket, so a failed connect says which one.A co-located server binds its socket only once the model has loaded, so the client waits for it as it waits for a cold backend over TCP. Two dials can mean the server is still coming: the path is absent, and the path refuses. Every other
OSError— a permission the path denies, a limit this process has hit — is settled, and surfaces at once rather than spending the deadline on an answer that will not change. A refusal then reads the path, which tells a restarting server from a path naming something that is not a socket.Server
PolicyServer(uds=...)andserve --udsbind that socket path in place ofhost:port. Every vendor server CLI takes the flag, because each one is a config ofserve.The bind is the claim, so two servers starting together cannot both take one path: the loser's bind fails. A socket file an earlier run left is replaced, which lets a restart bind again; a live socket, and anything at the path that is not a socket, are refused and left as they are. The server hands uvicorn the descriptor rather than the path, so uvicorn never unlinks it, and shutdown leaves the file for a successor to read as stale.
The handshake reports the socket path under its own
udskey, and carries nohostand noport: a filesystem path is not a network address, and a Unix socket has no port. Every document that lists the server's flags names--uds.Tested
list_models, the handshake, its metadata, oneinfer, and a model id past the socket path.The test socket lives under a short
/tmpdirectory. A macOStmp_pathcan pass the 104-byte limit on a socket path, and CI runs the core tests on macOS.Positronic-Robotics/internal#1242