Skip to content

Commit 14807a2

Browse files
authored
fix(fspy): use memfd for Linux shared memory (#523)
## Motivation Stop Linux file-access tracking from consuming the container's `/dev/shm` quota and crashing with `SIGBUS`. ## Changes - Replaces POSIX shared memory with a sealed `memfd`. - Uses an abstract Unix socket broker to send the descriptor to each opener. - Runs the broker asynchronously and keeps the preload client synchronous. - Stops new opens when the owner is dropped while keeping existing mappings valid. - Sets close-on-exec and derives the mapping length from the sealed descriptor. - Covers concurrent and invalid opens. - Updates the constrained-`/dev/shm` fixture to succeed. Fixes #353
1 parent 9e72876 commit 14807a2

14 files changed

Lines changed: 458 additions & 31 deletions

File tree

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
# Changelog
22

3+
- **Fixed** Automatic file-access tracking on Linux now works in containers and Kubernetes runners with limited `/dev/shm` space ([#353](https://github.com/voidzero-dev/vite-task/issues/353), [#523](https://github.com/voidzero-dev/vite-task/pull/523)).
34
- **Fixed** Windows automatic input tracking now records executable image reads when process creation performs the lookup inside `NtCreateUserProcess` ([#518](https://github.com/voidzero-dev/vite-task/pull/518)).
45
- **Fixed** Failures while waiting for a started task process to exit no longer incorrectly say the process failed to spawn ([#515](https://github.com/voidzero-dev/vite-task/pull/515)).
56
- **Fixed** Missing env vars requested through `@voidzero-dev/vite-task-client` now return `undefined` instead of `null`, preserving Vite production `NODE_ENV` semantics when builds run through `vp run` ([#508](https://github.com/voidzero-dev/vite-task/pull/508)).

Cargo.lock

Lines changed: 8 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Cargo.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -116,6 +116,7 @@ ref-cast = "1.0.24"
116116
regex = "1.11.3"
117117
rusqlite = "0.39.0"
118118
rustc-hash = "2.1.1"
119+
rustix = "1.1"
119120
# SeccompAction::UserNotif (SECCOMP_RET_USER_NOTIF) was added after the latest published release (v0.5.0)
120121
seccompiler = { git = "https://github.com/rust-vmm/seccompiler", rev = "08587106340b8e3cb361c7561411510039436857" }
121122
serde = "1.0.219"

crates/fspy_shared/Cargo.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ assert2 = { workspace = true }
2727
ctor = { workspace = true }
2828
rustc-hash = { workspace = true }
2929
subprocess_test = { workspace = true }
30+
tokio = { workspace = true, features = ["macros", "net", "rt-multi-thread", "time"] }
3031

3132
[lints]
3233
workspace = true

crates/fspy_shared/src/ipc/channel/mod.rs

Lines changed: 20 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -113,9 +113,12 @@ impl Deref for Sender {
113113
}
114114
}
115115

116-
#[expect(
117-
clippy::non_send_fields_in_send_ty,
118-
reason = "`Sender` holds a shared file lock that ensures there's no reader, so `shm` can be safely written to"
116+
#[cfg_attr(
117+
not(target_os = "linux"),
118+
expect(
119+
clippy::non_send_fields_in_send_ty,
120+
reason = "`Sender` holds a shared file lock that ensures there's no reader, so `shm` can be safely written to"
121+
)
119122
)]
120123
/// SAFETY: `Sender` holds a shared file lock that ensures there's no reader, so `shm` can be safely written to.
121124
unsafe impl Send for Sender {}
@@ -131,9 +134,12 @@ pub struct Receiver {
131134
shm: Shm,
132135
}
133136

134-
#[expect(
135-
clippy::non_send_fields_in_send_ty,
136-
reason = "Receiver doesn't read or write `shm`. It only pass it to `ReceiverLockGuard` under the lock"
137+
#[cfg_attr(
138+
not(target_os = "linux"),
139+
expect(
140+
clippy::non_send_fields_in_send_ty,
141+
reason = "Receiver doesn't read or write `shm`. It only passes it to `ReceiverLockGuard` under the lock"
142+
)
137143
)]
138144
/// SAFETY: `Receiver` doesn't read or write `shm`. It only passes it to `ReceiverLockGuard` under the lock.
139145
unsafe impl Send for Receiver {}
@@ -200,8 +206,8 @@ mod tests {
200206

201207
use super::*;
202208

203-
#[test]
204-
fn smoke() {
209+
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
210+
async fn smoke() {
205211
let (conf, receiver) = channel(100).unwrap();
206212
let cmd = command_for_fn!(conf, |conf: ChannelConf| {
207213
let sender = conf.sender().unwrap();
@@ -220,9 +226,9 @@ mod tests {
220226
assert!(frames.next().is_none());
221227
}
222228

223-
#[test]
229+
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
224230
#[expect(clippy::print_stdout, reason = "test diagnostics")]
225-
fn forbid_new_senders_after_locked() {
231+
async fn forbid_new_senders_after_locked() {
226232
let (conf, receiver) = channel(42).unwrap();
227233
let _lock = receiver.lock().unwrap();
228234

@@ -233,9 +239,9 @@ mod tests {
233239
assert_eq!(B(&output.stdout), B("false"));
234240
}
235241

236-
#[test]
242+
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
237243
#[expect(clippy::print_stdout, reason = "test diagnostics")]
238-
fn forbid_new_senders_after_receiver_dropped() {
244+
async fn forbid_new_senders_after_receiver_dropped() {
239245
let (conf, receiver) = channel(42).unwrap();
240246
drop(receiver);
241247

@@ -246,8 +252,8 @@ mod tests {
246252
assert_eq!(B(&output.stdout), B("false"));
247253
}
248254

249-
#[test]
250-
fn concurrent_senders() {
255+
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
256+
async fn concurrent_senders() {
251257
let (conf, receiver) = channel(8192).unwrap();
252258
for i in 0u16..200 {
253259
let cmd = command_for_fn!((conf.clone(), i), |(conf, i): (ChannelConf, u16)| {

crates/fspy_shared/src/ipc/channel/shm_io.rs

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -672,6 +672,18 @@ mod tests {
672672

673673
const SHM_SIZE: usize = 1024 * 1024;
674674

675+
// On Linux, `fspy_shm::create` spawns the mapping's broker onto the
676+
// ambient tokio runtime, which serves the child processes' opens.
677+
#[cfg(target_os = "linux")]
678+
let runtime = tokio::runtime::Builder::new_multi_thread()
679+
.worker_threads(1)
680+
.enable_io()
681+
.enable_time()
682+
.build()
683+
.unwrap();
684+
#[cfg(target_os = "linux")]
685+
let _guard = runtime.enter();
686+
675687
let shm = fspy_shm::create(SHM_SIZE).unwrap();
676688
let shm_name = shm.id().to_owned();
677689

crates/fspy_shm/Cargo.toml

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,12 +7,22 @@ license.workspace = true
77
publish = false
88
rust-version.workspace = true
99

10-
[dependencies]
10+
[target.'cfg(not(target_os = "linux"))'.dependencies]
1111
shared_memory = { workspace = true, features = ["logging"] }
1212

13+
[target.'cfg(target_os = "linux")'.dependencies]
14+
memmap2 = { workspace = true }
15+
passfd = { workspace = true, features = ["async"] }
16+
rustix = { workspace = true, features = ["fs", "net", "process"] }
17+
tokio = { workspace = true, features = ["macros", "net", "rt", "time"] }
18+
tokio-util = { workspace = true }
19+
tracing = { workspace = true }
20+
uuid = { workspace = true, features = ["v4"] }
21+
1322
[dev-dependencies]
1423
ctor = { workspace = true }
1524
subprocess_test = { workspace = true }
25+
tokio = { workspace = true, features = ["macros", "rt-multi-thread"] }
1626

1727
[lints]
1828
workspace = true

crates/fspy_shm/README.md

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,10 @@ The public API is defined in [`src/lib.rs`](src/lib.rs).
2929

3030
The channel hides that difference with its lock file. [`ChannelConf::sender`](../fspy_shared/src/ipc/channel/mod.rs) opens and locks the receiver's exact lock-file path before it calls `fspy_shm::open`. The receiver removes that path before dropping the owner, so a sender that starts later fails before opening shared memory.
3131

32-
## Backend boundary
32+
## Platform designs
3333

34-
At this point in the stack, `fspy_shm` delegates mapping creation and opening to the [`shared_memory`](https://crates.io/crates/shared_memory) crate. Because callers use only the API above, later changes can replace the backend on each platform without changing the channel protocol.
34+
Each platform keeps its implementation rationale beside its source:
35+
36+
- [Linux: `memfd` with a descriptor broker](src/linux/README.md)
37+
38+
At this point in the stack, Windows and macOS still delegate mapping creation and opening to the [`shared_memory`](https://crates.io/crates/shared_memory) crate.

crates/fspy_shm/src/lib.rs

Lines changed: 25 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,18 @@
11
#![doc = include_str!("../README.md")]
22

3+
#[cfg(not(target_os = "linux"))]
34
use std::io;
45

6+
#[cfg(target_os = "linux")]
7+
mod linux;
8+
9+
#[cfg(target_os = "linux")]
10+
pub use linux::{Shm, create, open};
11+
#[cfg(not(target_os = "linux"))]
512
use shared_memory::{Shmem, ShmemConf};
613

714
/// An owned shared-memory mapping.
15+
#[cfg(not(target_os = "linux"))]
816
pub struct Shm {
917
inner: Shmem,
1018
}
@@ -18,6 +26,7 @@ pub struct Shm {
1826
/// # Errors
1927
///
2028
/// Returns an error if the platform cannot create or map the region.
29+
#[cfg(not(target_os = "linux"))]
2130
pub fn create(size: usize) -> io::Result<Shm> {
2231
let conf = ShmemConf::new().size(size);
2332
#[cfg(target_os = "windows")]
@@ -36,6 +45,7 @@ pub fn create(size: usize) -> io::Result<Shm> {
3645
/// # Errors
3746
///
3847
/// Returns an error if the mapping does not exist or cannot be mapped.
48+
#[cfg(not(target_os = "linux"))]
3949
pub fn open(id: &str) -> io::Result<Shm> {
4050
let conf = ShmemConf::new().os_id(id);
4151
#[cfg(target_os = "windows")]
@@ -45,6 +55,7 @@ pub fn open(id: &str) -> io::Result<Shm> {
4555
Ok(Shm { inner })
4656
}
4757

58+
#[cfg(not(target_os = "linux"))]
4859
#[expect(clippy::len_without_is_empty, reason = "shared-memory mappings are always non-empty")]
4960
impl Shm {
5061
/// Returns this mapping's opaque platform identifier.
@@ -86,11 +97,11 @@ mod tests {
8697

8798
use super::{Shm, create, open};
8899

89-
// Page-aligned on supported macOS and Windows targets.
100+
// Page-aligned on all supported targets.
90101
const SIZE: usize = 64 * 1024;
91102

92-
#[test]
93-
fn create_and_open_are_shared() {
103+
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
104+
async fn create_and_open_are_shared() {
94105
let owner = create(SIZE).unwrap();
95106
assert_eq!(owner.len(), SIZE);
96107
assert_eq!(owner.as_ptr() as usize % align_of::<usize>(), 0);
@@ -107,8 +118,8 @@ mod tests {
107118
assert_eq!(read_byte(&owner, SIZE - 1), 29);
108119
}
109120

110-
#[test]
111-
fn mapping_is_visible_across_processes() {
121+
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
122+
async fn mapping_is_visible_across_processes() {
112123
let owner = create(SIZE).unwrap();
113124
write_byte(&owner, 0, 17);
114125

@@ -117,21 +128,25 @@ mod tests {
117128
assert_eq!(read_byte(&opened, 0), 17);
118129
write_byte(&opened, SIZE - 1, 29);
119130
});
120-
assert!(Command::from(command).status().unwrap().success());
131+
let success =
132+
tokio::task::spawn_blocking(move || Command::from(command).status().unwrap().success())
133+
.await
134+
.unwrap();
135+
assert!(success);
121136
assert_eq!(read_byte(&owner, SIZE - 1), 29);
122137
}
123138

124-
#[test]
125-
fn owner_drop_prevents_new_opens() {
139+
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
140+
async fn owner_drop_prevents_new_opens() {
126141
let owner = create(SIZE).unwrap();
127142
let id = owner.id().to_owned();
128143
drop(owner);
129144

130145
assert!(open(&id).is_err());
131146
}
132147

133-
#[test]
134-
fn opened_mapping_survives_owner_drop() {
148+
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
149+
async fn opened_mapping_survives_owner_drop() {
135150
let owner = create(SIZE).unwrap();
136151
let id = owner.id().to_owned();
137152
let opened = open(&id).unwrap();
Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# Linux backend
2+
3+
The Linux backend stores data in a sealed `memfd`. A process opens the mapping by connecting to an abstract Unix-domain socket and receiving the descriptor from a broker. It then accesses the mapped memory directly.
4+
5+
The backend must avoid `/dev/shm` quotas, expose a large mapping without allocating every page up front, support synchronous opens from preload code, and stop new opens when the owner is dropped without invalidating existing views.
6+
7+
## Options considered
8+
9+
| Option | Decision |
10+
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------- |
11+
| POSIX shared memory | Rejected because Linux stores it in `/dev/shm`, so it shares that mount's size limit. |
12+
| System V shared memory | Rejected because IPC namespace limits affect availability and the owner must explicitly remove the segment. |
13+
| Sparse temporary file | Rejected because dirty pages may reach disk, and sharing the path and deleting the file require additional handling. |
14+
| `memfd` with descriptor broker | Selected. It avoids `/dev/shm` and System V limits. The kernel keeps it alive while a descriptor or mapping refers to it. |
15+
16+
The broker accepts and serves clients with Tokio. Opening is synchronous because it can run before `main`, so creating an owner must occur inside a Tokio runtime.
17+
18+
## Why not `shared_memory`
19+
20+
`shared_memory` uses POSIX `shm_open` on Linux and cannot construct a mapping from a `memfd`, so it retains the `/dev/shm` dependency.
21+
22+
## Lifetime semantics
23+
24+
Dropping the owner stops the broker. Existing views remain valid; later opens fail.

0 commit comments

Comments
 (0)