Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions ansible/roles/developer-rust/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,6 +189,8 @@ cargo on PATH, so a developer or an agent who knows none of this runs
Which instrument yields on which host, and why it needs the `zram_swap` role,
are in
[docs/rust-build-governor.md](../../../docs/rust-build-governor.md).
The role also writes `agents.slice`, the cgroup bounding Claude Code sessions,
from the same RAM budget (`--tags agents-slice`, `agents_*` variables).
The host-wide picture is
[docs/concurrent-dev-cache.md](../../../docs/concurrent-dev-cache.md).

Expand Down
72 changes: 64 additions & 8 deletions ansible/roles/developer-rust/defaults/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -155,8 +155,10 @@ rust_sccache_service_manage: true
# shim, the slice and the config.
#
# MEMORY is BOUNDED, by rustbuild.slice: MemoryHigh throttles by reclaim and
# MemoryMax kills the build rather than the host. Both are percentages of the
# host's own RAM, so one number fits a laptop and a build box.
# MemoryMax kills the build rather than the host. Both are sized at converge
# from the host's own RAM (ansible_facts memtotal_mb), so one number fits a
# laptop and a build box and the desktop keeps the rest. A host that gains RAM
# needs a re-converge (--tags rust-governor) to pick it up.
#
# CPU is not bounded at all -- it is DEFERRED. Builds run at nice 19 under a
# below-desktop CPUWeight, so they take every core while nobody else wants one
Expand Down Expand Up @@ -184,14 +186,68 @@ rust_governor_enabled: true
# REFUSE the build outright rather than fall back to compiling.
rust_governor_no_incremental: false

# Percentages of the HOST's physical RAM (systemd resolves them all against
# RAM, the swap cap included), so the same numbers ship everywhere. High
# throttles and reclaims; Max kills the build -- the build dies, never the
# host.
rust_governor_memory_high_pct: 50
rust_governor_memory_max_pct: 70
# Share of the host's RAM the whole build slice may use. High throttles and
# reclaims, Max kills the build -- the build dies, never the host. Max is a
# third of RAM so the desktop and its sessions keep the other two thirds.
# Raise both on a host dedicated to builds (a remote build target), e.g. 60 and
# 50, in its host or group vars.
rust_governor_memory_high_pct: 27
rust_governor_memory_max_pct: 33

# Smallest slice limits in MiB, so a small host can still build. Applied after
# the percentage, and High never exceeds Max.
rust_governor_memory_high_floor_mb: 3072
rust_governor_memory_max_floor_mb: 4096

# Percentage of the host's RAM, resolved by systemd, so it needs no sizing here.
rust_governor_swap_max_pct: 25

# ---------------------------------------------------------------------------
# Agent sessions: agents.slice
# ---------------------------------------------------------------------------
# Every process a Claude Code session starts (shell commands, subagents, MCP
# servers, browsers) inherits the session's cgroup, so one cap on agents.slice
# bounds all of them. Builds get rustbuild.slice and the desktop keeps the
# rest, which means no agent can cause a host-wide OOM: an overrun kills the
# biggest process inside the slice and nothing outside it. The role writes only
# the slice. hyperi-ai moves each session into an agents-<session>.scope under
# it when a session starts, and does nothing where the slice is missing.
agents_slice_enabled: true

# The desktop's reserve: the larger of a share of RAM and a floor in MiB. The
# floor covers a small host, where a share alone leaves a browser and an editor
# too little.
agents_reserve_pct: 30
agents_reserve_floor_mb: 9216

# MemoryMax is RAM minus the reserve minus rustbuild.slice's MemoryMax (the
# governor off means nothing to subtract). The floor in MiB stops a small host
# computing a slice too small to run one session. MemoryHigh is a share of Max.
agents_memory_max_floor_mb: 2048
agents_memory_high_pct: 90

# Below the desktop's 100 so sessions yield to it under contention, and a
# process ceiling of this many tasks per vCPU so a runaway fork loop stops short
# of the host's pid limit.
agents_cpu_weight: 25
agents_tasks_per_cpu: 1024

# systemd-oomd drop-in: kill inside agents.slice when memory pressure stays over
# the limit, so a thrashing slice is cut down before it stalls the desktop. Off
# by default because oomd must also watch the user manager, which is a per-host
# decision. The duration property exists in systemd 259 and may be unknown to
# an older systemd, which logs it and ignores it.
agents_oomd_enabled: false
agents_oomd_pressure_limit: "80%"
agents_oomd_pressure_duration: "20s"

# Delegate cpuset and io to the user manager, so agents.slice can reserve cores
# and an I/O weight. System scope (/etc/systemd/system), so it needs sudo, and
# the user manager reads it only at its next start (re-login or reboot). Off by
# default for that reason.
agents_delegate_enabled: false
agents_delegate_controllers: "cpu cpuset io memory pids"

# Contention-only: the desktop wins under load, and an idle machine still gives
# builds every core. 100 is the systemd default, so 30 is a bit under a third of
# the desktop's share when both want the CPU at once. This, not the shim's nice
Expand Down
116 changes: 116 additions & 0 deletions ansible/roles/developer-rust/tasks/agents_slice.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
---
# agents.slice: the cgroup that bounds every Claude Code session on the host.
# Linux only. hyperi-ai attaches each session to it at SessionStart, so the role
# only writes the unit. `agents_slice_enabled: false` removes what it deployed.

- name: Deploy the agents slice
when:
- ansible_facts['system'] == 'Linux'
- agents_slice_enabled | bool
block:
- name: Create the user systemd directory
ansible.builtin.file:
path: "{{ user_home }}/.config/systemd/user"
state: directory
owner: "{{ actual_user }}"
mode: '0755'
become: true
become_user: "{{ actual_user }}"

- name: Install the agents slice
ansible.builtin.template:
src: agents.slice.j2
dest: "{{ user_home }}/.config/systemd/user/agents.slice"
owner: "{{ actual_user }}"
mode: '0644'
become: true
become_user: "{{ actual_user }}"
register: developer_rust_agents_slice

- name: Create the agents slice drop-in directory
ansible.builtin.file:
path: "{{ user_home }}/.config/systemd/user/agents.slice.d"
state: directory
owner: "{{ actual_user }}"
mode: '0755'
become: true
become_user: "{{ actual_user }}"
when: agents_oomd_enabled | bool

- name: Install the oomd drop-in for the agents slice
ansible.builtin.template:
src: agents-oomd.conf.j2
dest: "{{ user_home }}/.config/systemd/user/agents.slice.d/oomd.conf"
owner: "{{ actual_user }}"
mode: '0644'
become: true
become_user: "{{ actual_user }}"
register: developer_rust_agents_oomd
when: agents_oomd_enabled | bool

- name: Remove the oomd drop-in for the agents slice
ansible.builtin.file:
path: "{{ user_home }}/.config/systemd/user/agents.slice.d/oomd.conf"
state: absent
become: true
become_user: "{{ actual_user }}"
register: developer_rust_agents_oomd_removed
when: not (agents_oomd_enabled | bool)

- name: Reload the user manager so the agents slice is live
ansible.builtin.systemd:
daemon_reload: true
scope: user
become: true
become_user: "{{ actual_user }}"
environment: "{{ developer_rust_user_env }}"
when:
- not ansible_check_mode
- developer_rust_agents_slice is changed
or developer_rust_agents_oomd is changed
or developer_rust_agents_oomd_removed is changed
failed_when: false

rescue:
- name: Record that the agents slice did not deploy
ansible.builtin.set_fact:
deploy_warnings: >-
{{ deploy_warnings | default([])
+ ['Agents slice: ' ~ (ansible_failed_task.name | default('unknown task'))
~ ' -- ' ~ (ansible_failed_result.msg | default('no message'))] }}

- name: Remove the agents slice (opt-out)
when:
- ansible_facts['system'] == 'Linux'
- not (agents_slice_enabled | bool)
block:
- name: Remove the agents slice and its drop-in
ansible.builtin.file:
path: "{{ user_home }}/.config/systemd/user/{{ item }}"
state: absent
loop:
- agents.slice
- agents.slice.d
become: true
become_user: "{{ actual_user }}"
register: developer_rust_agents_slice_removed

- name: Reload the user manager after removing the agents slice
ansible.builtin.systemd:
daemon_reload: true
scope: user
become: true
become_user: "{{ actual_user }}"
environment: "{{ developer_rust_user_env }}"
when:
- not ansible_check_mode
- developer_rust_agents_slice_removed is changed
failed_when: false

rescue:
- name: Record that the agents slice removal did not complete
ansible.builtin.set_fact:
deploy_warnings: >-
{{ deploy_warnings | default([])
+ ['Agents slice removal: ' ~ (ansible_failed_task.name | default('unknown task'))
~ ' -- ' ~ (ansible_failed_result.msg | default('no message'))] }}
5 changes: 3 additions & 2 deletions ansible/roles/developer-rust/tasks/governor.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,9 @@
{{ developer_rust_retired_governor_vars | join(', ') }}: retired. Builds
are no longer rationed -- there is no semaphore, no CPU reserve and no
job count, so nothing reads these. Memory is bounded by rustbuild.slice
(rust_governor_memory_high_pct / _memory_max_pct) and CPU is deferred by
nice 19 plus rust_governor_cpu_weight. Delete the setting.
(rust_governor_memory_high_pct / _memory_max_pct, sized from the host's
RAM) and CPU is deferred by nice 19 plus rust_governor_cpu_weight.
Delete the setting.
vars:
developer_rust_retired_governor_vars: "{{ lookup('ansible.builtin.varnames',
'^rust_governor_(serialize|slots|cpu_reserve_cores|build_allowance_gb|lock_wait_seconds|macos_qos)$',
Expand Down
17 changes: 17 additions & 0 deletions ansible/roles/developer-rust/tasks/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,23 @@
tags: ['developer-rust', 'rust-governor']
tags: ['developer-rust', 'rust-governor']

# The cgroup for agent sessions, sized in the same RAM budget as the governor's
# rustbuild.slice. Its own tag so it applies without a toolchain converge.
- name: Bound agent sessions
ansible.builtin.include_tasks:
file: agents_slice.yml
apply:
tags: ['developer-rust', 'agents-slice']
tags: ['developer-rust', 'agents-slice']

# System scope and off by default, so it stays out of the slice's task file.
- name: Delegate controllers to the user manager
ansible.builtin.include_tasks:
file: user_delegate.yml
apply:
tags: ['developer-rust', 'agents-delegate']
tags: ['developer-rust', 'agents-delegate']

# A `--tags removals` run reaches the cargo-home sweep here, since rust.yml is
# not selected by that tag. An install run sweeps from rust.yml instead.
- name: Remove cargo-home strays (opt-in)
Expand Down
55 changes: 55 additions & 0 deletions ansible/roles/developer-rust/tasks/user_delegate.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
---
# System-scope drop-in on user@.service, kept apart from agents_slice.yml
# because it is the one change here made under /etc. Off unless
# `agents_delegate_enabled: true`, and turning it off removes the file. The user
# manager reads it at its next start, so a converge does not apply it live and
# never restarts the manager (that would end the desktop session).

- name: Delegate more controllers to the user manager
when:
- ansible_facts['system'] == 'Linux'
- agents_delegate_enabled | bool
block:
- name: Create the user@.service drop-in directory
ansible.builtin.file:
path: /etc/systemd/system/user@.service.d
state: directory
owner: root
mode: '0755'
become: true

- name: Install the controller delegation drop-in
ansible.builtin.template:
src: user-delegate.conf.j2
dest: /etc/systemd/system/user@.service.d/hyperi-delegate.conf
owner: root
mode: '0644'
become: true
register: developer_rust_user_delegate

- name: Say the delegation waits for the next login # noqa: no-handler
ansible.builtin.set_fact:
deploy_warnings: >-
{{ deploy_warnings | default([])
+ ['Agents slice: user@.service now delegates '
~ agents_delegate_controllers
~ '. The user manager reads it at its next start, so log out'
~ ' and in (or reboot) for it to apply.'] }}
when: developer_rust_user_delegate is changed

rescue:
- name: Record that the delegation drop-in did not deploy
ansible.builtin.set_fact:
deploy_warnings: >-
{{ deploy_warnings | default([])
+ ['Agents slice delegation: ' ~ (ansible_failed_task.name | default('unknown task'))
~ ' -- ' ~ (ansible_failed_result.msg | default('no message'))] }}

- name: Remove the controller delegation drop-in
ansible.builtin.file:
path: /etc/systemd/system/user@.service.d/hyperi-delegate.conf
state: absent
become: true
when:
- ansible_facts['system'] == 'Linux'
- not (agents_delegate_enabled | bool)
7 changes: 7 additions & 0 deletions ansible/roles/developer-rust/templates/agents-oomd.conf.j2
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Deployed by hyperi-developer (developer-rust role). Edits are overwritten.
# Lets systemd-oomd kill inside agents.slice when its memory pressure stays over
# the limit for the duration, before a thrashing slice stalls the desktop.
[Slice]
ManagedOOMMemoryPressure=kill
ManagedOOMMemoryPressureLimit={{ agents_oomd_pressure_limit }}
ManagedOOMMemoryPressureDurationSec={{ agents_oomd_pressure_duration }}
17 changes: 17 additions & 0 deletions ansible/roles/developer-rust/templates/agents.slice.j2
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Deployed by hyperi-developer (developer-rust role). Edits are overwritten.
[Unit]
Description=Agent sessions (hyperi-developer)

# No dash in the name, so this is a direct child of the user manager and a true
# sibling of app.slice, where the desktop runs. See rustbuild.slice.j2.
[Slice]
# Sized from this host's RAM at converge: RAM minus the desktop's reserve minus
# the rust build slice's MemoryMax, so the three can never add up to more than
# the host has. An overrun is an OOM inside this cgroup, which kills its biggest
# process and leaves the desktop alone.
MemoryHigh={{ developer_rust_agents_high_mb }}M
MemoryMax={{ developer_rust_agents_max_mb }}M
# Yields to the desktop under contention and takes every idle core otherwise.
CPUWeight={{ agents_cpu_weight }}
# One ceiling for every session together, a fork-bomb stop rather than a budget.
TasksMax={{ developer_rust_agents_tasks_max }}
13 changes: 7 additions & 6 deletions ansible/roles/developer-rust/templates/rustbuild.slice.j2
Original file line number Diff line number Diff line change
Expand Up @@ -8,18 +8,19 @@ Description=Governed Rust builds (hyperi-developer)
# not the 30 below, that faces app.slice. Dash-free, this slice is a direct
# child of the user manager and a true sibling of the desktop's slices.
[Slice]
# Percentages resolve against this host's own RAM and swap, so one unit fits
# Memory limits are sized from this host's RAM at converge, so one role fits
# every machine. Both capture prongs land here: cargo trees via the
# hyperi-rust-govern scope, sccache-hosted compiles via Slice= on
# hyperi-sccache.service. This is the ONLY memory bound on a build: nothing
# queues builds or caps their job count, so a build over budget is throttled
# here and, at MemoryMax, killed here -- the build, never the host.
#
# MemorySwapMax only means anything where swap exists. On a swapless host
# MemoryHigh has nothing but page cache to reclaim, so it stalls a build
# instead of slowing it -- the zram_swap role is what makes this budget behave.
MemoryHigh={{ rust_governor_memory_high_pct }}%
MemoryMax={{ rust_governor_memory_max_pct }}%
# MemorySwapMax is a percentage systemd resolves against RAM, and only means
# anything where swap exists. On a swapless host MemoryHigh has nothing but
# page cache to reclaim, so it stalls a build instead of slowing it -- the
# zram_swap role is what makes this budget behave.
MemoryHigh={{ developer_rust_slice_high_mb }}M
MemoryMax={{ developer_rust_slice_max_mb }}M
MemorySwapMax={{ rust_governor_swap_max_pct }}%
# Yields to the desktop under contention; no effect while the machine is idle.
# This, not nice, is what makes a build defer on Linux: cgroup v2 splits CPU
Expand Down
6 changes: 6 additions & 0 deletions ansible/roles/developer-rust/templates/user-delegate.conf.j2
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# Deployed by hyperi-developer (developer-rust role). Edits are overwritten.
# The user manager only delegates cpu, memory and pids by default. cpuset and io
# let agents.slice reserve cores and an I/O weight. Read at the user manager's
# next start.
[Service]
Delegate={{ agents_delegate_controllers }}
29 changes: 29 additions & 0 deletions ansible/roles/developer-rust/vars/main.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
---
# Limits worked out from this host's RAM and CPUs. Role vars, not defaults,
# because they are derived: set the inputs in defaults/main.yml instead.
# All sizes are MiB, the unit memtotal_mb reports.

developer_rust_ram_mb: "{{ ansible_facts['memtotal_mb'] | int }}"

# rustbuild.slice. High never exceeds Max.
developer_rust_slice_max_mb: >-
{{ [developer_rust_ram_mb | int * rust_governor_memory_max_pct | int // 100,
rust_governor_memory_max_floor_mb | int] | max }}
developer_rust_slice_high_mb: >-
{{ [[developer_rust_ram_mb | int * rust_governor_memory_high_pct | int // 100,
rust_governor_memory_high_floor_mb | int] | max,
developer_rust_slice_max_mb | int] | min }}

# What the desktop keeps, and what is left for agent sessions once builds have
# their share. A host without the governor gives builds no share to reserve.
developer_rust_agents_reserve_mb: >-
{{ [developer_rust_ram_mb | int * agents_reserve_pct | int // 100,
agents_reserve_floor_mb | int] | max }}
developer_rust_agents_max_mb: >-
{{ [developer_rust_ram_mb | int - developer_rust_agents_reserve_mb | int
- (developer_rust_slice_max_mb | int if rust_governor_enabled | bool else 0),
agents_memory_max_floor_mb | int] | max }}
developer_rust_agents_high_mb: >-
{{ developer_rust_agents_max_mb | int * agents_memory_high_pct | int // 100 }}
developer_rust_agents_tasks_max: >-
{{ agents_tasks_per_cpu | int * ansible_facts['processor_vcpus'] | int }}
Loading
Loading