Skip to content

Fix gc sweeping binaries - #1

Merged
pedroperrone merged 9 commits into
mainfrom
fix-gc-sweeping-binaries
Jul 22, 2026
Merged

pedroperrone merged 9 commits into
mainfrom
fix-gc-sweeping-binaries

Conversation

@pedroperrone

Copy link
Copy Markdown
Member

ArrayInterface.from_tensor/1 calls Nx.to_binary/1 on the tensor, captures the raw memory address of that binary via get_binary_address/1, and stores the address in the struct. The XGBoost NIF later receives this address as a plain integer embedded in a JSON string and dereferences it directly to read the tensor data.

There are two issues with this.

1. Small tensors produce heap binaries with unstable addresses

The BEAM allocates binaries in two ways:

  • Heap binaries (≤64 bytes): live on the calling process's heap. The GC can move them during heap compaction, invalidating any captured address.
  • Refc binaries (>64 bytes): live off-heap in a reference-counted area. Their address is stable — GC never moves them.

A {1, 4} float32 tensor (a common input shape for single-sample inference) is only 16 bytes — well below the threshold. The captured address becomes stale as soon as GC compacts the heap, and the NIF reads garbage from it. Because ExUnit runs tests in random order and GC is triggered by heap pressure, whether the bug manifests depends on which tests ran before the prediction call and how much heap they allocated. This is why it's seed-dependent.

The fix is to pad the binary to above 64 bytes before capturing its address, forcing a refc binary with a stable off-heap address. XGBoost reads only the bytes corresponding to the tensor's actual shape, so the padding zeros are never accessed.

defp ensure_off_heap_binary(binary) when byte_size(binary) > 64, do: binary
defp ensure_off_heap_binary(binary),
  do: binary <> :binary.copy(<<0>>, 65 - byte_size(binary))

2. The binary has no live reference during the NIF call

In inplace_predict, the ArrayInterface struct is piped directly into Jason.encode! and then discarded — only the resulting JSON string is kept. After that line the struct, and the binary it holds, have no live references. This means the refc binary's reference count can drop to zero and its off-heap memory can be freed while the dirty NIF is still running and reading from that address.

The fix is to bind the ArrayInterface struct to a variable before encoding it and keep a reference to its binary alive past the NIF call, so it remains a live GC root for the full duration of the call.

interface = ArrayInterface.from_tensor(data)
data_interface = Jason.encode!(interface)
{shape, preds} = EXGBoost.NIF.booster_predict_from_dense(...) |> Internal.unwrap!()
_keep_alive = interface.binary

@pedroperrone
pedroperrone force-pushed the fix-gc-sweeping-binaries branch from 943ca50 to 2f43a21 Compare July 22, 2026 18:48
@pedroperrone
pedroperrone force-pushed the fix-gc-sweeping-binaries branch from 2f43a21 to 1bc708e Compare July 22, 2026 18:52
@pedroperrone
pedroperrone merged commit fcb4ac0 into main Jul 22, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant