nanobind: extremely rare test failure on macos only

[rpoyner-tri]

What happened?

First seen in CI: https://drake-cdash.csail.mit.edu/tests/2201738892

The //bindings/pydrake/geometry:alt_binder/common_test fails, with some very rare frequency, perhaps 1% and in some circumstances, lower.

Only ever observed on macos, both Sequoia and Tahoe machines can exhibit the problem.

History bisect indicates that this failure has been around for as long as running that test with nanobind has been working at all. That means at least since merge of PR #24837. That PR doesn't appear to me to be related at all.

The sampling that seems to find failures ever is over separate invocations/processes running the test. It doesn't seem that repeating the vulnerable tests in a healthy process has much chance of increasing failure rate.

Version

v1.55.0 plus current master

What operating system are you using?

macOS 26 (Tahoe)

What installation option are you using?

compiled from source code using Bazel

Relevant log output

======================================================================
FAIL [0.012s]: test_proximity_properties (common_test.TestGeometryCore.test_proximity_properties)
Tests the utility functions (not related to hydroelastic contact) for
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/ricopoyner/Library/Caches/bazel/_bazel_ricopoyner/27b47a6d9b400570878eb2115555e985/sandbox/darwin-sandbox/22438/execroot/_main/bazel-out/darwin_arm64-dbg/bin/bindings/pydrake/geometry/alt_binder/common_test/common_test.runfiles/_main/bindings/pydrake/geometry/test/common_test.py", line 521, in test_proximity_properties
    self.assertEqual(
    ~~~~~~~~~~~~~~~~^
        props.GetProperty("hydroelastic", "resolution_hint"), res_hint
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
AssertionError: <pydrake.multibody.plant.CoulombFriction object at 0x1096a7400> != 0.175

----------------------------------------------------------------------
Ran 16 tests in 0.019s

FAILED (failures=1, skipped=14)

[rpoyner-tri]

The log pasted above is representative, but there are some variations in what line from the unit test trips the bug. What does seem pretty much constant is that a py::cast from a c++ AbstractValue[double] returns a python AbstractValue[CoulombFriction]. I haven't been able to dissect the nanobind cast implementation fully yet, but the best theory I have is that some subtle UB eventually poisons the type lookup data internal to nanobind.

[rpoyner-tri]

Switching from Sequoia to Tahoe seemed to reduce the failure rate, and switching from release to debug builds seemed to reduce it even further, but not to zero. Maybe to 0.05% (ugh).

[rpoyner-tri]

From discussions, Nimmer suggested that we might have an instance pointer in a nanobind type map that is not getting evicted when the instance changes/goes away. It's a good theory, but tracking it down on macos will still be a chore.

FWIW, I took a look at the source for pre-release nanobind 3.0. It shuffles the type lookup code around a bit, but still does the same pointer based association algorithm. I don't expect it would make this symptom disappear.