Fuzzer livelock and possible lost AMs with AM fragmentation

[elliottslaughter]

I'm tracking an apparent livelock that is potentially related to StanfordLegion/realm#410. Initial debugging by @lightsighter suggested possible lost messages in the Realm barrier implementation: https://legion.zulipchat.com/#narrow/dm/210059,501553-dm/near/598962286

Configuration that replicates (rare, about 1 in 20,000 runs):

• Realm PRs: 410, 428, 435, 444 (now merged)
• GASNet PSHM disabled
• GASNet AM medium limit: 1024 bytes
• Legion MRs: 2052 (merged with master)
• -fuzz:ops 10 -ll:util 2 -ll:cpu 3 -fuzz:replicate 1 -fuzz:width 128 -fuzz:branch 16

Configuration that does NOT replicate (after 6 jobs of 20,000 runs each):

• Same as above except:
• No Realm PR 410
• GASNet PSHM enabled
• GASNet AM medium limit: 64 KB

I'm currently extending the run to 100 ops to see if that will trigger something. Otherwise my next idea is to turn PSHM off in the non-replicating configuration to see if that tweaks the timing enough to make it replicate.

Current hypotheses:

• Bug in Realm PR 410
• Bug in GASNet IBV conduit (hidden in the non-replicating run due to PSHM; seems unlikely)
• Bug is latent but hidden by some combination of the above

[elliottslaughter]

While it's not the same failure mode, I want to note here that I just hit a validation failure in a test configuration identical to the original except:

• No Realm PR 410
• GASNet AM medium limit: 64 KB
• (GASNet PSHM is disabled like the original)

I.e., the key difference vs my set of 6 runs is PSHM is disabled. Therefore, either there is a bug in the network path somewhere or it twiddles the timing enough to expose new bugs (whether related to the original failure mode or otherwise).

Edit: this has been reported as #1946.

[elliottslaughter]

I'm currently extending the run to 100 ops to see if that will trigger something.

I finished roughly 20,000 runs in this configuration and did not hit any failures or hangs of any type. While I did hit the new failure mode in #1946, that is so far the only new notable failure I have hit.

Therefore, current evidence suggests that the livelock is specific to StanfordLegion/realm#410.