Commit 893d2d3
Don't reify a JIT frame when there is no thread state
Summary:
On D115743871 when I ran the AI Labs test `apf_perf_cogwheel_mtml_ig_fm_multihost_train` I was hitting an error in all the runs with the JIT enabled.
AI Labs Run: https://www.internalfb.com/family_of_labs/test_results/175921865618156?test_name=ai_lab.APF.apf_perf_cogwheel_mtml_ig_fm_multihost_train
Treatment #1 MAST job logs [link](https://www.internalfb.com/mlhub/pipelines/runs/mast/aps-cogwheel_aps_multihost_ig_fm_v4-a3ec2e8726?job_attempt=2&version=0&tab=logs)
**Relevant Stack Trace (condensed)**
```
PyErr_GetRaisedException ← faults here
_PyFrame_ReifyFrame
PyUnstable_InterpreterFrame_GetLine
(unknown) ×2 ← profiler's stack walker, unsymbolized
──── libc_sigaction ────
epoll_wait
select_epoll_poll ← inside Py_BEGIN_ALLOW_THREADS
cinderx::jit::rt::call ×2 ← JIT frames
```
It seems like this happened when strobelight was trying to get the line number of a jit frame and the frame was being reified. Because the GIL was released it wasn't possible to do a local thread state get in `_PyFrame_ReifyFrame()`. The fix here is to not reify the frame if there is no thread state.
Reviewed By: DinoV
Differential Revision: D116513961
fbshipit-source-id: efb877259d5342c9cfc05958e4076064ed0b728c1 parent 4fc352a commit 893d2d3
1 file changed
Lines changed: 6 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
149 | 149 | | |
150 | 150 | | |
151 | 151 | | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
152 | 158 | | |
153 | 159 | | |
154 | 160 | | |
| |||
0 commit comments