Skip to content

Commit 893d2d3

Browse files
Subbarao Garlapatimeta-codesync[bot]
authored andcommitted
Don't reify a JIT frame when there is no thread state
Summary: On D115743871 when I ran the AI Labs test `apf_perf_cogwheel_mtml_ig_fm_multihost_train` I was hitting an error in all the runs with the JIT enabled. AI Labs Run: https://www.internalfb.com/family_of_labs/test_results/175921865618156?test_name=ai_lab.APF.apf_perf_cogwheel_mtml_ig_fm_multihost_train Treatment #1 MAST job logs [link](https://www.internalfb.com/mlhub/pipelines/runs/mast/aps-cogwheel_aps_multihost_ig_fm_v4-a3ec2e8726?job_attempt=2&version=0&tab=logs) **Relevant Stack Trace (condensed)** ``` PyErr_GetRaisedException ← faults here _PyFrame_ReifyFrame PyUnstable_InterpreterFrame_GetLine (unknown) ×2 ← profiler's stack walker, unsymbolized ──── libc_sigaction ──── epoll_wait select_epoll_poll ← inside Py_BEGIN_ALLOW_THREADS cinderx::jit::rt::call ×2 ← JIT frames ``` It seems like this happened when strobelight was trying to get the line number of a jit frame and the frame was being reified. Because the GIL was released it wasn't possible to do a local thread state get in `_PyFrame_ReifyFrame()`. The fix here is to not reify the frame if there is no thread state. Reviewed By: DinoV Differential Revision: D116513961 fbshipit-source-id: efb877259d5342c9cfc05958e4076064ed0b728c
1 parent 4fc352a commit 893d2d3

1 file changed

Lines changed: 6 additions & 0 deletions

File tree

Python/frame.c

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -149,6 +149,12 @@ _PyFrame_ClearExceptCode(_PyInterpreterFrame *frame)
149149
int
150150
_PyFrame_InitializeExternalFrame(_PyInterpreterFrame *frame)
151151
{
152+
// Profilers can reach this from a signal handler on a thread that has
153+
// released the GIL, where there's no thread state to reify with.
154+
if (PyThreadState_GetUnchecked() == NULL) {
155+
return 0;
156+
}
157+
152158
PyObject *executor = PyStackRef_AsPyObjectBorrow(frame->f_executable);
153159
if (PyUnstable_JITExecutable_Check(executor)) {
154160
PyUnstable_PyJitExecutable *jit_exec = (PyUnstable_PyJitExecutable *)executor;

0 commit comments

Comments
 (0)