fix(lambda): share HTTP sessions so URLSession teardown stops killing the process - #48
Merged
Merged
Conversation
… the process
`/image` returned a 500 on roughly one request in four. The handler always
finished its work first; the process then died with:
Object 0x... of class _MultiHandle deallocated with non-zero retain count 2.
and Lambda reported `Runtime.ExitError`.
`NetworkRequestManager` built a fresh Alamofire `Session` on every call, and
`URLProcessor` calls into it up to a dozen times per request. On Linux an
Alamofire `Session` owns a `URLSession`, and releasing it runs
swift-corelibs-foundation's `URLSession._MultiHandle.deinit`. That deinit calls
`curl_multi_remove_handle`/`curl_multi_cleanup`, which synchronously re-enter the
registered `CURLMOPT_TIMERFUNCTION`; for a zero timeout that reaches
`updateTimeoutTimer(to: .immediate)`, which does `queue.async { nonisolatedSelf... }`
and takes a strong reference to the object being deinitialized. The Swift runtime
turns that into a fatal error, or segfaults on the dangling reference.
The three per-call sessions become process-lifetime statics, keeping their exact
headers and timeouts. Nothing is deallocated, so the teardown never runs. The
DigitalNZ requests already went through Alamofire's process-lifetime `AF`.
Reproduced in a Linux container (swift:6.3-amazonlinux2023, arm64) running the
Lambda's local server under CPU pressure: 5 of 40 requests failed, all five with
the `_MultiHandle` fatal error. After the fix, 60 of 60 passed under the same
harness.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
GET /imagereturned{"message":"Internal Server Error"}on roughly one request in four. CloudWatch showed the handler completing all of its work first, then the process dying:Root cause
NetworkRequestManagerbuilt a fresh AlamofireSessionon every call, in five different methods, andURLProcessorcalls into it up to a dozen times per request.On Linux an Alamofire
Sessionowns aURLSession(Session.deinitcallsinvalidateAndCancel), and releasing thatURLSessionruns swift-corelibs-foundation'sURLSession._MultiHandle.deinit. That deinit callscurl_multi_remove_handleandcurl_multi_cleanup, which synchronously re-enter the registeredCURLMOPT_TIMERFUNCTIONcallback. For a zero timeout the callback reachesupdateTimeoutTimer(to: .immediate):which takes a strong reference to the object currently being deinitialized. The Swift runtime detects the resurrected reference and aborts the process (sometimes it segfaults on the dangling reference instead). Source: swift-corelibs-foundation,
swift-6.3-RELEASE,Sources/FoundationNetworking/URLSession/libcurl/MultiHandle.swift.The crash backtrace from the local repro confirms the path end to end:
It is a race, so it only fires under scheduling pressure, which a 512 MB Lambda has plenty of.
The fix
The three distinct session configurations become process-lifetime
static lets onNetworkRequestManager(browserSession,shortTimeoutBrowserSession,rangeProbeSession), keeping exactly the headers and timeouts each caller had before. Nothing is ever deallocated, so the teardown never runs. DigitalNZ requests already used Alamofire's process-lifetimeAFglobal, which is why the two DigitalNZ calls in the logs never crashed.Alternative considered and rejected for now: migrating to AsyncHTTPClient. It would also avoid corelibs
URLSession, but it rewrites every request path, validation, decoding, redirect and HEAD/Range handling across 50+ collection strategies, for no additional benefit over a session that is never released.Verification
Linux container (
swift:6.3-amazonlinux2023, arm64,--cpus 0.3 -m 512m, plus busy loops for scheduling pressure) running the Lambda's local server, hammeringPOST /invokewith random collections:ok=35 fail=5of 40, all 5 with the_MultiHandlefatal errorok=60 fail=0of 60, zero fatal errorsCPU pressure is required to reproduce: unstressed runs pass 60/60 even on the broken build.
swift test: 132 tests, 0 failures.Regression test
Tests/NZImageApiLambdaTests/NetworkRequestManagerSessionTests.swiftchecks each shared session still carries its expected User-Agent,Rangeheader and timeout, and scans the Lambda target's source for any session construction outside the single shared factory. That source scan was verified to actually fail: reintroducing a per-callSession(configuration:)makes it report the offending file and line.Docs
CLAUDE.mdgains a gotcha, and.claude/rules/architecture.mdgains an "HTTP client lifecycle" section explaining the rule and how to re-verify it.🤖 Generated with Claude Code