fix: stream large file uploads from disk to avoid memory exhaustion and proxy timeouts

[devin-ai-integration]

Summary

When write_binary is called with a file path (string) for files larger than 5 MB, the SDK previously loaded the entire file into memory with Path.read_bytes() before passing it to the multipart upload path. For multi-GB files (e.g., 17 GB), this causes:

1. Memory exhaustion — Python can't allocate 17 GB of RAM on most systems
2. Proxy timeouts — If the file is sent as a single PUT request body instead of chunked multipart, CloudFront/OpenResty times out after ~120 seconds

Fix: Added _upload_file_with_multipart() which uses os.pread() to read 5 MB chunks directly from the file descriptor at absolute offsets, uploading them in parallel batches without ever holding more than ~15 MB in memory (3 parallel × 5 MB).

Changes applied to both async (default/filesystem.py) and sync (sync/filesystem.py) versions.

Review & Testing Checklist for Human

• Test uploading a large file (>1 GB) from a local path: sandbox.fs.write_binary(\"/remote/path\", \"/local/path/to/bigfile\")
• Verify that memory usage stays flat during upload (not proportional to file size)
• Verify upload completes without 504 timeout for files that previously failed
• Confirm small file uploads (<5 MB) still work normally via the regular code path

Notes

• os.pread() is used for positional reads (thread-safe, no seek needed) — available on all POSIX systems and Windows (Python 3.3+)
• The existing _upload_with_multipart(data: bytes) method is preserved for callers passing in-memory bytes directly
• Related PRs: sandbox-api #207 (async assembly), sdk-typescript #310 (same fix for TypeScript), seaweedfs #24 (FUSE crash fix)

Link to Devin session: https://app.devin.ai/sessions/25c0534a826d4e088768cf2c15fa1d53
Requested by: @Joffref

---

Note

Adds streaming multipart upload for large files (>5 MB) in both async and sync filesystem clients. Uses os.pread() to read chunks directly from disk, avoiding loading multi-GB files into memory. A follow-up commit adds thread exception propagation in the sync version to prevent _complete_multipart_upload from being called with missing parts on partial failure.

^{Written by Mendral for commit 2f79407.}

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR that start with 'DevinAI' or '@devin'.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[mendral-app]

LGTM

The previous bug (silently swallowed thread exceptions) is fixed — exceptions list is now populated and re-raised after join(). Both async and sync paths correctly abort the multipart upload on failure. No new issues found.

_{Tag @mendral-app with feedback or questions. View session}