[Feature Request] Chunked File Transfer System with Configurable Partition Size

[mthsmb]

The feature I would like to see.

I've been dealing with a frustrating issue that's been around since Pterodactyl (issue #4696 - closed but never actually fixed). For those of us running infrastructure behind Cloudflare Zero Trust, transferring large files is basically impossible right now.

The main problems I'm hitting:

• Cloudflare Tunnels hard limit of 100MB per upload
• Transfers timing out around the 2-minute mark
• Nginx proxy getting in the way
• When something fails, I'm starting from scratch - no resume option
• Zero recovery if a transfer dies mid-way

This makes working with any substantial files a nightmare, and the workarounds either don't work reliably or defeat the whole purpose of using Zero Trust in the first place.

How I'd like to see it work.

I'd love to see a proper chunked upload system that actually works with Cloudflare's limitations. Here's what I think would make this work:

Chunk Management:

• Default to 95MB chunks (staying safely under that 100MB limit)
• Auto-detect Cloudflare and enforce the 95MB cap
• Handle splitting files automatically and putting them back together
• Verify each chunk completed successfully

Configuration options I'd want:

• Let me adjust chunk size (with that 95MB hard cap when Cloudflare is detected)
• Configure how many chunks can go at once
• Set retry attempts per chunk
• Adjust timeouts

Recovery system:

• Retry failed chunks independently
• Pick up where I left off instead of restarting everything
• Verify chunk integrity
• Actually tell me what went wrong and where

UI improvements:

• Show me progress per chunk and overall
• Let me pause and resume transfers
• Clear error messages that tell me which chunk failed

Why this matters to me.

This has been plaguing the community for years. The original Pterodactyl issue acknowledged the problem but never shipped a fix before closing it. For anyone serious about security and using Zero Trust, this is a daily headache.

What would make Pelican stand out here:

• Smart Cloudflare detection to automatically adjust chunk sizes
• Efficient chunk handling without crazy overhead
• Actually useful error messages
• Save progress so long transfers survive restarts
• Fall back to normal transfers when Cloudflare isn't in the picture

Since Pelican is positioning itself as the next evolution of Pterodactyl, solving this would be huge for the community - especially those of us running production deployments where security isn't optional.

[NerdsCorp]

@mthsmb out of curiosity, have you tried sftp over zero trust? I believe there are still issues with node to node transfers the team is going to work on.

[Marrcell]

I noticed that the problem with Cloudflare will disappear when the target server will download the transfer files, and currently it is the source server that pushes them, Cloudflare has no limit when downloading files (when the files are downloaded from the target server)

[lancepioch]

Closing as a duplicate of #846, please continue the conversation there.