Repository navigation
mega65_ftp: faster Ethernet uploads, and fixes for files of 2GB and larger - #247
Open
Mikey351AU wants to merge 7 commits into
Open
Mikey351AU wants to merge 7 commits into
Mikey351AU wants to merge 7 commits into
Conversation
- Use the 28-packet window on R5/R6 (xc7a200t, 32 ETH RX buffers), not 3 - Flush the write queue at 64KB in Ethernet mode (helper accepts 128-sector batches) - Send queued sector writes in ascending order so file data forms one batch Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SaBDrQJU9NJb1UxBiLfpfd
- Start the free-cluster search at the FAT32 FSInfo next-free hint and keep it updated in-session; wrap to the start of the FAT if nothing fits after it - Write the hint and an adjusted free-cluster count back to FSInfo after upload, mkdir and delete - Read the FAT 128 sectors per request while scanning (was 16) - mkdir: '..' of a directory in the root now points at cluster 0 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SaBDrQJU9NJb1UxBiLfpfd
The helper verified each 128-sector batch before sending the final ack, so the PC sat idle for the whole read-back. Now the verify runs from process() after the ack is out, before any new packet is handled (the RX buffers hold the next batch meanwhile). A failed verify still halts the helper. mega65_ftp does one extra round trip at the end of upload/mkdir/delete so a failure in the last batch is still reported. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SaBDrQJU9NJb1UxBiLfpfd
- upload: remaining length was an int, so files >= 2GB wrote no data at all but still created a directory entry with the full size (and one cluster) - directory entries: file size was assembled with int shifts and stored in a long, so sizes >= 2GB went negative and those files were hidden from dir/get/del; now parsed as unsigned 32-bit into a long long - download: remaining byte count is now 64-bit - reject files over the FAT32 limit (4GB - 1) with a clear error Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SaBDrQJU9NJb1UxBiLfpfd
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SaBDrQJU9NJb1UxBiLfpfd
…6_address discoverfd is an array, so 'discoverfd < 0' compared the array address and a failed socket() call was never noticed. It is also a hard error in g++ 13+, which stopped the gtest suite from building. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SaBDrQJU9NJb1UxBiLfpfd
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SaBDrQJU9NJb1UxBiLfpfd
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
mega65_ftp: faster Ethernet uploads, and fixes for files of 2GB and larger
About this PR: I'm a MEGA65 owner, not a developer. The code changes were written by Claude (Anthropic's AI assistant) at my request, and the commits are authored that way. I tested the result on my MEGA65 R6 over Ethernet from Windows. Details of what was and wasn't tested are below.
Why
Uploading over Ethernet to my R6 ran at about 190 KB/s. Uploading a 5.3MB core took 27 seconds, and about half of that was spent before any data was sent. Files of 2GB or larger "uploaded" instantly but no data was written.
Changes (one commit each)
determine_ethernet_window_size()gives the 28-packet window to R1/R3/R4/MEGAphone (0x01/0x03/0x04/0x21), but not to R5 (0x05) or R6 (0x06). Those fall back to 3, even thoughmega65r5.vhdlandmega65r6.vhdlsetnum_eth_rx_buffers => 32, like R3/R4.This commit also changes the write queue:
lcommand, andremotesd_ethaccepts batches of up to 128 sectors.XXX - Sort sector number order...comment.find_contiguous_clusters()scanned the FAT from cluster 0 on every upload. On a 120GB card with 4KB clusters, 239,574 sectors per FAT and about 10GB used, that read about 10MB of FAT, around 13 seconds per upload. Now:mkdirin the root directory, which wrote the parent cluster into..when it should be 0 (fsck.fat reported "Invalid '..' entry").remotesd_eth: ack a write batch before verifying it. The read-back verify now runs after the last ack has been sent, and before any new packet is processed. A failed verify still halts the helper.mega65_ftpmakes one extra round trip at the end of upload/mkdir/delete, so a failure in the last batch is still reported. To be clear: this gave only about 2% in my tests, because the SD card is still busy during the verify. Feel free to drop this commit.remaining_lengthwas anint, so files of 2GB or more wrote no data but still created a directory entry with the full size and a single cluster.d_filelenwas built withintshifts into along(32-bit on Windows), so sizes of 2GB or more came out negative and those files were hidden fromdir,getanddel.ldirprinted local file sizes of 2GB or more as negative numbers.etherload_common.c:if (discoverfd < 0)should bediscoverfd[idx]. As written, a failedsocket()call was never caught, and it's a hard error in g++ 13+, which stopped the gtest suite from building.Results (MEGA65 R6, Windows, 5,268,104-byte .cor file)
The window fix (1) on its own made no measurable difference in this test, because the free cluster search and the SD card were the bottlenecks. It's included because R5/R6 clearly have the same 32 RX buffers as R3/R4.
Testing
-dmode:dir,get(byte-identical) anddelfsck.fat -nreports the image clean after each of thesemega65_ftp.testgives 29 passed, 1 failed (CreateDeleteDirectoryWithFiles). The same test fails on unmodifieddevelopment, so it's not caused by this PR.