Skip to content

mega65_ftp: faster Ethernet uploads, and fixes for files of 2GB and larger - #247

Open
Mikey351AU wants to merge 7 commits into
MEGA65:developmentfrom
Mikey351AU:ethernet-upload-speedup
Open

Mikey351AU wants to merge 7 commits into
MEGA65:developmentfrom
Mikey351AU:ethernet-upload-speedup

Conversation

@Mikey351AU

Copy link
Copy Markdown

mega65_ftp: faster Ethernet uploads, and fixes for files of 2GB and larger

About this PR: I'm a MEGA65 owner, not a developer. The code changes were written by Claude (Anthropic's AI assistant) at my request, and the commits are authored that way. I tested the result on my MEGA65 R6 over Ethernet from Windows. Details of what was and wasn't tested are below.

Why

Uploading over Ethernet to my R6 ran at about 190 KB/s. Uploading a 5.3MB core took 27 seconds, and about half of that was spent before any data was sent. Files of 2GB or larger "uploaded" instantly but no data was written.

Changes (one commit each)

  1. Ethernet window for R5/R6. determine_ethernet_window_size() gives the 28-packet window to R1/R3/R4/MEGAphone (0x01/0x03/0x04/0x21), but not to R5 (0x05) or R6 (0x06). Those fall back to 3, even though mega65r5.vhdl and mega65r6.vhdl set num_eth_rx_buffers => 32, like R3/R4.
    This commit also changes the write queue:
    • In Ethernet mode it flushes at 64KB instead of 32KB. The 32KB limit only matters for the serial l command, and remotesd_eth accepts batches of up to 128 sectors.
    • Sector writes go out in ascending sector order. Before, FAT and directory sectors queued in the middle of the file data split one contiguous run into several batches, and each batch boundary means a full ack drain plus a read-back verify. This was the XXX - Sort sector number order... comment.
  2. Free cluster search. find_contiguous_clusters() scanned the FAT from cluster 0 on every upload. On a 120GB card with 4KB clusters, 239,574 sectors per FAT and about 10GB used, that read about 10MB of FAT, around 13 seconds per upload. Now:
    • The search starts from the FAT32 FSInfo next-free hint and wraps to the start of the FAT if nothing fits after it.
    • The hint is kept up to date during a session and written back to FSInfo after upload, mkdir and delete. The FSInfo free-cluster count is adjusted too; before, it was never updated.
    • FAT scans read 128 sectors per request instead of 16.
    • It also fixes mkdir in the root directory, which wrote the parent cluster into .. when it should be 0 (fsck.fat reported "Invalid '..' entry").
  3. remotesd_eth: ack a write batch before verifying it. The read-back verify now runs after the last ack has been sent, and before any new packet is processed. A failed verify still halts the helper. mega65_ftp makes one extra round trip at the end of upload/mkdir/delete, so a failure in the last batch is still reported. To be clear: this gave only about 2% in my tests, because the SD card is still busy during the verify. Feel free to drop this commit.
  4. Files of 2GB and larger.
    • Upload: remaining_length was an int, so files of 2GB or more wrote no data but still created a directory entry with the full size and a single cluster.
    • Directory listings: d_filelen was built with int shifts into a long (32-bit on Windows), so sizes of 2GB or more came out negative and those files were hidden from dir, get and del.
    • Download: the remaining-bytes count was 32-bit.
    • Files over 4GB-1 are now rejected with a clear error.
  5. ldir printed local file sizes of 2GB or more as negative numbers.
  6. etherload_common.c: if (discoverfd < 0) should be discoverfd[idx]. As written, a failed socket() call was never caught, and it's a hard error in g++ 13+, which stopped the gtest suite from building.
  7. clang-format on the changed lines.

Results (MEGA65 R6, Windows, 5,268,104-byte .cor file)

Original This PR
Free cluster search 13.7 s 0.08 s (once FSInfo has a good hint)
Data transfer 13.4 s ~13–16 s (the SD card is the limit: write plus read-back verify)
Total 27 s 16 s

The window fix (1) on its own made no measurable difference in this test, because the free cluster search and the SD card were the bottlenecks. It's included because R5/R6 clearly have the same 32 RX buffers as R3/R4.

Testing

  • On hardware (R6, Windows build): uploads, deletes, and a partial upload of a 4GB file that was stopped part-way. Not tested on hardware: mkdir, or a complete upload of a file of 2GB or more.
  • FAT32 card images, Linux, direct -d mode:
    • 2.2GB file: upload, dir, get (byte-identical) and del
    • a file over 4GB is rejected
    • on a fragmented image: upload, overwrite, mkdir, delete, and FSInfo hint missing / wraparound cases
    • fsck.fat -n reports the image clean after each of these
  • Write queue: checked with a harness that stubs out the Ethernet layer. Every sector arrives intact, and there are 16 batches instead of 40, with a largest batch of 126 sectors instead of 62.
  • gtest: mega65_ftp.test gives 29 passed, 1 failed (CreateDeleteDirectoryWithFiles). The same test fails on unmodified development, so it's not caused by this PR.

- Use the 28-packet window on R5/R6 (xc7a200t, 32 ETH RX buffers), not 3
- Flush the write queue at 64KB in Ethernet mode (helper accepts 128-sector batches)
- Send queued sector writes in ascending order so file data forms one batch

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SaBDrQJU9NJb1UxBiLfpfd
- Start the free-cluster search at the FAT32 FSInfo next-free hint and keep it
  updated in-session; wrap to the start of the FAT if nothing fits after it
- Write the hint and an adjusted free-cluster count back to FSInfo after
  upload, mkdir and delete
- Read the FAT 128 sectors per request while scanning (was 16)
- mkdir: '..' of a directory in the root now points at cluster 0

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SaBDrQJU9NJb1UxBiLfpfd
The helper verified each 128-sector batch before sending the final ack, so the
PC sat idle for the whole read-back. Now the verify runs from process() after
the ack is out, before any new packet is handled (the RX buffers hold the next
batch meanwhile). A failed verify still halts the helper. mega65_ftp does one
extra round trip at the end of upload/mkdir/delete so a failure in the last
batch is still reported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SaBDrQJU9NJb1UxBiLfpfd
- upload: remaining length was an int, so files >= 2GB wrote no data at all but
  still created a directory entry with the full size (and one cluster)
- directory entries: file size was assembled with int shifts and stored in a
  long, so sizes >= 2GB went negative and those files were hidden from
  dir/get/del; now parsed as unsigned 32-bit into a long long
- download: remaining byte count is now 64-bit
- reject files over the FAT32 limit (4GB - 1) with a clear error

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SaBDrQJU9NJb1UxBiLfpfd
…6_address

discoverfd is an array, so 'discoverfd < 0' compared the array address and a
failed socket() call was never noticed. It is also a hard error in g++ 13+,
which stopped the gtest suite from building.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SaBDrQJU9NJb1UxBiLfpfd
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SaBDrQJU9NJb1UxBiLfpfd
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants