README clarification

[bladeacer]

Based on your README it says:

Tool Time Speed vs Surge
Surge 28.93s 35.40 MB/s —
aria2c 40.04s 25.57 MB/s 1.38× slower
curl 57.57s 17.79 MB/s 1.99× slower
wget 61.81s 16.57 MB/s 2.14× slower

is this true for surge get in the CLI?

surge's TUI seems much faster, though that might be due to the fact that there is perceived progress bar and stats in the TUI.

[SuperCoolPencil]

Hi!

Yes, the CLI, TUI, and the headless instance all use the same backend, so the actual performance is identical across all of them.

The difference you're feeling is likely just perceived because of how the TUI handles the progress bars and live stats.

If you want to see the numbers, you can check out the latest runs from our benchmark CI in GitHub Actions (it uses get command), the speed results there should match up with what's in the README

[bladeacer]

Thanks for the clarification :D

I would like to contribute benchmark tests on Wayland EndeavourOS (Arch Linux based). Would that be fine with you?

[SuperCoolPencil]

That would be awesome!

[bladeacer]

Thanks for letting me know.

Hmm, I realised that the choice of Wayland or X11 should not really affect the overall download speed.

Like you mentioned there is the factor of CDN cache, network latency and one's hardware specs.

In general surge seems to be much faster (under a minute) while wget and curl both take over 200 seconds.

For more benchmarking features you could consider using hyperfine. Though that being said the Python script is fine.

https://github.com/sharkdp/hyperfine

Not sure if wget, curl or aria2c has chunking capabilities. A comparison with max settings (and no too many requests error) could be good.

[bladeacer]

stdout is real nice

Surge Benchmark Suite
========================================

  Test URL:   https://sin-speed.hetzner.com/1GB.bin
  Iterations: 5
  Temp Dir: /tmp/surge_bench_sgt_r9yf

SETUP
----------------------------------------
  [OK] Go found
  Building surge...
    [OK] Surge built successfully
    [X] aria2c not found (install aria2)
    [OK] wget found
    [OK] curl found

BENCHMARKING
----------------------------------------
  Downloading: https://sin-speed.hetzner.com/1GB.bin
  Exec Order:  Interlaced (3 tools x 5 runs)


  [ Iteration 1/5 ]
    Running surge (current)... 52.50s
    Running wget... 220.51s
    Running curl... 218.92s

  [ Iteration 2/5 ]
    Running wget... 227.77s
    Running surge (current)... 46.00s
    Running curl... 225.40s

  [ Iteration 3/5 ]
    Running curl... 218.46s
    Running wget... 209.23s
    Running surge (current)... 46.00s

  [ Iteration 4/5 ]
    Running curl... 209.42s
    Running wget... 219.11s
    Running surge (current)... 47.00s

  [ Iteration 5/5 ]
    Running curl... 225.23s
    Running surge (current)... 46.00s
    Running wget... 211.48s

======================================================================
  BENCHMARK RESULTS
======================================================================

  Tool                 │ Status   │ Avg Time   │ Avg Speed    │ Size
  ─────────────────────┼──────────┼────────────┼──────────────┼───────────
  surge (current)      │ OK       │ 46.34s     │ 22.10 MB/s   │ 1024.0 MB
    └─ Runs: 46.00s, 46.00s, 47.00s
  wget                 │ OK       │ 217.03s    │ 4.72 MB/s    │ 1024.0 MB
    └─ Runs: 211.48s, 219.11s, 220.51s
  curl                 │ OK       │ 220.87s    │ 4.64 MB/s    │ 1024.0 MB
    └─ Runs: 218.46s, 218.92s, 225.23s

======================================================================

======================================================================

  WINNER: surge (current) @ 22.10 MB/s


  SPEED COMPARISON
  --------------------------------------------------
  surge (current)      │ ██████████████████████████████████████████████████ 22.10 MB/s
  wget                 │ ██████████                                         4.72 MB/s
  curl                 │ ██████████                                         4.64 MB/s

Cleaning up temp directory...
  Done.

would test later with aria2c later.

relevant specs:

CPU: 12th Gen Intel(R) Core(TM) i7-1255U (4+8) @ 4.70 GHz
Memory: 5.16 GiB / 15.33 GiB (34%)

[SuperCoolPencil]

Yeah wget/curl are single threaded by default so that probably explains the gap here. Surge just opens way more connections to saturate the bandwidth.

Hyperfine is a good suggestion btw, definitely cleaner than a custom script. might add a standard config for it later so it's easier to run these.

Thanks for running this. let me know if you get around to testing aria2c, that’s the one Im actually curious about since it also does concurrency.

[bladeacer]

Here you go.

aria2c is actually really close for me somehow.

Surge Benchmark Suite
========================================

  Test URL:   https://sin-speed.hetzner.com/1GB.bin
  Iterations: 1
  Temp Dir: /tmp/surge_bench_lr6ewt9g

SETUP
----------------------------------------
  [OK] Go found
  Building surge...
    [OK] Surge built successfully
    [OK] aria2c found
    [OK] wget found
    [OK] curl found

BENCHMARKING
----------------------------------------
  Downloading: https://sin-speed.hetzner.com/1GB.bin
  Exec Order:  Interlaced (4 tools x 1 runs)


  [ Iteration 1/1 ]
    Running surge (current)... 52.51s
    Running wget... 226.51s
    Running aria2c... 56.32s
    Running curl... 221.54s

======================================================================
  BENCHMARK RESULTS
======================================================================

  Tool                 │ Status   │ Avg Time   │ Avg Speed    │ Size
  ─────────────────────┼──────────┼────────────┼──────────────┼───────────
  surge (current)      │ OK       │ 52.51s     │ 19.50 MB/s   │ 1024.0 MB
  aria2c               │ OK       │ 56.32s     │ 18.18 MB/s   │ 1024.0 MB
  wget                 │ OK       │ 226.51s    │ 4.52 MB/s    │ 1024.0 MB
  curl                 │ OK       │ 221.54s    │ 4.62 MB/s    │ 1024.0 MB

======================================================================

======================================================================

  WINNER: surge (current) @ 19.50 MB/s


  SPEED COMPARISON
  --------------------------------------------------
  surge (current)      │ ██████████████████████████████████████████████████ 19.50 MB/s
  aria2c               │ ██████████████████████████████████████████████     18.18 MB/s
  curl                 │ ███████████                                        4.62 MB/s
  wget                 │ ███████████                                        4.52 MB/s

Cleaning up temp directory...
  Done.

I suppose concurrency plays a big role.

Realised I ran benchmarks once instead of five times 🗿 , mb.

[bladeacer]

Considering aria2c is written in C++ and has been around for quite a while, I do not see how your project is going to beat it drastically in terms of performance metrics. That aside even having a progress bar for the CLI and the aesthetics of the TUI make it a joy to use.

aria2c uses max concurrent downloads of 5 and max connections per server of 1 (based on the manual page), not sure if this maps to your MaxConnectionsPerHost of 32 or MaxGlobalConnections of 100.

that aside CDN cache acts as a sort of cache warm up since we are querying the same URL frequently (when running benchmarks on the same URL multiple times).

[bladeacer]

here you go

Surge Benchmark Suite
========================================

  Test URL:   https://sin-speed.hetzner.com/1GB.bin
  Iterations: 5
  Temp Dir: /tmp/surge_bench_ok834jeo

SETUP
----------------------------------------
  [OK] Go found
  Building surge...
    [OK] Surge built successfully
    [OK] aria2c found
    [OK] wget found
    [OK] curl found

BENCHMARKING
----------------------------------------
  Downloading: https://sin-speed.hetzner.com/1GB.bin
  Exec Order:  Interlaced (4 tools x 5 runs)


  [ Iteration 1/5 ]
    Running wget... 225.59s
    Running surge (current)... 55.00s
    Running aria2c... 55.37s
    Running curl... 220.64s

  [ Iteration 2/5 ]
    Running wget... 217.51s
    Running surge (current)... 52.00s
    Running curl... 222.64s
    Running aria2c... 55.18s

  [ Iteration 3/5 ]
    Running wget... 228.06s
    Running aria2c... 58.25s
    Running curl... 223.67s
    Running surge (current)... 54.50s

  [ Iteration 4/5 ]
    Running aria2c... 59.65s
    Running curl... 215.19s
    Running wget... 223.40s
    Running surge (current)... 53.01s

  [ Iteration 5/5 ]
    Running curl... 228.72s
    Running aria2c... 73.22s
    Running surge (current)... 71.52s
    Running wget... 249.47s

======================================================================
  BENCHMARK RESULTS
======================================================================

  Tool                 │ Status   │ Avg Time   │ Avg Speed    │ Size
  ─────────────────────┼──────────┼────────────┼──────────────┼───────────
  surge (current)      │ OK       │ 54.17s     │ 18.90 MB/s   │ 1024.0 MB
    └─ Runs: 53.01s, 54.50s, 55.00s
  aria2c               │ OK       │ 57.76s     │ 17.73 MB/s   │ 1024.0 MB
    └─ Runs: 55.37s, 58.25s, 59.65s
  wget                 │ OK       │ 225.68s    │ 4.54 MB/s    │ 1024.0 MB
    └─ Runs: 223.40s, 225.59s, 228.06s
  curl                 │ OK       │ 222.32s    │ 4.61 MB/s    │ 1024.0 MB
    └─ Runs: 220.64s, 222.64s, 223.67s

======================================================================

======================================================================

  WINNER: surge (current) @ 18.90 MB/s


  SPEED COMPARISON
  --------------------------------------------------
  surge (current)      │ ██████████████████████████████████████████████████ 18.90 MB/s
  aria2c               │ ██████████████████████████████████████████████     17.73 MB/s
  curl                 │ ████████████                                       4.61 MB/s
  wget                 │ ████████████                                       4.54 MB/s

Cleaning up temp directory...
  Done.

[SuperCoolPencil]

Yeah, networks can be quite unpredictable!
But I am pretty sure surge outperforms aria2c (with 16 threads) by a decent margin.
In benchmark.py we use aria2c -x 16 -c 16 and when running on the CI we do get the ratio of 1.38x thats mentioned in the readme

[bladeacer]

Does the CI mainly use Windows? Hmm I checked that it uses Ubuntu.

Your README states the use of Windows 11 Pro as a test environment, so I am not sure for which the 1.38x is referring to.

You mentioned aria2c -x 16 -c 16, is aria2c using the equivalent counterparts of the concurrency and all you have enabled with Go?

Not sure if this is a OS specific difference in how your code and aria2c handles downloads, though it seems my network is getting slow.

[SuperCoolPencil]

Yeah, maybe running benchmark.py with -n 5 would be more appropriate.
All if there is one thing I've learned from this project it is that networks are really really unpredictable.
Could be anything from server CDN to someone else using your wi-fi at the time :D

[bladeacer]

not sure if this would be valid benchmarking of the sort but you could add some means of spawning a local http server so you get to configure the cache or other parts of it by using the binaries to download from said server.

this could complement rather than replace existing tests.

[SuperCoolPencil]

We actually tried testing with a local HTTP server, but ran into issues.

If you don't limit the speed, the download is instant. If you do limit it (basically by making the network sleep), it just comes down in bursts.

Neither really mimics real conditions, so we thought it was safer to stick to external servers. If you know a more robust way to handle this, though, definitely let us know.

[bladeacer]

if you have another device around you could try exposing it with reverse proxy and docker overhead

that being said I don't think curl, wget or aria2c do not really bother with benchmarking too much.

If a benchmark is reliant on variable factors out of control like network latency or DNS cache and to an extent system hardware it becomes difficult to reproduce.

[SuperCoolPencil]

completely agree