Potential `io_uring` optimizations, experiments and improvements.

[kavirajk]

Silk uses io_uring as ground truth. However the current state of silk uses io_uring without much optimizations. I feel like there are some opportunities.

These optimizations are heavily inspired by the paper io_uring for High-Performance DBMSs: When and How to Use It and bit of playing with io_uring raw apis.

Would like to use this issue to track such optimizations, improvements, experiments and discussions related to it.

Following TODOs are just here to experiment with, no guarantee it will end up in silk. Also these TODOs can be added/removed/updated as I experiment with.

TODOs

• Fixed buffers instead of per-io buffer allocation.
• PMC counters support (at least basic counters like cycles, instructions and context-switches); This will be used to measure the impact of io_uring optimizations
• Register files. Avoid file descriptors sharing contention between different threads of same process.
• Register Ring file descriptors. Blocked by submissionLock.
• SINGLE_ISSUER, DEFER_TASKRUN, COOP_TASKRUN flags. Blocked by submissionLock
• Direct access to NVMe queues. Bypassing the entire kernel storage IO stack (OP_URING_CMD opcode)
• Visibility on kernel io_worker fallback (frequent fallback to io_worker may signal wrong IO pattern)
• IOSQE_CQE_SKIP_SUCCESS on internal SQEs (currently Silk submits SQEs whose successful CQE is intentionally discarded)
• Multishot accept. Silk currently does two scheduler round-trips per accepted connection (poll() + accept4). And can be replaced with single. One persistent multishot SQE delivers a CQE per connection with the accepted fd carried inline.

Blockers

In silk, thread-mode fibers and proxy fibers can call read/write/poll from a thread that is not the target CPU's scheduler thread. Silk has submissionLock to have synchronization between these different actors. This is one of the single biggest blocker to use io_uring's SINGLE_ISSUER optimization.

Which will enable us to use other optimizations like DEFER_TASKRUN which enables kernel to run task_run (an task that actually moves completed IO to CQE) only during io_uring_syscall_enter instead of interrupting the user-space randomly. Expectation is to have more predictable IO performance (less variance in p99 tail latency)

SINGLE_ISSUER also enables us to do io_uring_register_ring_fd optimization which reduces the per-syscall overhead of grabbing ring fd reference every single time io_uring_enter syscall is invoked.

One workaround I can think of is to use submissionInbox model (similar to how silk already uses cancelQueue and sleepQueue) where still multiple threads can submit the IO, but they submit to the inbox and one actor per ring (per CPU) actually submits the IO via io_uring_enter syscall without needing any synchronization. But I don't how this will impact overall latency and throughput. Need to be measured before adding this.

Methodology

1. Every optimizations have to be evaluated using PMC counters (direct impact)
2. Perf benchmarks (file-perf, net-perf, etc) - For end-to-end throughput and latency gains.
3. Clear documentation about the optimizations, whether it targets CPU bound or IO bound workloads.