Why are cold starts so bad?

[nleroy917]

We should identify why we cannot replicate the fast cold-start times of the original benchmark.

• are we missing something from core?
• is there something sub-optimal about the current implementation?

[nleroy917]

Overview

There are two "puffin implementations": 1) The original by brian-ogrady which lived inside a fork of qdrant, and 2) my neptune implementation here in this repo. After benchmarking neptune, we realized there were some discrepancies in cold-start latencies with speculation that the neptune implementation is missing some key optimizations within qdrant.

I spent some time digging into it to understand why this might be happening. There are four main takeaways:

The original implementation had custom optimizations within qdrant, but they were not related to S3 transport

The original qdrant-fork implementation did have custom code in qdrant, however it had nothing to do with S3 data fetching. It was a new method: GraphLinks::load_from_ranged_mmap. This simply optimizes the graph loading which was only shaving ~10ms from total search latency.

Critically, both implementations utilized the same object_store crate for S3 transport. The original implementation had no optimizations here, and in fact neptune is more optimized for this.

Neptune has more performant S3 transport

Our implementation is actually faster now that we are using concurrent fetches hitting ~200MiB/s speeds. The original implementation and the first version of neptune utilized a single-stream architecture instead of fanning out to ~16 concurrent fetches.

The previous benchmark measured latency with fresh, pre-warm TLS pools

The reason the previous implementation looked better is because it was measuring S3 latency on an in-process, already established, warm TLS pool. Whereas neptune started measuring latency before a pool was even created.

Its actually even more egregious. The original implementation uploaded the puffin file immediately before measuring latency, thereby giving it three key advantages over neptune: 1) an in-process TLS/TCP pool warmed by the upload that happened moments earlier, 2) an S3 frontend that was hot for an object just written and repeatedly re-fetched, and 3) it reported the median-of-5-in-one-process for latencies.

Both implementations are nearly identical

I investigated both and both implementations are nearly identical (aside from #1 above). I re-ran the benchmark on an in-region EC2 and am are getting ~212 ms cold-start blob fetches using the new concurrent implementation and measuring with warm TLS pools. So, we can achieve that if we are smart about keeping TLS pools warm in a serverless/serverfull environment.

TL;DR we hit 212 ms cold-start time-to-first-result with no warm state at all beating the old implementation's 684 ms even though that was measured on pre-warmed connections.