Consistently higher latency, sporadically much higher latency, with fred vs deadpool-redis

[banool]

I'm experimenting with using Fred vs deadpool-redis. See the attached graph. I upgraded the image in prod to one using fred at about 17:09, and then reverted around 17:23.

You can see that with Fred, latency overall worsens; higher baseline latency, spikier spikes.

I'm building my pool like this:

/// Build a Redis pool. We call `init` here rather than letting the caller do it to make
/// the client safer to use, since it won't work if you don't call init.
///
/// We do not use `prefer_connected` on the advice of the creator of the crate:
/// https://github.com/aembke/fred.rs/discussions/254
///
/// Prefer not to use `.next` with the pool that comes from this, just use the pool
/// directly, e.g. `pool.ping().await` is better than `pool.next().ping().await`.
///
/// Be careful with using `.replicas()`. See the module comment for more info.
///
/// This also returns the JoinHandle for the task used by the client in the background.
/// You should not throw this out, instead wait on it along with all other tasks.
pub async fn build_pool(
    &self,
) -> Result<(ApiGatewayRedisPool, JoinHandle<Result<(), RedisError>>)> {
    let redis_url_str = self.redis_url.as_str();
    let config = if self.redis_cluster {
        RedisConfig::from_url_clustered(redis_url_str)
    } else {
        RedisConfig::from_url_centralized(redis_url_str)
    };
    let config = config.context("Failed to build Redis config")?;
    let pool = Builder::from_config(config)
        // The default ReconnectPolicy is to try to reconnect forever. We set out own
        // policy here to prevent that behavior.
        .set_policy(ReconnectPolicy::new_constant(4, 1500))
        // This configuration affects what happens for each connection / connection
        // attempt.
        .with_connection_config(|config| {
            config.connection_timeout = Duration::from_secs(5);
            config.unresponsive = UnresponsiveConfig {
                max_timeout: Some(Duration::from_secs(5)),
                ..Default::default()
            };
            config.replica = ReplicaConfig {
                lazy_connections: true,
                primary_fallback: true,
                connection_error_count: 1,
                ..Default::default()
            };
        })
        // This enables otel tracing.
        .with_config(|config| {
            config.tracing.enabled = true;
            config.tracing.default_tracing_level = tracing::Level::INFO;
        })
        .with_performance_config(|config| {
            config.auto_pipeline = true;
        })
        .build_pool(self.redis_pool_size)
        .context("Failed to create Redis connection pool")?;

    // These options apply to all requests made using a connection from the pool.
    // Setting with_options here is equivalent to doing it for each request.
    let options = Options {
        max_attempts: Some(2),
        timeout: Some(Duration::from_secs(6)),
        fail_fast: true,
        ..Default::default()
    };

    info!("Initializing and pinging Redis pool...");

    // Init the pool. You have to do this before you can use it. We use tokio::timeout
    // because `init` can hang (e.g. if Redis isn't there), and we don't want it to
    // just hang forever.
    let handle = tokio::time::timeout(Duration::from_secs(15), pool.init())
        .await
        .context("Failed to init Redis pool")??;

    // Ping to confirm that the pool is working.
    pool.ping().await.context("Failed to ping Redis")?;

    info!("Initialized and pinged Redis pool");

    Ok((pool.with_options(&options), handle))
}

In this case I'm not using Redis Cluster or replicas, this is just with a boring old normal Redis.

Is there something I can do with my config to get equivalent performance to what I was getting with deadpool-redis? I'd love to keep using fred due to the improved configurability, the cleaner API (e.g. one client instead of different clients for normal vs cluster Redis), active development, etc.

[banool]

I see this issue suggests some stuff: #145. I'll experiment with disabling backpressure, though I doubt that's the issue because the default is 10,000 before it kicks in and we shouldn't exceed more than 1,000 or so at a time.

[banool]

That didn't help unfortunately. I confirmed that going back to deadpool-redis improved things a lot. Perhaps fred works better when I have more workers for the Tokio runtime? I only have like 1-2 workers so there isn't much parallelism on offer (though idk if that's relevant, it doesn't affect concurrency, depends on what fred does in between await points I guess).

[aembke]

Interesting, yeah backpressure or max_feed_count (socket flushing) or nagles (TCP_NODELAY) would be my guess. Do you know whether the TCP_NODELAY settings are the same on both?

Otherwise I would use full-tracing and see what that shows. The extra spans are set up specifically to measure all the potentially long-running .await points (like channel communication, flushing sockets, etc), so I would hope that shows the culprit.

To your other question about concurrency - yeah the interface and actor-model based routing layer here is really designed to offer better performance at higher concurrency, so I would expect you see to comparatively better QPS/throughput vs deadpool the more tasks share a client (assuming auto_pipeline is true). However, in the worst case there shouldn't be any negative effects from using fewer tasks. I'm curious, have you measured the QPS/throughput for your app with the two libraries?

For what it's worth, or in case it helps with testing/debugging, there's a benchmarking tool (https://github.com/aembke/fred.rs/tree/main/bin/benchmark) that might help. It also shows how to set up tracing.

tokio-console might also be helpful. This is the line that spawns the task that acts as the root of the supervision tree for all the connections.

[aembke]

Also, can you show some of the code that uses the pool and makes requests to the server?