Async Runtime Deadlock Issue with rgb-lib and Tokio Integration

[dcorral]

Summary

We're integrating rgb-lib into a Lightning node application (RLN) that uses tokio as its async runtime. We've encountered a deadlock that occurs when running with a single tokio worker thread but works fine with multiple threads. After investigation, we believe the root cause lies in how rgb-lib handles async operations internally.

The Problem

Our test getchannelid_success hangs indefinitely when run with:

#[tokio::test(flavor = "multi_thread", worker_threads = 1)]

But passes when run with worker_threads = 3 or higher.

This is a classic symptom of a blocking operation starving the async runtime.

Root Cause Analysis

1. Async Runtime Mismatch

rgb-lib uses async-std (deprecated) as its sea-orm runtime:

# rgb-lib Cargo.toml
sea-orm = { version = "1.1.12", default-features = false, features = [
    "macros",
    "runtime-async-std-rustls",  # async-std runtime
    "sqlx-sqlite",
    "with-json",
] }

Our application uses tokio:

# Our Cargo.toml
sea-orm = { version = "1", features = ["sqlx-sqlite", "runtime-tokio-rustls", "macros"] }

When we call rgb-lib methods from our tokio runtime, rgb-lib's internal block_on() calls create a separate async-std executor that competes with tokio for the thread. Given async-std is deprecated, consider moving to runtime-tokio-rustls. I already tested rgb-lib with tokio runtime locally and all test pass. I only changed the Cargo.toml file to use tokio runtime. Please check if further investigation needs to be done to make the change.

2. Pervasive block_on() Usage

Looking at rgb-lib/src/database/mod.rs, every database operation wraps async sea-orm calls with futures::executor::block_on():

pub(crate) fn set_asset(&self, asset: DbAssetActMod) -> Result<i32, InternalError> {
    let res = block_on(Asset::insert(asset).exec(self.get_connection()))?;
    Ok(res.last_insert_id)
}

pub(crate) fn get_asset(&self, asset_id: String) -> Result<Option<DbAsset>, InternalError> {
    Ok(block_on(
        Asset::find()
            .filter(asset::Column::Id.eq(asset_id))
            .one(self.get_connection()),
    )?)
}

There are 48+ instances of this pattern across all database operations.

3. Single Connection Pool

In rgb-lib/src/wallet/offline.rs, the connection pool is configured with only 1 connection:

let mut opt = ConnectOptions::new(connection_string);
opt.max_connections(1)      // Only 1 connection allowed
    .min_connections(0)
    .connect_timeout(Duration::from_secs(8))
    .idle_timeout(Duration::from_secs(8))
    .max_lifetime(Duration::from_secs(8));
let db_cnn = block_on(Database::connect(opt));

How the Deadlock Happens

With a single tokio worker thread:

[Tokio Worker Thread]
    |
    +-- HTTP request arrives (e.g., /openchannel)
    |
    +-- We call spawn_blocking(|| rgb_wallet.some_method())
    |       |
    |       +-- rgb-lib method calls block_on(sea_orm_query)
    |           |
    |           +-- futures::executor blocks the thread
    |           |   waiting for async-std future
    |           |
    |           +-- SQLite connection pool has only 1 slot
    |
    +-- Meanwhile, another task also needs the database
    |       |
    |       +-- Queued, waiting for the single connection
    |
    +-- DEADLOCK: The single thread is blocked, connection is held,
                  nothing can make progress

With 3+ worker threads, other threads can continue processing while one is blocked, avoiding the deadlock.

Workaround We're Using

We wrap all rgb-lib calls in tokio::task::spawn_blocking():

let result = tokio::task::spawn_blocking(move || {
    rgb_wallet.some_method()
}).await?;

This moves the blocking work to tokio's blocking thread pool, but it's not a complete solution because:

• It adds latency and complexity
• The single connection pool still creates bottlenecks
• Nested blocking can still cause issues. For instance the getchannelid_success test, does not pass with 1 worker_thread, needs minimum of 3 (Still trying to find where the exact deadlock is happening)

Proposed Solution

Expose an Async API and move to tokio runtime.

Add async versions of wallet methods that callers can await directly:

// Keep existing sync API for backwards compatibility
pub fn get_asset(&self, asset_id: String) -> Result<Option<DbAsset>, InternalError> {
    block_on(self.get_asset_async(asset_id))
}

// New async API
pub async fn get_asset_async(&self, asset_id: String) -> Result<Option<DbAsset>, InternalError> {
    Ok(Asset::find()
        .filter(asset::Column::Id.eq(asset_id))
        .one(self.get_connection())
        .await?)
}

This would let tokio applications await rgb-lib operations natively without runtime conflicts.

(Optional): Increase Connection Pool Size

At minimum, increasing the pool size would reduce contention:

opt.max_connections(5)  // Instead of 1
    .min_connections(1)  // Keep at least 1 connection warm

Questions for Discussion

1. Is there a reason for the single-connection pool limit? Would increasing it cause issues with SQLite?
2. Would the rgb-lib team be open to exposing async APIs? We'd be happy to contribute a PR.

Environment Details

• rgb-lib version: 0.3.0-beta.4
• sea-orm version: 1.1.19 with tokio runtime
• Database: SQLite

Happy to provide more details or test any proposed fixes. Thanks for your time!

[zoedberg]

Our test getchannelid_success hangs indefinitely when run with:

To be clear: this error is happening on a modified fork of RLN, not sure an issue here makes sense without first analyzing your changes, which I assume are not ready for review since there's no open PR.

Anyway, it seems the test needs 3 workers to pass there because in that codebase you're using tokio::task::spawn_blocking() to call rgb-lib methods, in that case the behavior is normal, that's not "the problem", it's a result of your changes.

Async Runtime Mismatch

I agree to drop async-std since deprecated and using tokio in order to use the same runtime RLN uses. The related PR you opened #67 will be merged ASAP

Pervasive block_on() Usage

These are necessaries since sea-orm is an async ORM while rgb-lib is not async

Single Connection Pool

Since rgb-lib methods are blocking it doesn't make sense to allow more than 1 connection

Workaround We're Using

I'm confused, you are saying that tokio::task::spawn_blocking() is your workaround, but previosly where you describe "How the Deadlock Happens" you say We call spawn_blocking(|| rgb_wallet.some_method()). It's not clear what the actual problem is here, please try to be more precise.

Proposed Solution

It doesn't make sense to expose async APIs in rgb-lib (and as a consequence to increase the connection limit), since both the library and the RGB libraries that it uses don't support async usage.

Is there a reason for the single-connection pool limit? Would increasing it cause issues with SQLite?

Yes, as explained above.

Would the rgb-lib team be open to exposing async APIs? We'd be happy to contribute a PR.

No, not until we implement async on the underlying RGB libraries. Also please note that enabling async methods is not just about adding async function signatures, it's about adding DB transactions, using locks to prevent dangerous concurrent accesses and so on.

[dcorral]

After revisiting the code and working on a new approach here my original analysis was off.

Test needs 3 workers:

Our tests now pass with worker_threads = 1. The issue I described was from an earlier broken state of our code, not rgb-lib.

Async Runtime Mismatch:

Thanks for looking into merging PR #67, I'm working on the comments there.

block_on() Usage:

Makes sense to me now since sea-orm is async and rgb-lib is not.

Single Connection Pool:

With blocking methods, more connections wouldn't help anyway.

spawn_blocking confusion

Fair point. I contradicted myself there. Using spawn_blocking() for rgb-lib calls is just the correct pattern. I need to step up on my async rust understanding!

On async APIs

I understand it's not just about adding async to function signatures and it requires transactions, locks, and async support in the underlying RGB libraries.

---

The original issue was based on a misunderstanding. After fixing our integration, everything works as expected. I'll close this issue now with this comment, feel free to reopen if further discussion is needed.

Thanks again for the thorough response @zoedberg 🙏