bitreq: Can we make the crate async-runtime-agnostic?

[tcharding]

Currently the async feature enables tokio. Is it possible for this not to be the case by such things as using futures::io::AsyncRead instead of tokio::AsyncRead?

If we can do this then @apoelstra will be more open to patching in async support into jsonrpc.

[tnull]

Hmm, well, for one tokio is by now more or less the default executor for many projects and many other efforts like async-std have basically gone away. The bitreq crate is also tied to tokio due to the use of tokio-rustls and tokio-native-tls. Also note that there are a lot of subtleties to do with the tokio runtime, so we'd need to be very careful to not mess up here if we'd use future::io::AsyncRead (see for example https://www.reddit.com/r/rust/comments/1hyjrkq/comment/m6jrr37/ on the differences) and it would force user to use https://docs.rs/tokio-util/latest/tokio_util/compat/struct.Compat.html I guess.

I guess if it's a big showstopper we could consider refactoring it into a more abstracted bitreq-core crate that does #418 and then make use of that via more specific bitreq-tokio crate or similar.

However, to me that currently seems like a bunch of work for very little actual benefit, as I imagine currently 99% of dependent crates will end up using tokio anyways.

[apoelstra]

So, nobody can use async without a massive dep tree, heavyweight executor, and std? And even within this, nobody can use smol or custom executors?

I guess for a crate doing HTTP that's okay...but it's pretty sad. I don't understand why libraries need to be tied to tokio like this. What was the point of the Future trait and all this crazy compiler-assisted abstraction if in the end people just treat it like "Go but with harder to understand bugs"?

[tnull]

So, nobody can use async without a massive dep tree, heavyweight executor, and std? And even within this, nobody can use smol or custom executors?

Yes, I think this is mostly true in general, and in the rust-bitcoin 'ecosystem' in particular I'm not aware of any project that uses something else but tokio (if you don't count the WASM runtime, I guess). Given the pain we had because of tokios idiosyncrasies we considered exploring smol before, but unfortunately it doesn't seem super mature, all benchmarks I saw (e.g., https://gist.github.com/fundon/b16c5d9271d7521b7c91f4dbe4a4d8b5, https://zenoh.io/blog/2022-04-14-rust-async-eval/) indicate considerably worse performance compared to tokio, and switching away from all the tokio tooling (tokio::sync, tokio::time, etc) is non-trivial.

I guess for a crate doing HTTP that's okay...but it's pretty sad. I don't understand why libraries need to be tied to tokio like this. What was the point of the Future trait and all this crazy compiler-assisted abstraction if in the end people just treat it like "Go but with harder to understand bugs"?

In theory I agree, but given that the standard library is lacking a lot of async utilities, which often are also tied to the executor/runtime in subtle ways (for example, tokio::time only works because the runtime keeps some dedicated magic tasks around that drive the timers, if they are blocked or die, all your timers will stop firing, etc pp), tokio seems to be the defacto 'default' for many use cases.

[apoelstra]

Jeez, okay. This is pretty different from the initial "just have two versions of every function, where one has async and await added everywhere" scope of this async project.

I guess if that's what's needed for async then it's what we've gotta do.

[jharveyb]

Don't want to derail the thread too much, but would something like https://crates.io/crates/tor-rtcompat be a workable (starting) solution? It seems intended to solve a very similar problem, including some handling for timers and TLS. I read through it a bit and it seems pretty thin / approachable.

The only dependent project I see is Arti, so not sure how much actual use this has. There's https://crates.io/crates/async-compat for using Tokio-dependent dependencies from smol, but that doesn't address the dependency tree issue.

[apoelstra]

Hey Jonathan! I like the premise of that project but it looks like it brings in yet another huge pile of depenedencies.

I'm leaning toward "we just need to be tokio-exclusive", though in that case we need to make sure that our crates have a -tokio suffix so users are aware upfront that they're not generally usable in an async context.

Later we can write -smol versions and so on, assuming anybody is willing to do the async work again. I suspect that along the way we'd need to make our own tor-rtcompat which hopefully would be smaller.