Async file & http IO without {future}

[mmuurr]

I'm curious if anyone knows anything about any work being done on async versions of packages like {readr} and {httr2}. In particular with {readr} making use of libuv's existing async architecture and performing the IO on background threads (via the underlying libuv C/C++ calls).

Yes, I'm aware that {future} can be used as a mechanism to handle parallelism , but in some cases this should be achievable without forking (or any equivalent use of an additional R process) ... {httpuv} wraps existing async C/C++ libuv calls and avoids {future}, for example. For file IO and some libcurl calls, this should be achievable with {later} (to handle R callbacks when C/C++ operations complete) and then ideally abstracted to use {promises} for programmatic ease.

{readr} and {httr}/{httr2} seem like the lowest-hanging fruit in the tidyverse for this, and was simply wondering if there's active work being done there. And if not, if there's an appetite for it, or if {future} works well-enough for most people.

[nirgrahamuk]

In version 2 of readr the vroom engine was adopted supporting both lazy-loading and multithreaded reads. Are you exploring an idea to improve on this status quo?

[mmuurr]

@nirgrahamuk, yes, because even lazy reads still block on the main loop in R when the time comes to finally act on that data. I'm specifically looking for solutions where the main thread remains available (until, say, a callback is executed upon completion of the IO, which at that point would run on the main R thread).