Task Queue is implemented in R for asynchronous tasks based on PostgreSQL database. This package is only suitable for parallel computing without any communication among parallel tasks (i.e. Embarrassingly parallel).
Several R packages have been using for parallel computing (e.g. High-Performance and Parallel Computing with R which are not suitable for asynchronous tasks.
- Uneven load among cores/workers. Some workers run faster than others and then wait for others to stop.
- Cannot utilise the new available workers after a parallel task is started.
taskqueue is designed to utilise all available computing resources until all tasks are finished through dynamic allocating tasks to workers.
Install the developing version from Github.
remotes::install_github('byzheng/taskqueue')Please refer to
- Getting Started vignette for more details.
- Simple workflow example article for a complete simplified workflow.