POC: Signal progress via progressr

[DavisVaughan]

Closes #138
Closes #263

This implements the ideas in futureverse/progressr#85, all of the arguments there are still valid, and the global calling handler part of progressr makes this way easier to swallow as well.

• Remove TODO! in docs
• Rewrite progress vignette entirely
  • Show how to customize handler
  • Show how to set handler "globally"
  • Show how to "bring your own" progressr if you don't want to use .progress = TRUE (for advanced cases only)
• NEWS bullet
• Can we resolve the handle_cli() slowness? It is very bad! Do not force cli to refresh the display progressr#190

---

@HenrikBengtsson this is an experimental ideal that emits progressr::progressor() backed progress updates after each .f invocation when .progress = TRUE. We create a progressr::progressor() in the main R process which has as many steps as there are elements to iterate over, and that is then exported to all of the workers. The workers just call p() after each .f call.

I know we've had various conversations about where the user vs developer split is on progress bars. I feel like this is actually a pretty reasonable solution, but I'm very interested to hear your thoughts!

• The user is still in charge of displaying progress. They must opt in with with_progress() or (my preference) handlers(global = TRUE).
• The user can customize progress output with handlers(handler_cli()) and friends.
• Interactive users of furrr retain an easy way to add a progress bar to furrr functions.
• Developers building with furrr get an easy way to access the workflow of: 1) create a scoped progressr::progressor() 2) call it at some interval.
  • IMO developers should expose a progress = FALSE/TRUE flag of their own, which their users should opt into for progression updated. I feel this is particularly important because progression signaling is definitely not "free" and should not really be used with short running tasks.

I noticed that handler_cli() is extremely slow. I see futureverse/progressr#167 and would love to see if we can get to the root of that. The progress bar in purrr doesn't seem to have this issue and is backed by cli. Something weird must be happening here.

---

I imagine user scripts to look something like this

library(furrr)

progressr::handlers(global = TRUE)
progressr::handlers(progressr::handler_cli())

plan(multisession, workers = 2)

xs <- seq_len(100)

model_fn <- function(x) {
  Sys.sleep(.1)
  x
}

out <- future_map(xs, model_fn, .progress = TRUE)

[HenrikBengtsson]

Yes, this all makes sense to me. Providing a .progress = TRUE/FALSE will also cover parts what users expect coming from purrr.

Also, analogously to the new futurize::futurize(), I'm working on adding support for progressify(), e.g.

ys <- furrr::future_map(xs, fcn) |> progressify()
ys <- purrr::map(xs, fcn) |> progressify() |> futurize()

can simply transpile info:

ys <- furrr::future_map(xs, fcn, .progress = TRUE)

IMO developers should expose a progress = FALSE/TRUE flag of their own, which their users should opt into for progression updated. I feel this is particularly important because progression signaling is definitely not "free" and should not really be used with short running tasks.

Agree - this is tricky. I've been hoping something would emerge over time. If a progressify() transpiler could become a standard, the one could disable progressify() with a global settings - which would be very cheap. Anyway, more on that later.

[DavisVaughan]

@HenrikBengtsson I'm a bit worried about merging this without some kind of proactive advise to users about the change.

I'd love any thoughts on what you think a nice upgrade path would be.

Here's my main worry. This used to emit a progress bar without any additional code:

library(furrr)

plan(multisession, workers = 2)

xs <- seq_len(100)

model_fn <- function(x) {
  Sys.sleep(.1)
  x
}

out <- future_map(xs, model_fn, .progress = TRUE)

That will no longer emit a progress bar, and this change will be completely silent from a user's perspective.

Users will have to either:

• Add progressr::handlers(global = TRUE) at the top of their script, or their .Rprofile, etc, which is my preference for most users
• Wrap future_map() in progressr::with_progress()

i.e. they need to do this now

library(furrr)

# THIS IS NEW. It must be in your script somewhere or in an `.Rprofile` if you always want progressr updates
progressr::handlers(global = TRUE)

plan(multisession, workers = 2)

xs <- seq_len(100)

model_fn <- function(x) {
  Sys.sleep(.1)
  x
}

out <- future_map(xs, model_fn, .progress = TRUE)

But how do we communicate this change to them?

Here's a few ideas!

.progress = TRUE + progressr::handlers(global = NA)

If we want to keep .progress = TRUE as the way to turn on progression updates, then furrr could try and proactively detect when .progress = TRUE is set but no handlers are "active".

I could detect when progressr::handlers(global = NA) returns FALSE and throw a warning that if you set .progress = TRUE, then you need to set progressr::handlers(global = TRUE) somewhere to see progress.

But I don't think that would work with with_progress() right?

I don't think there is a general mechanism to detect that "showing progress is on", right? IIUC, that's probably purposeful, because you don't want developers to really know when its on or off, because you don't want them to make decisions based on this.

But that might mean this idea won't work.

A new argument

We could say that .progress is deprecated in furrr and no longer has any effect. We'd throw a deprecation warning about usage of .progress = TRUE and give a nice message about how to transition to the new argument (see below).

Instead, we'd add a new argument called .progression = TRUE, and that would do what .progress = TRUE does in this PR - it just signals progression updates. That name might even be slightly more clear? i.e. it hopefully signals to users that this is just about emitting progress updates, and that they still need to make them visible with progressr::handlers(global = TRUE) or with_progress()

This is different from purrr::map(.progress = TRUE), which is unfortunate, but they also work pretty differently since purrr doesn't require something like progressr::handlers(global = TRUE) to show progress, so having the same argument on the furrr side is also a bit of a lie.

[DavisVaughan]

Oooh here's another option that I do actually like quite a bit!

First let's note that furrr (and future.apply) live in an interesting space:

• They are used by end users, where .progress = TRUE is an explicit opt in to wanting a visual progress bar

• They are used by developers, who want a way to easily instrument furrr to emit progressr's progress conditions, but still want the end user to be in control of whether a progress bar is shown or not.

So my proposal is that furrr functions have two progress modes:

• .progress = TRUE:

  • Signals progressr's progression conditions after each .f invocation
  • Internally wraps in progressr::with_progress() to respect that an end user is explicitly opting in to progress bars here

• .progress = "signal":

  • Signals progressr's progression conditions after each .f invocation

This seems like it would work quite nicely!

For end users:

• Easy opt in to progress bars with .progress = TRUE
• Easy customization of progress bars with progressr::handlers()
• Now get live progress updates on things like cluster or mirai backends
• Good backwards compatibility with old furrr, where .progress = TRUE already emits a visual progress bar

For developers:

• Can expose .progress directly to end users (or even other developers)
  • Makes it very easy for developers to add progress bars to their map calls
• Can hardcode .progress = "signal" if you don't want to expose a .progress option, but you still want end users to be able to opt in to visual progress with handlers(global = TRUE) or with_progress()
• For more complex scenarios, create your own progressor() and tick it yourself inside the .f you provide to furrr, and leave .progress = FALSE

[HenrikBengtsson]

I agree the path forward with this change in behavior is not obvious. My suggestion would be:

1. punt on this to another release - keep the current behavior as-is
2. digest the different alternatives, and see if other ones pop up
3. wait for progressify() - by using it, I hope and expect that new patterns and ideas will pop-up and be proposed by others

I've got a working progressify(), but it'll be some time before I can get around to release it. ETA is uncertain due to other commitments.

A parallel task would be to work with R Core (Luke) to allow for registering global condition handlers on package load (futureverse/progressr#78). Somewhere, maybe in person, I think Luke mentioned that he might be open to it. A conservative approach could be to have namespace-specific global calling handlers so that packages couldn't step on each other and overwrite each others handlers. That would make it possible for progressr to make progressr::handlers(global = TRUE) the default, which would be a game changer and solve lots of these complicated discussions.

[DavisVaughan]

That would make it possible for progressr to make progressr::handlers(global = TRUE) the default, which would be a game changer and solve lots of these complicated discussions.

I'm a bit surprised to hear that you'd be open to that.

I thought you wanted users to have to explicitly opt in to visual progress notifications.

But this suggests that you recognize that progressr::handlers(global = TRUE) would be a nice default, as long as users have a way to change how progress is actually displayed (including explicitly opting out with an explicit handlers(global = FALSE)). Is that right?

---

If this was the default then .progress = TRUE could just do exactly what's in this PR (i.e. just signal progression conditions), and people would get a progress bar immediately without furrr needing to wrap in with_progress(). That'd be nice. And then people could opt out instead with progressr::without_progress() or handlers(global = FALSE), which feels like a nicer API to me.

[HenrikBengtsson]

That would make it possible for progressr to make progressr::handlers(global = TRUE) the default, which would be a game changer and solve lots of these complicated discussions.

I'm a bit surprised to hear that you'd be open to that.
...

Yes, I'm thinking opt-out instead of opt-in. It's also important that only progressr would be allow to register this (i.e. no other packages) to avoid competing interests - I guess that goes for all global calling handlers. We lack standards for this in R, so it's a bit "exploratory".

We already have R option progress.enable. I think it would be more natural that the user can just use that, instead of that + progressr::handlers(global = TRUE). Another way to look at it, if progressr::handlers(global = TRUE) could happen on package load, this discussion would move to whether progress.enable should default to TRUE or FALSE. (Right now it's TRUE in interactive sessions.)

I might disagree with myself on all this in the future ¯_(ツ)_/¯