Faster pagination with mclapply

[Melkiades]

e.g.

iris3 <- cbind(iris2, gp = rep(c(1,2,3,4,5,6),1000))
rlist3 <- as_listing(iris3, key_cols = "Species", disp_cols = c("Sepal.Length", "Sepal.Width", "Petal.Width", "Petal.Length"))

start.time <- Sys.time() 
rlist3  %>% split(rlist3$gp) %>% lapply(., paginate_to_mpfs) -> kkk
end.time <- Sys.time()

time.taken <- end.time - start.time
time.taken

> Time difference of 36.06119 secs

[Melkiades]

mclapply does not seem an optimal solution for pagination as there is no large lapply to act upon but many different subroutines. The only two possible applications account for 30% of the time.

[gmbecker]

paginate_to_mpfs(matrix_form(rlist3))

is literally twice as fast for me as

paginate_to_mpfs(rlist3)

That is because the first version calls .compress_mat 2 times on each of the matrices (and is a NOOP with your example code, btw)), while the original version calls it 4 times for each matrix.

Furthermore listings shouldn't need all the newline handling right? at the very least it should be called once and then not invoked again when operating on (a subset of) the same strings.

There should be a way to cheaply-ish check whether you actually need the compress mat step at all, right?

[shajoezhu]

I am not 100% sure this is the place to improve our efficiency. for the long list, I think it is more of an operation optimization on the user end. The parallel process will be more efficient downstream.

[Melkiades]

paginate_to_mpfs(matrix_form(rlist3))

is literally twice as fast for me as

paginate_to_mpfs(rlist3)

Yes that was due to 2 unnecessary internal calls to matrix_form which accounted for a bit less than half of the time. handle_newlines cant be avoided due to wrapping. Now it is dou

There should be a way to cheaply-ish check whether you actually need the compress mat step at all, right?

This a very good point. Adding the check!