perf: Hot path performance optimizations (P2, P5, P13, partial P1)

[dshkol]

Summary

Performance optimizations for frequently executed code paths, targeting the "hot paths" identified in the code audit.

Changes

P2: parse_metadata pre-split (parse_metadata)

• Pre-split meta3 and meta2 by dimension_id before the loop using split()
• O(1) hash table lookup instead of O(n) dplyr::filter() per column

P5: Factor conversion gsub loop (normalize_cansim_values)

• Replace for loop with across() for single-pass regex processing
• Use vapply for pre-checking which fields need processing

P13: lapply %>% unlist chain optimizations (multiple locations)

• Replace lapply(length) %>% unlist with vectorized lengths()
• Replace lapply(...) %>% unlist with purrr::map_chr() for string extraction
• Replace lapply(gsub...) %>% unlist with vectorized gsub() directly
• Replace lapply(class) %>% unlist with vapply(..., character(1))

P1 (partial): fold_in_metadata member ID extraction

• Replace lapply(.data$...pos, ...) %>% unlist with purrr::map_chr()
• Note: Full batch join restructuring deferred - complex refactor requiring careful design

Files Modified

• R/cansim.R: normalize_cansim_values, fold_in_metadata_for_columns, categories_for_level
• R/cansim_metadata.R: parse_metadata, read_notes, get_cansim_cube_metadata

---

Benchmark Results

P2: parse_metadata pre-split - ✅ 84-86% improvement

Test methodology: Simulated metadata structure matching real StatCan table metadata, testing the filter-per-iteration vs pre-split approach.

Test data:

• Small test: 10 dimensions × 500 members = 5,000 rows
• Large test: 20 dimensions × 2,000 members = 40,000 rows

Benchmark code:

library(microbenchmark)
library(dplyr)

n_dimensions <- 20
n_members_per_dim <- 2000
total_rows <- n_dimensions * n_members_per_dim

dimension_ids <- rep(seq_len(n_dimensions), each = n_members_per_dim)
meta3 <- data.frame(
  dimension_id = dimension_ids,
  member_id = seq_len(total_rows),
  stringsAsFactors = FALSE
)

# Original: filter inside loop
original <- function(meta3, n_dims) {
  for (col_idx in seq_len(n_dims)) {
    meta3 %>% filter(dimension_id == col_idx)
  }
}

# Optimized: pre-split once
optimized <- function(meta3, n_dims) {
  meta3_split <- split(meta3, meta3$dimension_id)
  for (col_idx in seq_len(n_dims)) {
    meta3_split[[as.character(col_idx)]]
  }
}

microbenchmark(
  original = original(meta3, n_dimensions),
  optimized = optimized(meta3, n_dimensions),
  times = 20
)

Results (40,000 rows, 20 dimensions):

Unit: microseconds
      expr      min       lq     mean   median       uq       max neval
  original 6326.177 6783.040 8172.669 7162.208 9262.863 13016.475    20
 optimized  977.809 1027.337 1317.330 1117.332 1278.339  3566.016    20

Improvement: 84.4%

Results (5,000 rows, 10 dimensions):

Unit: microseconds
      expr      min       lq      mean   median       uq      max neval
  original 1927.779 2002.235 2256.8327 2046.207 2268.366 5093.922    30
 optimized  245.016  271.051  376.5454  280.604  291.018 1788.830    30

Improvement: 86.3%

---

P5: normalize_cansim_values - ⚠️ 0.8% (negligible)

Test methodology: Benchmarked normalize_cansim_values() on real StatCan data comparing master vs optimized branch.

Test data: Table 17-10-0005 (Population by age and sex)

• 302,610 rows
• 24 columns
• 2 columns with classification codes to strip

Benchmark code:

library(cansim)
library(microbenchmark)

# Download test data once
data <- get_cansim("17-10-0005", refresh=TRUE)
saveRDS(data, "/tmp/cansim_benchmark_data.rds")

# Benchmark (run on master, then on optimized branch)
data <- readRDS("/tmp/cansim_benchmark_data.rds")
microbenchmark(
  normalize = normalize_cansim_values(data),
  times = 10
)

Results:

Master branch:
Unit: milliseconds
      expr      min       lq     mean  median       uq   max neval
 normalize 815.038 830.452 4412.136 842.119 846.833 35899    10

Optimized branch:
Unit: milliseconds
      expr      min       lq     mean   median       uq      max neval
 normalize 826.005 832.222 1652.057 835.336 849.808 8980.355    10

Master median:     842.119 ms
Optimized median:  835.336 ms
Improvement:       0.8%

Analysis: The improvement is within measurement noise. This is expected because the optimization only affects 2-3 classification columns per table - the across() change improves code clarity but doesn't measurably improve performance.

---

P13: lengths() optimization - ⚠️ -0.6% (negligible)

Test methodology: Benchmarked categories_for_level() which uses the lengths() vs lapply(length) %>% unlist optimization.

Test data: Same 302,610 row table from P5 test, with 3 hierarchy columns.

Benchmark code:

library(cansim)
library(microbenchmark)

data <- readRDS("/tmp/cansim_benchmark_data.rds")
microbenchmark(
  categories_level_1 = categories_for_level(data, "GEO", level=1),
  categories_level_2 = categories_for_level(data, "GEO", level=2),
  times = 50
)

Results:

Master branch:
Unit: milliseconds
               expr      min       lq     mean   median       uq      max neval
 categories_level_1 132.668 143.660 147.256 148.906 151.744 165.113    50

Optimized branch:
Unit: milliseconds
               expr      min       lq     mean   median       uq      max neval
 categories_level_1 130.193 146.447 147.957 149.750 150.930 158.385    50

Master median:     148.906 ms
Optimized median:  149.750 ms
Improvement:       -0.6%

Analysis: The lengths() optimization operates on unique hierarchy values only (15 unique values for GEO column), not on all 302,610 rows. With such a small input, there's no measurable difference between lengths() and lapply(length) %>% unlist.

---

Summary

Optimization	Test Data	Improvement	Status
P2 (pre-split)	40k rows, 20 dims	84.4%	✅ Significant
P5 (across gsub)	302k rows, 2 cols	0.8%	⚠️ Negligible
P13 (lengths)	15 unique values	-0.6%	⚠️ Negligible

Test Plan

• Run devtools::check() with no errors/warnings
• Benchmarks run on master vs optimized branch
• Verify normalize_cansim_values() produces identical output
• Test with various table sizes

🤖 Generated with Claude Code