Feature request: resample

[cmtso]

Wrapping existing building blocks are already there with the periodicity/aggregation/infilling modules in a convenience function, as well as Polars resampling function.

https://docs.pola.rs/user-guide/transformations/time-series/resampling/

Use case (based on this paper to study different sampling frequencies):

Original water quality dataset is every 7 hours for one year.

Data is sub-sampled to daily, weekly, bi-weekly, monthly, and seasonal basis.

https://doi.org/10.1007%2Fs10661-015-4637-9

[richjam]

Hi @cmtso - thanks for the feature request!

It's interesting looking at that example paper - gives us more real-world use cases to consider.

---

First up, I think the aggregation module in time-stream is the equivalent of the downsampling that's happening here (hourly data to daily, weekly, bi-weekly, monthy, seasonal). Perhaps just confusion over terminology that we need to address?

Having said that, we probably need an additional aggregation method adding for this use case. In the paper it says

The common sampling period was defined based on the characteristics of typical monitoring programs: sampling generally occurs during regular business hours over the course of a Monday-through-Friday work week. Hence, to create the daily dataset, for example, the original high-frequency dataset was scanned for individual days, and for every day, a single sample was randomly selected with the constraint that it had to have been collected during the common sampling time period. Lastly, seasonal sampling strategies were mimicked by randomly selecting one sample in each of the major seasons (spring, summer, and fall), there again with the constraint that the sample had to had been collected during the common sampling time period.

So having a "random" pick aggregation method, with optional time constraints could work.

---

Secondly, it's interesting that their original data is 7-hourly resolution. This would be classified as an "non epoch-agnostic" resolution - basically because 7 hours doesn't divide into 24 nicely! Currently, we are not handling these types of resolutions - but we have ideas of how we might in the future, potentially allowing a custom epoch-start to be defined, so we know where the 7-hour steps should sit.

I don't suppose you have more detail, or even the dataset? It would be interesting if it's actually every 7 hours: let's say like:

Day 1: 07:00, 14:00, 21:00
Day 2: 04:00, 11:00, 18:00
Day 3: 01:00, 08:00, 15:00, 22:00 ... etc.

Or if they measure each day the same, with 7-hours gaps

Day 1: 07:00, 14:00, 21:00
Day 2: 07:00, 14:00, 21:00
Day 3: 07:00, 14:00, 21:00

I think in the latter case, we'd actually define this as "hourly" resolution, just happens to be measured with a 7 hour gap... if that makes sense?

[richjam]

Also - to think further about "upsampling" (going from coarse to finer resolution, e.g. Daily to 30 minutes), I think the mechanics in time-stream would also allow for that - but we could also wrap it up in a convience method.