We hit this while grounding an East Africa forecast-verification study on sheerwater. It reproduces at 414344a with only a local filesystem cache. A separate one-line bug found at the same time (the 'fileysstem' typo) is fixed in #40.
A @timeseries function can return a cache that ends before the requested end_time
What happens. Take a function decorated @timeseries() @cache(cache_args=[...]) whose cache key leaves out start_time/end_time
(the usual pattern, used throughout sheerwater). If its cache ends at 2024-12-31 and you ask for data to 2025-06-30, or even to
2026-12-31, you get data only up to 2024-12-31. There is no warning and no recompute.
validate_data defaults to False (src/nuthatch/processor.py:31@414344a), so on the default path coverage is never checked.
- With
validate_data=True, timeseries.validate (src/nuthatch/processors/timeseries.py:129-134@414344a) accepts any cache whose
end lies within 365 days of the requested end.
- When validation does fail, it prints "Triggering recompute … set
validate_data=False" (timeseries.py:136-139). The wrapper then
raises ValueError("Data failed validation. Please manually overwrite data to proceed.") (processor.py:73-74) and does not
recompute. The recompute branch is commented out at processor.py:76-92.
Minimal repro (nuthatch.toml: [nuthatch] filesystem = "/tmp/nh/root_cache"):
import numpy as np, pandas as pd, xarray as xr
from nuthatch import cache
from nuthatch.processors import timeseries
@timeseries()
@cache(cache_args=['name'])
def daily(start_time, end_time, name='rain'):
t = pd.date_range(start_time, end_time, freq='1D')
return xr.Dataset({'precip': ('time', np.arange(len(t), dtype='f4'))}, coords={'time': t})
daily('2020-01-01', '2024-12-31', recompute=True, cache_mode='overwrite')
print(daily('2020-01-01', '2026-12-31').time.max().values) # 2024-12-31, no warning
print(daily('2020-01-01', '2025-06-30', validate_data=True).time.max().values) # 2024-12-31, accepted
daily('2020-01-01', '2026-12-31', validate_data=True) # prints "Triggering recompute", then raises ValueError
Expected: if a cached result doesn't cover the requested window, nuthatch either recomputes it or says so loudly.
Actual: truncated data comes back silently. On the opt-in path the message promises a recompute that doesn't happen.
Suggested fix (small). Any of these would help:
(a) log a warning in post_process when the sliced result ends noticeably before end_time, or starts after start_time;
(b) make the validate message match the behaviour, e.g. "…pass recompute=True, cache_mode='overwrite' to refresh";
(c) add an option for a strict-coverage check, perhaps with a configurable tolerance in place of the fixed 365 days.
Is the 1-year tolerance and the off-by-default check intentional, for example to allow for upstream data latency? If so,
a note in the README's timeseries section would stop users from reading a truncated result as complete.
🤖 Drafted with Claude Code
We hit this while grounding an East Africa forecast-verification study on sheerwater. It reproduces at
414344awith only a local filesystem cache. A separate one-line bug found at the same time (the'fileysstem'typo) is fixed in #40.A
@timeseriesfunction can return a cache that ends before the requestedend_timeWhat happens. Take a function decorated
@timeseries() @cache(cache_args=[...])whose cache key leaves outstart_time/end_time(the usual pattern, used throughout sheerwater). If its cache ends at 2024-12-31 and you ask for data to 2025-06-30, or even to
2026-12-31, you get data only up to 2024-12-31. There is no warning and no recompute.
validate_datadefaults toFalse(src/nuthatch/processor.py:31@414344a), so on the default path coverage is never checked.validate_data=True,timeseries.validate(src/nuthatch/processors/timeseries.py:129-134@414344a) accepts any cache whoseend lies within 365 days of the requested end.
validate_data=False" (timeseries.py:136-139). The wrapper thenraises
ValueError("Data failed validation. Please manually overwrite data to proceed.")(processor.py:73-74) and does notrecompute. The recompute branch is commented out at
processor.py:76-92.Minimal repro (
nuthatch.toml:[nuthatch] filesystem = "/tmp/nh/root_cache"):Expected: if a cached result doesn't cover the requested window, nuthatch either recomputes it or says so loudly.
Actual: truncated data comes back silently. On the opt-in path the message promises a recompute that doesn't happen.
Suggested fix (small). Any of these would help:
(a) log a warning in
post_processwhen the sliced result ends noticeably beforeend_time, or starts afterstart_time;(b) make the validate message match the behaviour, e.g. "…pass
recompute=True, cache_mode='overwrite'to refresh";(c) add an option for a strict-coverage check, perhaps with a configurable tolerance in place of the fixed 365 days.
Is the 1-year tolerance and the off-by-default check intentional, for example to allow for upstream data latency? If so,
a note in the README's timeseries section would stop users from reading a truncated result as complete.
🤖 Drafted with Claude Code