Skip to content

Fix data freq y new#68

Open
semvijverberg wants to merge 9 commits into
AI4S2S:mainfrom
Beyond-Weather-Git:fix_data_freq_y_new
Open

Fix data freq y new#68
semvijverberg wants to merge 9 commits into
AI4S2S:mainfrom
Beyond-Weather-Git:fix_data_freq_y_new

Conversation

@semvijverberg
Copy link
Copy Markdown
Member

@semvijverberg semvijverberg commented Jan 31, 2024

First of all, sorry for the confusing name of the PR. For us, this issue arises when we were dealing with yearly data (1 datapoint per year). Hence the initial name.

A better suited name would be: issues due to interval definition.

The following issue arise because the interval is not closed on both sides (but currently the intervals is closed on the left).

Example1 :
You have preprocessed target data with a single datapoint per year (say, the yield per year). You have assigned the timestamp of the first of October. The calender should check, [2015-10-01, 2015-10-02], but the last datapoint in 2015 is 2015-10-01, it cannot find the any closing date that is >= 2015-10-02 and thus it will discard the year 2015.

This exact problem also occurs when you have data up to 2015 and ask for an interval up to and including the 31st of December.

Example 2:
The target interval that include dates up to 31st of December, say the start point is the 1st of December.
Thus, intervals are defined as [2000-12-01 , 2001-01-01), [2001-12-01 , 2002-01-01), ...]. The last interval will be [2015-12-01 , 2016-01-01). When indexing this on our input array we will encounter missing data as we try to index 2016-01-01 but our data only runs until 2015-12-31. Hence, lilio discards the year 2015 as it cannot fit the data to the interval. The obvious solution, then, is to use open indices on both sides: thus our intervals become [2000-12-01, 2000-12-31].

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants