[PDR][Lake] Update predictPredictions and predictSlots to feature lastEventTimestamp

[idiom-bytes]

Motivation

As part of Incremental Lake + Analytics we want to make it easier to fetch/build our data lake incrementally.

To do so, for [1:n_event] entity tables like predictionPredictions and predictionSlots, we introduce a new parameter lastEventTimestamp .

lastEventTimestamp

I had outlined it as Solution B (and we're still working towards having all data saved on local_lake, but I think this will still reduce a lot of steps/work).

Where the last event to be processed, updates the param like such lastEventTimestamp = max(lastEventTimestamp, newEventTimestamp)

This param can then be filtered via lastEventTimestamp_lte or lastEventTimestamp_gte and enables us to approach all of this with less work and more ease.

DoD:

• predictPredictions table features lastEventTimestamp and is updated as described above
• predictSlots table features lastEventTimestamp and is updated as described above

[idiom-bytes]

Here is an example where the predcition.timestamp < payout.timestamp

Adding where clauses to each entity timestamp will work like an && clause across multiple filters (3x filtering/scan) where we really want an || clause across all timestamps, done efficiently (1x filter)

So rather than

• where timestamp_gte &&
• where trueval.timestamp_gte &&
• where payout.timestamp_gte)

We have 1 single param lastUpdateTimestamp to filter on, which already takes the max of whatever timestamp happened last.

[idiom-bytes]

One step towards improving performance is using predictPayouts as part of the predictPredictions query. This enables us to fetch all recent predictions and payouts.

Truevals however, must be fetched in a separate query.

query{
  predictPredictions(where:{ or: [{timestamp_gt:100},{payout_:{timestamp_gt:100}}]}){
    slot{
      id
    }
    user {
      id
    }
    stake
    payout{
      
      timestamp
      id
    }
  }
}