Data functions

[solvicode]

I don't think we can avoid this if Orca is going to become truly useful.

The purpose of a data function is to accept a window and in return produce some data of a known schema.

This opens up lots of possibilities like:

• Plug and play toolboxes
• Clear separation of responsibilities (algorithms run the algos, windows trigger them and data functions provide the data)
• Retrospective analysis - you can look at a data point and rerun it to analyse why that data point was produced. This can be done ad hoc with data functions

The function is idempotent. If it gets the same window, it will produce the same data. It is down to the developer to ensure this, but we could version the interface and config (as proposed in #95).

The data function is run in preparation to an algorithm being run, and it puts the data in a temporary store. It then passes a key to orca core. The core then passes this key to the algorithm for the algorithm to get the data and run on. It is the algorithms responsibility to get the data from the tmp store.

This introduces a lot of new complexity. This is not an easy feature.

It would be compatible with ELT patterns. E.g. you create a big window - the data function gets all the data for that window. This gets written to an in memory store. Child windows that are then triggered by the parent algorithm are now just accessing their segment of data from the in memory store. The data would need to be carefully partitioned ahead of time, so each child algorithm get's just it's data. This is not easy, and would require some careful thought.

[solvicode]

Might not be impossible.

Redis allows filtering by time: https://redis.io/docs/latest/develop/data-types/timeseries/

The data can then go off to redis. E.g., in Go!:

package main

import (
    "context"
    "fmt"
    "time"

    "github.com/redis/go-redis/v9"
)

func main() {
    ctx := context.Background()
    r := redis.NewClient(&redis.Options{
        Addr: "localhost:6379",
    })

    // Create the 3 series
    r.TSCreate(ctx, "sensor:temperature", &redis.TSOptions{DuplicatePolicy: "last"})
    r.TSCreate(ctx, "sensor:humidity",    &redis.TSOptions{DuplicatePolicy: "last"})
    r.TSCreate(ctx, "sensor:pressure",    &redis.TSOptions{DuplicatePolicy: "last"})

    // Write a single row
    ts := time.Now().UnixMilli()
    r.TSAdd(ctx, "sensor:temperature", ts, 22.5)
    r.TSAdd(ctx, "sensor:humidity",    ts, 60.1)
    r.TSAdd(ctx, "sensor:pressure",    ts, 1013.2)

    // Bulk insert with pipeline
    pipe := r.Pipeline()
    for i := 0; i < 10000; i++ {
        ts := time.Now().UnixMilli() + int64(i)
        pipe.TSAdd(ctx, "sensor:temperature", ts, 22.5+float64(i)*0.01)
        pipe.TSAdd(ctx, "sensor:humidity",    ts, 60.0+float64(i)*0.005)
        pipe.TSAdd(ctx, "sensor:pressure",    ts, 1013.0+float64(i)*0.001)
    }
    _, err := pipe.Exec(ctx)
    if err != nil {
        fmt.Println("Error:", err)
    }
}

Then the child algorithm receives it's child window, along with a key for the redis data. It queries for just it's range - this can be automated by the SDK, it processes and then goes on it's way.

The caveat is that the child function cannot retrospect the data if needed - it inherited it's data from the parent.

[solvicode]

The issue I will eventually bump into is the response schema validation.

i.e. how you offer a good developer experience when reading the data. current methods for metadata (for example) mean you need to validate in the algorithm - not good. I'd suggest we fix that first so we understand the pattern