Introspect Orca results

[solvicode]

After working with Orca, it becomes very natural to want to retrospect the internal results produced by Orca algorithms, within Orca.

Now the question is how you do it:

1. Build bindings in each SDK to the selected data store (currently just psql) and query direct from processor
2. Use core as the interface to query the results - all the SDKs need to provide is the interface to state what algorithm they want the results from and over what time frame.

1 might be more efficient as it avoids the latency of having to go through core (minimal), but does not scale well if many algorithms need the same kind of data from the database.

2 scales database load better as we can bake in to core a cache mechanism (this was the intention with redis), but it pushes more traffic through the network potentially hitting cloud cost. Not a major issue in my opionion as this is generally low if arranged properly.

[solvicode]

I think I have an ergonomic implementation in the python SDK:

@proc.algorithm(
    "SecondAlgo",
    "2.0.0",
    Every30Second,
    depends_on=[Lookback(my_algorithm, td=dt.timedelta, n=10)],
)
def second_algorithm(params: ExecutionParams) -> ValueResult: ...

We define the Lookback keyword to denote that this algorithm depends on this other algorithm, and the past n results.

Lookback is idempotent - accepts and returns an algorithm definition, but with fields hidden characterising the lookback.

Two arguments are available, td for "time delta" and n for count. td denotes the time to look back to, and n denotes the past n results. One can be provided but not both (the above example should fail).

Then when registering we add in this new field into the protobuf schema:

// AlgorithmDependency defines a requirement that one algorithm has on another's results.
// These dependencies form the edges in the processing DAG.
message AlgorithmDependency {
  // Name of the required algorithm
  // Must reference an existing algorithm name in the system
  string name = 1 [(buf.validate.field).required = true];

  // Version of the required algorithm
  // Must follow semantic versioning (e.g., "1.0.0")
  string version = 2 [(buf.validate.field).required = true];

  // Name of the processor that the algorithm is associated with
  string processor_name = 3 [(buf.validate.field).required = true];

  // Runtime of the processor that the algorithm is associated with
  string processor_runtime = 4 [(buf.validate.field).required = true];

  // A lookback field that specifies whether this dependency 
  // depends on past results of this algorithm
  oneof lookback {
    option (buf.validate.oneof).required = false;

    // Number of past results to depend on (if at all)
    uint32 lookback_num = 5;

    // Timeframe of past results to depend on (in milliseconds)
    uint64 lookback_time_delta = 6;
  }
}

[solvicode]

If a dependency changes, we need to flag this also when registering with a message saying the version needs to be bumped.

[solvicode]

SDK implementation and protobufs changed. Now modifying the core handling.

[solvicode]

The challange is determining at which point in the process we retreive the past results.