CRUDE: Flat Navigation Dispatch

[thorwhalen]

The design pattern I'm proposing is that when you need to dispatch a function like

train(learner, data)

where learner and data are complex python objects, you do it by creating three components:

• CRUD for learners
• CRUD for data
• train component, which points to the learner and data CRUD components to get its inputs

The CRUD components are made of two parts

• For the creation of the object: A specific UI component
• For all other CRUD operations: A (py2)store that can handle the type of objects being made

Sometimes the creation will itself require some complex object inputs. For example, the case of model in:
run(model, data)
But a model is just the output of train(learner, data), so we fall back on the train component,
which in turn relies on the learner and data components.

The GUI navigation that the use of this pattern enables is "flat" in the sense that the process structure is less of a tree or DAG, and more of a "choose your action, and if the inputs you require don't exist, go choose the action that will make these inputs available".

This flatness may or may not be desirable from a UX point of view, but is certainly desirable from an architectural simplicity point of view, and we can get a great bang for the buck as far as component reuse.

Any code that appears in this form:

data = get_data(data_key)
prepped_data_1 = prep1(data, another_param)
prepped_data_2 = prep2(data, yet_another, and_another)
result = do_something(prepped_data_1, prepped_data_2, configs)
...

can be dispatched to such a "flat GUI" dispatching get_data, prep1, prep2 and do_something functions, after wrapping these in a "dispatchable" form that uses stores to both solve the complex data entry problem and the action/navigation order problem. The store keys play the role of intermediate variables in the python code above.

Code example

A function such as:

from typing import Any

FVs = Any
FittedModel = Any

# to dispatch:
def apply_model(fvs: FVs, fitted_model: FittedModel, method='transform'):
    method_func = getattr(fitted_model, method)
    return method_func(list(fvs))

would have a dispatchable form (automatically created by wrappers) as such:

# This is what we want our "dispatchable" wrapper to look like

# There should be real physical stores for those types (FVs, FittedModel) that need them
fvs_store = dict(
    train_fvs_1=[[1], [2], [3], [5], [4], [2], [1], [4], [3]],
    train_fvs_2=[[1], [10], [5], [3], [4]],
    test_fvs=[[1], [5], [3], [10], [-5]],
)

fitted_model_store = dict(
    fitted_model_1=MinMaxScaler().fit(fvs_store['train_fvs_1']),
    fitted_model_2=MinMaxScaler().fit(fvs_store['train_fvs_2']),
)

# And a store for the result of the function if we want this result to be reused later
# (for example for further inputs)
apply_model_store = dict()


from typing import Any, Mapping, Tuple

FVsKey = str  # really, should be a str from a list of options, given by list(fvs_store)
FittedModelKey = str  # really, should be a str from a list of options, given by list(fitted_model_store)
Result = Any

# dispatchable function:
def apply_model_using_stores(
    fvs: FVsKey,
    fitted_model: FittedModelKey,
    method: str = 'transform',
    fvs_store: Mapping[FVsKey, FVs] = fvs_store,
    fitted_model_store: Mapping[FittedModelKey, FittedModel] = fitted_model_store,
    apply_model_store: Mapping[Tuple[FVsKey, FittedModelKey], Result] = apply_model_store
):
    # get the inputs
    fvs_value = fvs_store[fvs]
    fitted_model_value = fitted_model_store[fitted_model]
    # compute the function
    result = apply_model(fvs_value, fitted_model_value)
    # store the outputs
    apply_model_store[fvs, fitted_model] = result
    return result  # or not

[thorwhalen]

Substantially implemented: `front/crude.py` (1056 lines) now has a mature CRUDE machinery matching this issue's proposal closely -- `prepare_for_crude_dispatch`/`Crudifier` (class, `front/crude.py:800`) to wrap a dispatch function so its complex-object arguments are sourced from mall-backed stores by key, `crudify_based_on_names` for name-convention-based wiring, `_mk_store_for_param`/`simple_mall_dispatch_core_func` for the per-param CRUD-store plumbing, and `store_on_output` to persist results. This looks like it covers the "train(learner, data) dispatches via CRUD-for-learners + CRUD-for-data + a pointing component" pattern described here. I didn't do a line-by-line spec comparison against the original proposal (out of scope for this pass), so leaving open in case there's a specific remaining gap Thor has in mind -- but this reads as done or very close to it.