Scoped access

[thorwhalen]

When trying to access nested structures It's annoying to have to remember those convoluted paths to the objects we want to access.

For example, try finding in sklearn. Go on, try it!

If you don't use it every day, you won't remember it's from sklearn.metrics.cluster import expected_mutual_info_fast` unless you search google or your code.

Now, most well behaved libraries will be decent enough to expose their main functionalities at "the top" so you can do from sklearn import expected_mutual_info_fast. But most libraries are not well behaved.

Then when you consider nested data such as configuration data, or all kinds of junk you get off the internet, things get even nastier.

What scoped access proposes is just one piece of the puzzle: The ability to get specify easily how to search for the keys you're looking for.

Take this example:

d = {
    'a': {
        'b': {
            'c': 3,
            'd': 4,
            'e': {
                'a': 1
          }
        },
        'c': {
            'd': 44
        }
    },
    'b': {
        'c': 33
    }
}

We'd like to be able to do scoped_d = ScopedAccess(d, 'a.b', alt=... ) so that

>>> scoped_d['c']
3

Now if the search does not yield an exact match, the "alt" strategy is used instead. For example if we defined alt to be "look down farther the nested structure" then we would get the following:

>>> scoped_d['a']
1

Some typical choices for all:

• Raise KeyError (default)
• Return default object (like what dict.get(..., default) does
• Use a factory to generate an object and put it there (what defaultdict does)
  (So far I'm showing that all of these are INSTANCES of what I call "scoped access")
• Come back to the nested object and search one level deeper... (as in the example above)

Implementation ideas

A general form could be

def search(key, path, mapping, next_search: Callable):
    "Searches mapping[path] for key. If not found, calls search(*next_search(key, path, mapping))"

Particular forms:

• Searching an iterable of paths, like ChainMap
• Recursively searching the tree downwards until a match is found.
• Same as above, but search upwards. This corresponds to searching for the first global default.
• Searching all leaves. Customizable handling of duplicates.

Compile the mapping

One notable way to handle duplicates is to not handle them at all (through the runtime handler), but instead, pre-check the
mapping for duplicates, and handling duplicates then, so that we can ensure there are none later.

Every tree can be (lossless/duplicate-free) flattened to a {tree_path: val, ...} mapping and the paths themselves can be reduced to the minimum length needed to keep the keys distinct. This could be a {reduced_path: original_path, ...} or a direct `{reduced_path: val, ...} mapping according to the needs.

This can then be used as a compiled version of the mapping, or at least index thereof. When searching for a key, the reduced mapping's keys would be used to match, and if not found, the query key would be expanded to include it's direct parent, searched again, and so forth.

Related to

#10

Appendix

If you come up with this situation of finding an import path, you can using tec for that:

>>> from tec import ModulesReader, find
>>> import sklearn
>>> list(find('expected_mutual_info_fast', ModulesReader(sklearn), lambda q, i: q in i))
['sklearn.metrics.cluster.expected_mutual_info_fast']

[thorwhalen]

Still open: no "scoped access" helper found in the current codebase. Related infrastructure exists nearby though -- `front/state.py`'s `State(MutableMapping)`/`BoundVal`/`Binder` give scoped, chainable read/write access to a state mapping (see #10, #22), and could plausibly be the substrate a "find the deeply-nested thing without remembering its full path" helper would build on. Nothing does that specific job yet. Leaving open.