BIG IDEA: add metrics trees / input-output relations between metrics

[siljamardla]

The current spec focuses on the data model - essentially instructing a query compiler on what needs to be queried and how to get the required information out of the data source.

There's another side of metrics and their relations that is currently not covered at all.
There are different names to it, for example "metrics tree" or the "input-ouput" framework.

I will explain shortly, but for good visuals and business reasoning of why a metrics tree is important, refer to https://doubleloop.app/ or https://www.hellotrace.io/

Metric trees are about what impacts what. Here's a simple example:

• Revenue metric is defined on top of a sales table
• In a marketplace business the total revenue is impacted by the number of buyers online and by the number of sellers who have products on sale (defined on top of some customer tracking event table and some product listing event table)

With the current spec we are able to calculate each of the 3 metrics, because we define where the numbers should come from. To show the relation between these metrics, we'd have to establish those relation patterns outside of OSI.

Why it really matters? Ideally, a data driven business has output metrics which typically have a slow response time to actions, that in turn are driven by input metrics which can be directly impacted by responsible teams by setting up projects with the goal of improving the input metrics. Then we monitor the input and output metrics to see if our projects had the desired impact.

It's also good to note that there's a large variation of metric trees that can be created for the same metric. For example the revenue metric is also impacted by the average order value and the number of orders.

And, the impact can sometimes be expressed with an explicit expression (e.g. revenue = order count x average order value) while sometimes the relation is just... correlation.

[sfc-gh-wpugh]

Is the concept here a semantic one? Or one where the metrics actually derive from one another, so they can be determined by looking at lineage?

[kurtStirewalt]

@siljamardla , I agree that this is a nice feature that makes these models easier to read and reuse. Per @sfc-gh-wpugh 's comment, if the key idea is really to support derived metrics in the OSI -- i.e., one metric can be specified as a formula that refers to other metrics rather than to columns in base data sets -- then would you object to renaming the request to be for "derived metrics" rather than "metric trees"?

[siljamardla]

I am definitely not talking about derived metrics.
Derived metrics are also needed, absolutely. Roughly 1/3 of all metrics are derived metrics.

Here I'm talking about metrics that might have absolutely nothing to do with each other from the perspective of how they're calculated or what the data source are. The relation of these metrics is strictly about the business effect - improving the input metrics, the output metric is going to improve.

[sfc-gh-wpugh]

Ah. O.k. I think I get what you are talking about. This would help with something like looking at relationships between leading and lagging indicators?

For example, If I want to lose weight, and think I can do it by number of workouts per week. There is an implied relationship between workouts per week and weight trend. Even though, they are measuring completely different things.

I'm assuming the point of the metrics tree is to validate whether these relationships actually hold?

[khush-bhatia]

@sfc-gh-wpugh Validation would be one use case, but I am thinking a more important use case is giving better context to agents for reasoning. For example when someone asks a question why is revenue declining, the agent could leverage the metric tree .

[willpugh]

@khush-bhatia to be clear, my example was not about validation. It is about validating hypotheses and understanding correlation (and cause/effect).

I agree the agentic context is definitely useful. However, it seems like at its root answering the question of "Do these correlations I think exist, actually exist? Are they cause and effect" are tricky questions to answer and can lead to different answers.

[khush-bhatia]

@siljamardla Enhancing OSI to support correlated metrics for business reasoning is a great idea.

@sfc-gh-wpugh and @kurtStirewalt Thoughts on scoping this as part of the ontology working group ? It seems like a good use case for something that could be represented in an ontology.

[lucianof]

In Duodata (https://duodata.ai/), we call this the "value driver tree". For every metric, the business stakeholders can define which are the most important metrics that drive this given metric. We would be happy to contribute our experience with value driver trees and metrics management in general to the ontology working group!