What does fit do when calculating realized performance?

[shezadkhan137]

Hi, Thanks for all of your work on Nannyml!

We want to use nannyml to write realized performance data into a SQL database, and then use Grafana to plot these metrics over time. My question is is there any way to use the PerformanceCalculator without having to call fit first? I'm not exactly sure what fit is doing, and it seems to be quite slow, even on small amounts of data.

[santiviquez]

Hi @shezadkhan137!

When using the PerformanceCalculator method, the purpose of the fit function is to create the baseline that represents the realized performance on the reference set. This is where nannyML calculates the thresholds that can later be useful in determining whether there has been degradation on the analysis set, like in the image below.

Regarding your question about using PerformanceCalculator with out calling fit first, actually yes, you can.

You only need to call fit when the reference data changes. So, if your reference data hasn't change, you can call fit only once, persist the fitted calculator (we currently support multiple ways to persist calculator objects), and then use it to perform calculations whenever needed. This should reduce the running time.

I'll let you here a tutorial on how to store and load calculators

I hope this helps :) Let me know if you have any questions!

[nnansters]

Additional bit of information. The duration of fitting will depend on multiple factors:

• Size of the reference data set: the larger it is, the longer fitting will take.
• Chunking: the more chunks you end up with, the more computations will be executed during fitting, so the longer it will take.
• The operation itself. This should not take too much time for the realized performance calculation.

That being said, the intended flow is indeed to fit once and reuse the fitted calculator as long as the reference data doesn't change (too much).