SVI Support

[juanitorduz]

Tell us about it

When reading https://arxiv.org/pdf/1804.06788 I see that

SBC requires just one assumption: that we have a generative model for our data. Given
such a model, we can run any given algorithm over many simulated observations and the self
consistency condition (1) provides a target to verify that the algorithm is accurate over that
ensemble, and hence sufficiently calibrated for the assumed model.

This means we can still use this for variational inference methods, right? It would be great to have support!

Thoughts on implementation

Not sure, but happy to help.

[aloctavodia]

Here there is one example https://hyunjimoon.github.io/SBC/articles/computational_algorithm1.html (also notice that their docs are very good, like this one https://hyunjimoon.github.io/SBC/articles/rank_visualizations.html)

[juanitorduz]

Thanks! Do you have plans to support this in simuk? How "easy/hard" could this be?

[juanitorduz]

I think for numpyro, It would be "simple" to do, the only question would be the API 🤔 (as we need to receive a guide). Have you ever considered decoupling the fitting from the simulation step?

Another motivation would be a PyMC-Marketing model: We have a custom wrapper ala scikit-learn and it would be great to have these diagnostics applied once we have a fitted model. (food for thought) .

[aloctavodia]

It should be simple. The API is open to discussion. Currently, we receive a model and then sample from the prior predictive of that model. But we should also support users passing an arbitrary data-generation-process.

[juanitorduz]

ok! I might draft something because I am looking for practical diagnostics for SVI and this one seems a very good one to have!

I will first try numpyro + SVI to get some inspiration (PyMC and Pathfinder might benefit from this extension as well)

[aloctavodia]

awesome! Yeah ideally we should make easy for people to use different inference algorithms. For Bambi should be very easy as they use bayeux for that, but not sure about the details