[Hackathon Use Case] Exploring Integration of Legend with GCP BigQuery

[mishagavronsky]

Exploring Integration of Legend with GCP

Business Problem

We want to manage our logical data model in Legend, then create a physical manifestation of this logical model in GCP BigQuery. Ingest data into BigQuery physical model, use this data for machine learning type calculations that introduces a change to a physical model in BigQuery. As the last step, we want to synchronize changes to a physical data model in BiqQuery with a logical data model in Legend.

Expected Features

Bi-directional integration of 1) Legend to use GCP/BigQuery to manifest a physical data model, given a logical model defined in Legend 2) BigQuery to use Legend to update Legend logical model.

Use Legend JSON to update existing logical model in Legend.

FinosHackathonLegendUsecase.pdf

[maoo]

Thanks @mishagavronsky , this looks very interesting!

Side note, if you're looking on deploying Legend on GCP - which I understand that may be convenient, given the integration with BigQuery, I'd suggest looking at https://github.com/finos/legend-juju-bundle

For more info on the Juju GCP integrator, please checkout https://ubuntu.com/kubernetes/docs/charm-gcp-integrator

Hope this helps!

[pedroleaoc]

Ok, I think I have an idea that would make this a very pleasant automated experience.

If Legend allows you to export the model via an API, we can inject that into BigQuery with GCP's API. That python code could then be put into a charm, a really simple one, a GCP BigQuery Integrator charm. This charm would keep the credentials safe as K8s secrets and would have actions to extract and import models.

juju run-action extract-model myLegendModel
juju run-action import-model myBQModel

Bam!
Your model went from Legend to BigQuery and to Legend again. If BigQuery can tell the API went the job is done, then that can be automated even. Your credentials are safe as a secret and you can even write some validation code for the JSON. The integrator charm can also be deployed with the Legend Bundle.

Let me take a look to see if the python code would be too complicated.

[pedroleaoc]

This is a very interesting use case! I love the idea of allowing Legend models to be probed by external services. It opens up a world of possibilities.

[pedroleaoc]

Registering my progress here :)

[TheJuanAndOnly99]

Closing discussion in preparation for this years hackathon. Feel free to reopen and update if its still relevant.