Orchestration Tool POC: Dagster

[alexrichey]

TODO from Wed Dec 3 Demo

• deployment patterns. Namely, validating our approach for branch-specific runs.
• security - None. We'll have to use Azure RBAC via APIM
• Resources integration with our connectors
• code locations - what's the potential use?
• how do different partitions interact
• do sensors do what we think?
• external assets

Notes

Downsides

No RBAC in OSS deployment

[damonmcc]

notes on the topics listed above, some notes are more general than just dagster

deployment patterns. Namely, validating our approach for branch-specific runs.

we should probably differentiate between PRs that modify dagster and ones that modify a data product. PRs should only do one of the two

for PRs that modify dagster ("the app"), we could wither just use local deployments to QA and include details in the PR description for reviewers or use a shared staging app in Azure to deploy to

for PRs that modify a data product, seems like we can keep using DB schemas and blob directories to separate builds by branch (environment_helpers.py in Dagster's internal project repo). maybe configuring assets using envars would make this pretty easy?

security - None. We'll have to use Azure RBAC via APIM

anyone in citynet would be able to navigate to the relevant URL. primary concern is that they could then materialize assets using the Dagster UI

Resources integration with our connectors

resources are "reusable objects that supply external context, such as database connections, API clients, or configuration settings", for example to centralize a database connection across multiple assets

I need to review the latest patterns in our connectors but resources alone may not be enough for stuff like special blob directory things (draft, pulbish, etc.). but maybe connectors/resource don't have to be how stuff like that is handled

code locations - what's the potential use?

haven't thought of one yet since we have a monorepo. only idea was it might be helpful with branch-specific runs

how do different partitions interact

do sensors do what we think?

external assets