This repository contains code for the Penang catch data pipeline.
Pipeline input(s): A manually updated excel sheet containing catch information at Penangs landing sites.
Pipeline product(s): A BigQuery dataset with clean catch data. The dataset includes tables with (i) the full clean dataset, (ii) aggregated summaries, summaries per (iii) fisher, and (iv) species.
Stages:
- Input data (excel format) is stored in Google Storage. Updates are manual.
- Changes in the storage file are automatically detected and trigger a Google pub/sub notification.
- The pub/sub message triggers a call to an API processing function that reads the data from the storage. The function then validates, cleans, and transforms, the data. Finally, the clean data is deposited in a Google BigQuery table. The processing function runs serverless in Google CloudRun.
- Create a service
account
for this domain:
deployment/create-service-account.sh - Create a
key
for the service account:
deployment/create-service-account-key.sh - Upload the
key
to Google Secret Manager to simplify reuse:
deployment/create-secret.sh - Create the
storage
bucket:
deployment/create-storage-bucket.sh - Grant service account
permissions
to admin storage objects:
deployment/grant-service-account-storage-permissions.sh - Create topic for
message passing between the storage bucket and the validation API:
deployment/create-pubsub-validation-topic.sh - Enable
notifications
for raw data updates to the validation topic:
deployment/create-raw-data-notification.sh - Deploy data validation API using a local
build.
This will create the url for the service:
deployment/deploy-validation-api.sh - If its the first time running pubsub push service,
setup pubsub
permissions and create and invoker account for push notifications,
skip otherwise:
deployment/setup-pubsub-permissions.sh - Grant permissions to invoke validation api:
deployment/grant-validation-api-permisions.sh - Create pubsub
subscription
to data validation topic:
deployment/create-pubsub-validation-subscription.sh - Manually upload data to bucket
- Docker >= 18.0
- Google Cloud SDK >= 311.0.0 including the Local Builder