SkillCorner data pipelines — deployment guide

What you get

A complete, self-running data pipeline in your own Google Cloud project, fed by the SkillCorner API:

Raw match data on Cloud Storage, one file per match: match metadata, player-match metadata, physical data (v3.0.3), dynamic events (v3) and phases of play (v2), for every competition edition you choose, refreshed every hour as SkillCorner publishes new matches.
Season-level benchmarks in BigQuery: per player (per team and position) and per team, covering physical, in-possession and out-of-possession metrics, with per-30 / per-90 rates and ranks, recomputed every night.
Ready-to-use app files (parquet, one per competition edition, team and player) for a dashboard or analysis app.

Everything is created by one script from one config file; nothing runs on SkillCorner's side. ARCHITECTURE.md explains how the pieces fit together and which files to edit to adapt metrics, columns or names. CLAUDE.md gives Claude Code the same context, so you can open the repository with Claude and ask it to explain, adapt or extend the pipelines without briefing it first.

What you need

A Google Cloud project with billing enabled, where you are Owner (or hold the roles listed in 1.3).
SkillCorner API credentials (username and password) covering the competition editions or seasons you want to load.
A machine with the Google Cloud SDK (gcloud and bq), git and Python 3.10+; on Windows, Git Bash. No Docker: images are built in the cloud.

Details and install commands are in section 1.

Who it is for

Someone comfortable with a terminal and a Google Cloud console: you will run gcloud commands, edit a config file and read logs. No code changes are needed for a standard deployment. Basic Python helps if you later want to adapt metrics or columns. You need Owner (or the roles in 1.3) on a Google Cloud project with billing, and SkillCorner API credentials.

Time needed

Phase	Your time	Waiting
1. Prerequisites (tools, access)	15–30 min
2. Manual setup (service account, roles, secrets)	15–20 min
3. Automatic deployment (`check`, `all`)	5 min	30–45 min of image builds
3. First load (`bootstrap`)	1 min	1–4 h, depending on how many matches are in scope

About one hour of hands-on work; the platform is live by the end of the day.

The road map

flowchart TD
    subgraph manual["You, by hand, following given code - sections 1 and 2"]
        A["1. Install gcloud, bq, git and Python, log in to Google Cloud"]
        B["2.1 Enable the APIs"]
        C["2.2 to 2.4 Create the service account and grant its 10 roles"]
        D["2.5 Create the 3 secrets: SC_USERNAME, SC_PASSWORD, GCP_CREDENTIALS"]
        E["3.1 Copy config.env.example to config.env and fill it in"]
    end
    subgraph auto["deploy/setup_gcp.sh - section 3"]
        F["check: verifies everything above, prints the fix for anything missing"]
        G["all: buckets, datasets, images, services, jobs, workflows, schedulers"]
        H["bootstrap: loads the history, runs both pipelines, enables the schedulers"]
    end
    I(["Live: hourly ingestion, nightly benchmarks, app files refreshed"])
    A --> B --> C --> D --> E --> F --> G --> H --> I
    F -.-> D

Checklist

#	Step	Where	Done when
1	Tools installed, `gcloud auth login` done	1.1, 1.2 on Windows	`gcloud --version` lists `bq`
2	Project, roles for you, SkillCorner credentials at hand	1.3
3	APIs enabled	2.1	command returns without error
4	Service account created, 10 roles granted	2.2, 2.3
5	Three secrets created	2.5	`gcloud secrets list` shows all three
6	`deploy/config.env` filled in, scope chosen	3.1
7	`deploy/setup_gcp.sh check`	2.7	every line says `ok`
8	`deploy/setup_gcp.sh all`	3.2	ends without error; nothing runs yet
9	`deploy/setup_gcp.sh bootstrap`	3.3	app-file buckets populated, schedulers `ENABLED`
10	`deploy/setup_gcp.sh status`	3.4	matches the list in 3.4

The three script commands, once steps 1 to 6 are done:

deploy/setup_gcp.sh check

deploy/setup_gcp.sh all

deploy/setup_gcp.sh bootstrap

Everything the script does is idempotent: re-running a command updates what exists and creates what is missing. Section 4 is reference material you do not need for a first deployment.

1. Prerequisites

1.1 Tools on the machine that deploys

Tool	Why	Check
Google Cloud SDK (`gcloud`) with the `bq` component	creates every resource; `bq` creates the BigQuery datasets	`gcloud --version` lists `bq`
`bash` 4 or newer	the deploy script is bash. On Windows use Git Bash, not WSL (section 1.2)	`bash --version`
`git`	the image tag is the current commit hash	`git --version`
Python 3.10 or newer, standard library only	the script imports `src/config.py` to derive the bucket names; the unit tests	`python --version`

Docker is not needed: images are built by Cloud Build.

Install the SDK from https://cloud.google.com/sdk/docs/install, then add bq and log in:

gcloud components install bq

gcloud auth login

The script passes --project on every call, so it never depends on your active gcloud project. Your own ad-hoc gcloud commands do, so either pass --project too or set it:

gcloud config set project PROJECT_ID

1.2 Windows specifics

Python for the SDK. The Cloud SDK needs a real Python interpreter of its own. Without its bundled Python, gcloud may pick up the Microsoft Store placeholder and fail every command with "Python was not found". Point it at a real interpreter, in the shell you deploy from:

export CLOUDSDK_PYTHON="/c/Users/YOU/AppData/Local/Programs/Python/Python313/python.exe"

Which bash. Run the script from the Git Bash terminal. In PowerShell, plain bash resolves to WSL, which has its own filesystem view, its own gcloud and its own Python, and PowerShell environment variables do not reach it. To launch from PowerShell anyway, call Git Bash by its full path and set everything inside the command:

& "C:\Program Files\Git\bin\bash.exe" -lc 'cd /c/path/to/SC_cloud_pipeline_template && ./deploy/setup_gcp.sh check --no-pause'

Line endings. The script tolerates a config.env saved with Windows line endings and strips carriage returns from everything it reads back from gcloud and Python.

1.3 Accounts and access

A Google Cloud project with billing enabled.
For the manual setup (section 2) your Google account needs roles/iam.serviceAccountAdmin, roles/resourcemanager.projectIamAdmin and roles/secretmanager.admin on the project.
For the automatic deployment (section 3) it needs roles/serviceusage.serviceUsageAdmin, roles/storage.admin, roles/bigquery.admin, roles/artifactregistry.admin, roles/cloudbuild.builds.editor, roles/run.admin, roles/workflows.admin and roles/cloudscheduler.admin. Project Owner covers everything in both lists.
SkillCorner API credentials (username and password) with access to the competition editions or seasons you will configure (section 3.1).

1.4 Get the code

Clone the repository. All commands in this guide run from the repository root.

2. Manual setup (Google Cloud)

Everything here is done once, by hand, before the first deployment. Replace PROJECT_ID with your project id. The service-account name below is the default, sc-pipeline; it must match SA_NAME in deploy/config.env (section 3.1). The secret names are not configurable: the application code reads them by name.

2.1 Enable the APIs

gcloud services enable run.googleapis.com cloudbuild.googleapis.com artifactregistry.googleapis.com workflows.googleapis.com workflowexecutions.googleapis.com secretmanager.googleapis.com bigquery.googleapis.com bigquerystorage.googleapis.com storage.googleapis.com cloudscheduler.googleapis.com iam.googleapis.com iamcredentials.googleapis.com logging.googleapis.com --project=PROJECT_ID

(deploy/setup_gcp.sh apis does the same, and all runs it first.)

2.2 Create the service account

One account is the identity of everything: each Cloud Run job, each ingestion service, both workflows, the Cloud Scheduler tokens, and Cloud Build.

gcloud iam service-accounts create sc-pipeline --display-name="SkillCorner pipeline" --project=PROJECT_ID

An explicit account is required because, in projects created after April 2024, Cloud Build's default identity has no permissions of its own and builds fail to push.

Alternatively, reuse the project's default compute account, <project number>-compute@developer.gserviceaccount.com: skip this step and put its full email in SA_NAME. It works and skips a step, but it is shared with anything else in the project and, in older projects, carries broad permissions of its own. A dedicated account is the tidier choice where you can create one.

2.3 Grant it its roles

Eleven project-level roles, each for a specific thing the deployment does:

Role	Why it is needed
`roles/storage.objectAdmin`	every stage reads its input objects and writes its output objects
`roles/storage.bucketViewer`	the parquet read/write path goes through `gcsfs` and polars, which read bucket metadata, not just objects
`roles/bigquery.jobUser`	the workflows submit queries; stage 3 of each pipeline reads its benchmarks table
`roles/bigquery.dataEditor`	the workflows create and replace the external table and the materialised table
`roles/secretmanager.secretAccessor`	containers read the SkillCorner credentials and the key at startup
`roles/run.invoker`	the workflow starts each Cloud Run job, and Cloud Scheduler calls the four private ingestion services
`roles/run.viewer`	the workflow polls each job execution until it finishes; reading an execution is not part of `run.invoker`
`roles/workflows.invoker`	Cloud Scheduler starts a workflow execution
`roles/workflows.viewer`	the guard at the top of each workflow lists its own active executions to avoid overlapping runs
`roles/artifactregistry.writer`	Cloud Build runs as this account and pushes the images it builds
`roles/logging.logWriter`	Cloud Build writes its build logs as this account (`CLOUD_LOGGING_ONLY`)

Bash:

for ROLE in roles/storage.objectAdmin roles/storage.bucketViewer roles/bigquery.jobUser roles/bigquery.dataEditor roles/secretmanager.secretAccessor roles/run.invoker roles/run.viewer roles/workflows.invoker roles/workflows.viewer roles/artifactregistry.writer roles/logging.logWriter; do gcloud projects add-iam-policy-binding PROJECT_ID --member="serviceAccount:sc-pipeline@PROJECT_ID.iam.gserviceaccount.com" --role="$ROLE" --condition=None; done

PowerShell:

foreach ($ROLE in 'roles/storage.objectAdmin','roles/storage.bucketViewer','roles/bigquery.jobUser','roles/bigquery.dataEditor','roles/secretmanager.secretAccessor','roles/run.invoker','roles/run.viewer','roles/workflows.invoker','roles/workflows.viewer','roles/artifactregistry.writer','roles/logging.logWriter') { gcloud projects add-iam-policy-binding PROJECT_ID --member="serviceAccount:sc-pipeline@PROJECT_ID.iam.gserviceaccount.com" --role="$ROLE" --condition=None }

Nothing needs to be granted for pulling images: Cloud Run pulls as its own Google-managed service agent, which already has access to Artifact Registry in the same project. The last two roles replace Google's convenience role roles/cloudbuild.builds.builder, which would also work but grants far more than this build needs; reading the uploaded source is already covered by storage.objectAdmin.

Giving one account both the build roles and the runtime roles keeps the setup small at the cost of isolation: whatever can build images can also read every secret and write every bucket. Reasonable for a single-purpose project; split into two accounts if you later want builds to be less privileged.

2.4 Let yourself act as the account

Needed to pass --service-account when deploying, and to run a build as it. Skip if you are project Owner.

gcloud iam service-accounts add-iam-policy-binding sc-pipeline@PROJECT_ID.iam.gserviceaccount.com --member="user:YOU@example.com" --role="roles/iam.serviceAccountUser" --project=PROJECT_ID

2.5 Create the three secrets

They are mounted into containers as environment variables of the same name, so the secret name and the variable name must match exactly.

Secret	Contains	Read by
`SC_USERNAME`	SkillCorner API username	the four ingestion services, the first-load jobs, and stages 0 to 2 of both pipelines
`SC_PASSWORD`	SkillCorner API password	same
`GCP_CREDENTIALS`	the full JSON private key of the service account from 2.2	every `main.py`, parsed at start

printf %s 'YOUR_SC_USERNAME' | gcloud secrets create SC_USERNAME --replication-policy=automatic --data-file=- --project=PROJECT_ID

printf %s 'YOUR_SC_PASSWORD' | gcloud secrets create SC_PASSWORD --replication-policy=automatic --data-file=- --project=PROJECT_ID

For GCP_CREDENTIALS, generate a key for the account, store it, then delete the local file:

gcloud iam service-accounts keys create /tmp/sc-pipeline-key.json --iam-account=sc-pipeline@PROJECT_ID.iam.gserviceaccount.com --project=PROJECT_ID

gcloud secrets create GCP_CREDENTIALS --replication-policy=automatic --data-file=/tmp/sc-pipeline-key.json --project=PROJECT_ID && rm -f /tmp/sc-pipeline-key.json

The key must belong to the account named in SA_NAME. gcloud needs the identity at deploy time to set what the container runs as; the key is what the code presents at runtime; and the parquet read/write path authenticates as the runtime identity while every other call uses the key. If they name different accounts, some operations succeed and others fail with no obvious pattern. check compares them (section 2.7).

If key creation is refused, your organisation enforces iam.managed.disableServiceAccountKeyCreation (or the older iam.disableServiceAccountKeyCreation): request a project-level exception for this project.

To rotate a secret later, add a version rather than recreating it; deployments reference :latest, so the next revision picks it up:

printf %s 'NEW_VALUE' | gcloud secrets versions add SC_PASSWORD --data-file=- --project=PROJECT_ID

2.6 Check for name clashes in the project

Two kinds of resource cannot simply be created under the default names:

BigQuery datasets. A dataset's location can never be changed. If the project already holds datasets named external_tables, player_benchmarks or team_benchmarks in a location other than the one you will deploy to, set BQ_DATASET_PREFIX (section 3.1) so the deployment creates its own. The bigquery step detects this and refuses to continue otherwise.
Buckets. Bucket names are global across all of Google Cloud, so the base names in the code (match_metadata, …) are almost certainly taken already. BUCKET_PREFIX (section 3.1) makes them yours; choose something specific to your organisation.

2.7 Verify

check needs only PROJECT_ID in deploy/config.env plus an authenticated gcloud. It reads and changes nothing, and prints the exact command for anything missing.

deploy/setup_gcp.sh check

Two blind spots: roles held through a group or a custom role are invisible to it, because it reads the direct bindings in the project IAM policy, so if it reports a role you know is in place, run the deployment commands individually instead of all. And it can compare the key against SA_NAME only if it can read the secret and find a Python interpreter; otherwise it says so and treats that check as inconclusive.

3. Automatic deployment

3.1 Configuration files

deploy/config.env is your project's settings, gitignored. Create it from the template and fill it in:

cp deploy/config.env.example deploy/config.env

Variable	Meaning	Rules	Default
`PROJECT_ID`	target project		required
`REGION`	Cloud Run jobs and services, Artifact Registry, buckets		`europe-west4`
`WORKFLOW_REGION`	Cloud Workflows location	not every region offers Workflows	`europe-west2`
`SCHEDULER_REGION`	Cloud Scheduler location	validated against the available list	`WORKFLOW_REGION`
`BQ_LOCATION`	BigQuery dataset location	must be co-located with the buckets: the same region, or a multi-region (EU, US) containing it	`REGION`
`BQ_DATASET_PREFIX`	prefix on the dataset names `external_tables`, `player_benchmarks`, `team_benchmarks`	letters, digits, underscores. Use when the project has same-named datasets in another location (2.6)	empty
`AR_REPO`	Artifact Registry docker repository	lowercase letters, digits, hyphens only; no underscores; starts with a letter	`sc-pipeline`
`RESOURCE_PREFIX`	prefix on the name of every Cloud Run service and job, both workflows, the Scheduler jobs and the images	lowercase letters, digits, dashes, starts with a letter; at most 7 characters, since Cloud Run caps names at 49 and the longest base name is 42	empty
`SA_NAME`	the service account from 2.2	a bare name, completed with `@PROJECT_ID.iam.gserviceaccount.com`, or a full email for an account on another domain such as the default compute account	`sc-pipeline`
`BUCKET_PREFIX`	prefix on every bucket name	lowercase letters, digits, dash, underscore; starts with a letter or digit; no dots; total name at most 63 characters, the longest base name is 40	`acme-` in the template; change it
`IMAGE_TAG`	image tag		short git commit hash
`SC_COMPETITION_EDITIONS`	scope, option A: the SkillCorner competition edition ids this deployment covers, comma separated	digits and commas	empty
`SC_SEASONS`	scope, option B: SkillCorner season ids, comma separated; every competition edition the account can access in those seasons is covered, including ones granted later	digits and commas	empty
`TIMEZONE`, `FUNCTIONS_CRON`, `TEAMS_WORKFLOW_CRON`, `PLAYERS_WORKFLOW_CRON`	schedules, cron syntax in `TIMEZONE`	teams must finish before players starts (4.1)	hourly functions; teams 03:00; players 05:00

Exactly one of SC_COMPETITION_EDITIONS and SC_SEASONS must be set; it is the single scope setting, followed by the hourly functions, the first-load jobs and the app-metadata files (teams and players are filtered to the covered editions). Nothing outside it is ever fetched. Find the ids, with your API credentials, at https://skillcorner.com/api/competition_editions/?user=true (each edition carries its season.id) and https://skillcorner.com/api/seasons/.

The script validates every rule above when it loads the file and stops with the offending value before touching anything.

deploy/components.conf lists every deployable component with its CPU, memory, task count, parallelism, timeout and secrets. Edit this file to resize a job or change the parallelism of a first-load job.

3.2 Deploy everything: `all`

deploy/setup_gcp.sh all

Runs these steps in order, stopping at the first failure:

Step	Creates or does
`apis`	enables the APIs of 2.1
`check`	section 2.7; `all` does not continue if a prerequisite is missing
`buckets`	17 buckets, `BUCKET_PREFIX` + `analysis_app_metadata`, `match_data_collection`, `match_metadata`, `players_match_metadata`, `physical_match_data`, `dynamic_events_csv`, `phases_of_play_csv`, `dynamic_events_match_aggregates`, `merged_match_aggregates`, `competition_edition_player_aggregates`, `phases_of_play_match_aggregates`, `merged_match_team_aggregates`, `competition_edition_team_aggregates`, `bq_parquet_exports`, `app_player_files`, `app_team_files`, `pipeline_monitoring`; uniform access, public access prevented
`bigquery`	3 datasets, `BQ_DATASET_PREFIX` + `external_tables`, `player_benchmarks`, `team_benchmarks`, in `BQ_LOCATION`. Refuses to continue if a dataset of that name exists elsewhere
`build`	the Artifact Registry repository `REGION-docker.pkg.dev/PROJECT/AR_REPO` if missing, then one Cloud Build per component from the repository root, tagged with the git hash and `latest`. The slow step: a few minutes per component, 18 components
`functions`	4 private Cloud Run services: `dynamic-events-api-update`, `physical-match-data-api-update`, `match-metadata-api-update`, `app-metadata-api-update`
`jobs`	14 Cloud Run jobs: `players-pipeline-0…4-`, `teams-pipeline-0…4-`, and the one-off `data-ingestion-app-metadata`, `data-ingestion-match-metadata`, `data-ingestion-physical`, `data-ingestion-dynamic-events`
`workflow`	2 workflows, `players-pipeline-workflow` and `teams-pipeline-workflow`, with the regions and prefixes passed as workflow environment variables
`scheduler`	6 Scheduler jobs, one per ingestion service on `FUNCTIONS_CRON` and `teams-pipeline-workflow-trigger`, `players-pipeline-workflow-trigger` on their crons. All created paused

Every name above is the base name; with RESOURCE_PREFIX set, every service, job, workflow, Scheduler job and image carries the prefix, and the workflows call the prefixed jobs.

Nothing runs yet after all: jobs and workflows are definitions, and the schedulers are paused.

3.3 First run: `bootstrap`

On a scheduled run the ingestion functions fetch only matches modified in the last two or three hours, so a fresh project would ingest nothing. bootstrap loads the history and runs the pipelines once:

deploy/setup_gcp.sh bootstrap

data-ingestion-app-metadata: the competition editions in scope, their teams and players, and a matches_list.csv covering every one of their matches (stage 2 of both pipelines needs it). Single task.
data-ingestion-match-metadata: match and player metadata for every match.
data-ingestion-physical: physical v3.0.3 for every match that has it.
data-ingestion-dynamic-events: dynamic events v3 and phases of play v2 for every match that has them.
teams-pipeline-workflow, then players-pipeline-workflow, waiting for each.
Resumes every Scheduler job. From here on the deployment runs itself.

The four jobs list candidates from the API, skip matches that already have a file in the target bucket, and shard the rest across their Cloud Run tasks, so re-running is always safe and only fetches what is missing. Each task runs up to 15 concurrent requests; raise task counts in components.conf with the API's rate limits in mind. If your account holds only some datasets for an edition (physical but not dynamic events, say), see 4.6.

3.4 Check the deployment

deploy/setup_gcp.sh status

Lists services, jobs, workflows, schedulers, buckets, datasets and secrets. After a successful bootstrap you should see:

every raw bucket holding one file per match in scope, and analysis_app_metadata holding competition_editions.json, competition_editions_csv.csv, teams.csv, players.csv and matches_list.csv;
four BigQuery tables: external_tables.ce_player_benchmarks, player_benchmarks.ce_player_benchmarks, external_tables.ce_team_benchmarks, team_benchmarks.ce_team_benchmarks (under BQ_DATASET_PREFIX when set);
bq_parquet_exports with the three exported parquet files;
app_player_files and app_team_files with per-competition, per-team and per-player files;
Scheduler jobs in state ENABLED.

Job and workflow logs are in the Cloud Run and Workflows console pages, or from the terminal (with RESOURCE_PREFIX in the job name when set):

gcloud logging read 'resource.type="cloud_run_job" AND resource.labels.job_name="players-pipeline-4-app-files"' --project PROJECT_ID --limit 50 --format "value(textPayload)"

Every stage prints how many items it selected and why; a stage that fails exits non-zero, so the workflow execution fails and shows which stage broke.

3.5 Day-to-day operations

Ship a code change. Commit, then, from the repository root:

deploy/setup_gcp.sh deploy

Or for one component (base or prefixed name):

deploy/setup_gcp.sh deploy --only players-pipeline-2-ce-aggregates

deploy is build + functions + jobs + workflow. It creates definitions and runs nothing; the schedulers keep firing and pick up the new revision at the next tick. Images are tagged with the commit hash, so a Cloud Run revision tells you which code it runs, and rolling back is pointing it at an older tag. Since the image is built from your working tree, uncommitted edits are deployed too, which is why committing first keeps the tag honest.

Widen the scope. Add ids to SC_COMPETITION_EDITIONS or SC_SEASONS in deploy/config.env, run deploy so every function and job picks it up, then ingest: only the matches of the new editions are fetched. With SC_SEASONS, an edition newly granted to your account in a configured season needs no change at all: the next hourly run covers it, and ingest loads its history.

Fill a gap or re-run the first load. ingest re-runs all four first-load jobs, or one with --only; both only fetch what is missing. FORCE=1 in a job's environment refetches everything.

deploy/setup_gcp.sh ingest --only data-ingestion-physical

The hourly functions also accept ?hours=N (and ?refresh_players=1 for app metadata) for a small manual gap-fill, but each such call is one Cloud Run request capped at an hour; for anything larger use ingest.

See what a command would do. Every command accepts --dry-run, which prints the mutating gcloud commands instead of running them while still performing the read-only lookups:

deploy/setup_gcp.sh all --dry-run

Pause behaviour. Every command waits for Enter before exiting so the output stays readable in a window that closes on its own, also after a failure. The wait is skipped when there is no terminal, so pipes and CI never hang, and --no-pause disables it.

3.6 Commands and options

Command	Does
`check`	verifies the manual prerequisites and the scope setting; prints a fix command for each gap
`apis`	enables the required APIs
`buckets`	creates the buckets
`bigquery`	creates the datasets
`build`	Artifact Registry repository if needed, then the images
`functions`	deploys the four ingestion functions
`jobs`	deploys the pipeline stages and the first-load jobs
`workflow`	deploys both workflows
`deploy`	`build` + `functions` + `jobs` + `workflow`
`scheduler`	creates the Scheduler jobs paused; `--resume` enables them
`all`	`apis`, `check`, then every step above in order
`ingest`	runs the four first-load jobs in dependency order, waiting for each
`bootstrap`	`ingest`, both workflows once, then resumes the schedulers
`status`	inventory of what is deployed

Option	Effect
`--dry-run`	print mutating commands instead of running them
`--no-pause`	do not wait for Enter before exiting
`--only NAME`	restrict `build`, `functions`, `jobs`, `workflow`, `deploy`, `ingest` to one component, base or prefixed name
`--resume`	with `scheduler`: enable the Scheduler jobs

DRY_RUN=1, NO_PAUSE=1 and CONFIG_FILE=path work as environment variables too. The flags exist because an inline VAR=1 command prefix is bash syntax and fails in PowerShell.

4. Reference

4.1 Order of the pipelines

Stage 3 of the players pipeline also exports the team benchmarks table, and stage 4 joins team profiling columns onto the player files. The teams workflow must therefore run first; the default crons leave two hours between them. Adjust them if a teams run takes longer.

After its stage 2, each workflow runs CREATE OR REPLACE EXTERNAL TABLE over the stage-2 parquet bucket and then materialises the benchmarks table with CREATE OR REPLACE TABLE … AS SELECT. No BigQuery table is created by hand, and columns added by stage 2 propagate on the next run.

Stage 4 writes a competition edition when stage 2 refreshed it in the last six hours or when it has no output file yet, so a first or interrupted run always completes the app files.

4.2 SkillCorner API endpoints

All access is direct HTTPS with Basic auth in src/skillcorner_api.py, with retries on throttling and transient errors and pagination handled. No SkillCorner client library is needed. Reference: https://skillcorner.com/api/docs/

Data	Endpoint
competition editions in scope	`/competition_editions/?user=true` (filtered to the ids, or `&season=…`)
teams, players	`/teams/`, `/players/` with `competition_edition=`
match list with `*_last_modified__gte` filters	`/matches/`
which datasets exist per match	`/data_collections/`
match metadata, data-collection state	`/match/{id}/`, `/match/{id}/data_collection/`
dynamic events v3, phases of play v2	`/match/{id}/dynamic_events/`, `/match/{id}/dynamic_events/phases_of_play/` (CSV)
physical v3.0.3	`/physical/?match={id}&data_version=3.0.3&possession=all,tip,otip&period=full,h1,h2&physical_check_passed=true`

The physical endpoint returns one row per player with possession and period pivoted into column suffixes (total_distance_full_tip, minutes_h1_otip, …); those are the names stage 1 of both pipelines reads, and the physical first-load job checks the first response it gets against them.

4.3 How the code reaches the containers

Every image is built from the repository root with one of the two shared Dockerfiles in deploy/docker/, the component directory passed as a build argument. The shared src/ library is copied in; there is no package install from a private repository and no access token anywhere in the build.

4.4 Runtime environment of jobs and services

Variable	Set by	Used for
`GCP_PROJECT`	the script	BigQuery project in the stage-3 exporters
`BUCKET_PREFIX`	the script	bucket names, in `src/config.py`
`BQ_DATASET_PREFIX`	the script	dataset names in the workflows and the stage-3 exporters
`SC_COMPETITION_EDITIONS` or `SC_SEASONS`	the script, whichever is set in `config.env`	the scope everything covers
`GCP_CREDENTIALS`	Secret Manager	service-account key JSON
`SC_USERNAME`, `SC_PASSWORD`	Secret Manager	SkillCorner API
`CLOUD_RUN_TASK_INDEX`, `CLOUD_RUN_TASK_COUNT`	Cloud Run	task sharding inside the jobs
`FORCE`	you, on a first-load job	`1` refetches matches already present

To run a stage locally, export the same variables, with GCP_CREDENTIALS holding the JSON text of a key, and run python main.py from the stage directory with the repository root on PYTHONPATH.

4.5 Tests

No credentials needed:

python -m unittest discover -s tests -t .

4.6 Partial data access

A SkillCorner account may hold only some datasets for a competition edition: physical data without dynamic events, or the reverse. The deployment handles that per dataset:

The hourly functions and the first-load jobs select matches per dataset (physical_v3_0_3 and dynamic_events_v3 availability), and a match the API refuses with HTTP 403 is logged as "no access" and skipped, never counted as a failure. A first-load job whose matches are all refused exits successfully and says so; ingest and bootstrap carry on.
Stage 1 of both pipelines merges whatever exists: a match without dynamic events gets null event metrics, one without physical data gets null physical metrics. Stage 2 counts matches per dataset (count_match_physical, count_match_gi) and ranks only non-null values, so a physical-only edition yields physical benchmarks and ranks with the event columns empty.
Editions with no data at all simply do not appear in the tables or the app files.

Name		Name	Last commit message	Last commit date
api_updates		api_updates
data_ingestion		data_ingestion
deploy		deploy
players_pipeline		players_pipeline
src		src
teams_pipeline		teams_pipeline
tests		tests
.gcloudignore		.gcloudignore
.gitignore		.gitignore
ARCHITECTURE.md		ARCHITECTURE.md
CLAUDE.md		CLAUDE.md
README.md		README.md
pyproject.toml		pyproject.toml

Latest commit

History

Folders and files

Repository files navigation