Develop naming guideline for Airflow DAGs

[jordanpadams]

Description

Develop and document a naming guideline for all Airflow DAGs used across Nucleus (dag_id, filenames, task_ids, and tags). There is no official Airflow standard, but the conventions below reflect de facto best practice and should be adapted/ratified for Nucleus's MWAA/Airflow 2.x environment.

All identifiers (dag_id, filenames, task_ids, tags) should be lowercase.

dag_id conventions

• Use a stable, descriptive dag_id decoupled from the filename — Airflow keys off dag_id, not the file path, so files can be renamed without breaking history/state.
• Lead with the node abbreviation: {node}_{workflow}_{frequency}, e.g. sbn_ingest_daily, atm_registry_sweep_hourly, img_delayed_submission_daily. Prefixing with the node keeps all of a node's DAGs grouped together and makes them easy to search/sort alphabetically in the Airflow UI — important given Nucleus's multi-tenant, per-node structure.
• Avoid embedding version numbers or dates in dag_id unless intentionally versioning DAGs — task history and SLA tracking key off dag_id, so changing it orphans prior run history.
• All lowercase with underscores, no spaces — matches Python identifier conventions since dag_id often mirrors the DAG object's variable name.

File naming

• Match the filename to the dag_id where practical (sbn_ingest_daily.py) for easy grep/locate.
• Avoid generic names like dag.py or pipeline.py repeated across folders — the scheduler parses every .py in the DAGs folder, and ambiguous names make triage harder during backlog/failure debugging.

Task IDs within a DAG

• Keep task_ids short, lowercase, action-oriented, and unique within the DAG (extract_labels, validate_products, load_to_registry) rather than restating the DAG name.

Versioning approach

• Two options when a structural change would break comparability with historical runs: (1) mutate the DAG in place and rely on dag_id stability, or (2) use explicit version suffixes (sbn_ingest_v2).
• Check whether the target MWAA Airflow 2.x version includes native DAG versioning support (available in Airflow 2.2+).

Tags

• Use the tags=[] parameter (e.g. tags=["nucleus", "sbn-psi", "production"]) to filter in the UI by node, environment, or workstream — more flexible than cramming everything into the dag_id.

Given the per-node EFS and delayed-submission work happening on Nucleus, a {node}_{workflow}_{frequency} pattern with tags for environment (prod/dev) and node likely maps cleanly onto the multi-tenant structure already in use.

Acceptance Criteria

• Naming guideline documented (dag_id, filename, task_id, tags conventions)
• Guideline reviewed/agreed upon by team
• Guideline published in project docs (e.g. README or wiki)