ETL Workbench

A small local Apache Airflow workbench for trusted, code-defined ETL pipelines. It runs Airflow and, when requested, local PostgreSQL, ClickHouse, n8n and S3-compatible object storage. Pipeline code and data contracts stay in their own repositories.

This is a single-user development tool. It is not a shared scheduler, control plane, deployment platform, or isolation boundary for untrusted DAG code.

Requirements

• Docker Desktop or Docker Engine with Compose
• at least 4 GB of memory available to Docker
• a pipeline Git repository containing dags/
• Dockerfile.airflow in that repository when the pipeline needs its own image

Start a Git pipeline

For a public repository:

./bin/etl-workbench https://github.com/example/acme-pipeline.git

For a private SSH repository:

./bin/etl-workbench git@github.com:example/acme-pipeline.git \
  --ssh-key ~/.ssh/id_ed25519

The command builds the workbench image, builds the pipeline's Dockerfile.airflow, configures Airflow's native GitDagBundle, starts local PostgreSQL and object storage, and waits for the services to become healthy. Airflow then clones and refreshes the DAG bundle itself. Each task run records the Git version of the DAG code that produced it.

The local PostgreSQL service contains a dedicated airflow metadata database. Pipeline tables remain separate in their own databases or schemas.

Open http://127.0.0.1:18080. The generated local login is stored inside the airflow-home volume:

docker compose exec airflow \
  cat /var/lib/airflow/simple_auth_manager_passwords.json.generated

To expose only the authenticated Airflow UI on a trusted local network, set AIRFLOW_UI_HOST=0.0.0.0 when starting the launcher. Database and object-store ports keep their localhost-only defaults.

Useful options:

--ref VERSION            branch, tag, or commit; default: main
--subdir PATH            DAG directory; default: dags
--image IMAGE            use a prebuilt pipeline image
--bundle-manifest FILE   load several Git DAG sources; requires --image
--env FILE               pipeline-owned runtime environment
--external-db            do not start local PostgreSQL
--external-objects       do not start local object storage
--git-connection ID      use an existing Airflow Git connection
--analytics              start a dedicated local ClickHouse analytics database
--inference-gpu          start vLLM on one NVIDIA GPU
--automation             start local n8n workflow automation

With --external-db, set AIRFLOW_METADATA_DATABASE_URL to a supported PostgreSQL SQLAlchemy URL. Do not put that URL in a committed environment file.

With --ssh-key, the launcher writes a generated Airflow connection to the ignored .workbench/runtime.env with mode 0600. The private key is used by Docker BuildKit and the local Airflow container; it is not copied into the image. Host-key checking uses ~/.ssh/known_hosts by default.

Several product sources in one Airflow

One Airflow can load DAG entrypoints from several independent Git repositories. Use a versioned JSON manifest when a shared factory serves several trusted products:

{
  "version": 1,
  "sources": [
    {
      "name": "learning-platform",
      "repository": "git@github.com:example/learning-platform.git",
      "ref": "main",
      "subdir": "airflow/dags"
    },
    {
      "name": "beavers-data",
      "repository": "git@github.com:example/beavers-data-pipelines.git",
      "ref": "main",
      "subdir": "dags"
    }
  ]
}

Then start the factory with an image which contains the compatible Python packages of every listed product:

./bin/etl-workbench \
  --bundle-manifest trusted-products.json \
  --image trusted-airflow-pipelines:2026-07-22 \
  --ssh-key ~/.ssh/id_ed25519

The factory creates one Git Connection per source and configures Airflow's native GitDagBundle list. A Git bundle provides DAG files only; it must never install arbitrary dependencies at parse time. The shared image is therefore an explicit release artifact, built and tested from pinned product revisions.

Keep source-specific Connections, object prefixes and Pools named by product. That separates operational ownership inside one trusted Airflow, but does not turn this local workbench into an isolation boundary for untrusted code.

Pipeline repository contract

The smallest repository contains one or more DAG files:

acme-pipeline/
├── dags/
│   └── pipeline.py
└── Dockerfile.airflow

A pipeline image can add Python packages or application code:

ARG ETL_WORKBENCH_IMAGE=etl-workbench:local
FROM ${ETL_WORKBENCH_IMAGE}

COPY --chown=airflow:root pyproject.toml src/ /tmp/pipeline/
RUN pip install --no-cache-dir /tmp/pipeline

The launcher overrides ETL_WORKBENCH_IMAGE with the locally built workbench image. Runtime secrets belong in an ignored pipeline environment file and are passed with --env; never bake them into the image or DAG files.

Airflow discovers compatible DAGs from the Git bundle and displays them in its UI. The pipeline repository owns schemas and migrations, retry and idempotency behavior, object keys and retention, and all business logic.

Local profile connection IDs are local_postgres, local_s3, local_clickhouse, local_n8n and llm_local_vllm; the local bucket is etl-local. SeaweedFS supplies the local S3-compatible endpoint. External connections may be created in the Airflow UI or provided as AIRFLOW_CONN_* variables in the pipeline environment file.

Airflow metadata

Local Airflow metadata is stored in the dedicated PostgreSQL database airflow, not in the pipeline database etl. On the first start after upgrading an existing Workbench, the metadata initializer:

1. creates a consistent SQLite backup;
2. initializes the PostgreSQL schema with airflow db migrate;
3. copies all Airflow tables in one PostgreSQL transaction;
4. validates row counts and resets PostgreSQL sequences;
5. writes a private migration marker.

The original SQLite file is not removed. The stable rollback copy and marker are stored inside the airflow-home volume:

/var/lib/airflow/backups/airflow-sqlite-pre-postgres.db
/var/lib/airflow/.metadata-postgres-migrated.json

Both files use mode 0600; the backup still contains sensitive encrypted metadata and must not be copied into the repository. The launcher stops Airflow before the first migration so the SQLite snapshot cannot miss concurrent task updates.

Verify the active backend:

scripts/check-airflow-metadata-contract.sh

To roll back, stop Airflow, preserve the PostgreSQL and airflow-home volumes, restore the backup as /var/lib/airflow/airflow.db, and run the previous Workbench release. Do not delete either volume while investigating a migration.

LLM connections

The workbench image includes the Airflow OpenAI provider. Create each provider as an independent openai Connection in the Airflow UI; its Password is the provider-specific API key. Use the Host field for the OpenAI client's base URL (or set openai_client_kwargs.base_url in Extra).

Connection ID	Host
llm_kimi	https://api.moonshot.ai/v1
llm_deepseek	https://api.deepseek.com
llm_gemini	https://generativelanguage.googleapis.com/v1beta/openai/
llm_qwen	Model Studio endpoint for the selected region and workspace
llm_mistral	https://api.mistral.ai/v1
llm_xai	https://api.x.ai/v1

Pipeline code selects the conn_id and model name. It must not contain API keys. For portability across these providers, use the Chat Completions API and avoid OpenAI-specific APIs unless that pipeline is intentionally tied to OpenAI.

Connection testing is enabled for this trusted, single-user workbench. It makes a live request with the stored credential; for the OpenAI provider, this is a model-list request. Gemini's OpenAI-compatible endpoint does not expose that model-list route, so validate a Gemini connection with a Chat Completions task instead.

Local GPU inference with vLLM

On a Linux host with an NVIDIA GPU, NVIDIA Container Toolkit and enough GPU memory for the selected model, add --inference-gpu:

./bin/etl-workbench https://github.com/example/acme-pipeline.git \
  --inference-gpu

The profile starts the official vLLM OpenAI server on http://127.0.0.1:18000/v1. Airflow receives an openai Connection named llm_local_vllm; pipeline tasks keep using the same conn_id plus model contract as cloud providers.

The conservative default downloads Qwen/Qwen3-0.6B and exposes it under the stable API name local-model. Change the weights and API name independently:

VLLM_MODEL=Qwen/Qwen3-0.6B \
VLLM_SERVED_MODEL_NAME=local-qwen \
  ./bin/etl-workbench https://github.com/example/acme-pipeline.git \
  --inference-gpu

The Hugging Face cache is persistent in the vllm-huggingface-cache volume. Set HF_TOKEN only for a model whose repository requires it. The image and model weights are large downloads; the profile is never started implicitly. Usage-stat collection is disabled.

The committed vllm-local API key is a loopback-only development credential. If the Compose network is shared, override both VLLM_API_KEY and AIRFLOW_CONN_LLM_LOCAL_VLLM so the server and Airflow Connection remain in sync.

Verify model discovery and one synthetic Chat Completions request:

scripts/check-vllm-contract.sh

vLLM exposes Prometheus metrics at /metrics. LLM request telemetry is not bundled. Keep model prompts, responses, credentials and personal data out of infrastructure logs and metric labels.

Docker Desktop on macOS cannot expose the Apple GPU to this CUDA container. For Mac-local inference, run Ollama or an MLX server on the host and create another openai Connection pointing to http://host.docker.internal:<port>/v1. The platform contract stays conn_id + model; Ollama and MLX are not bundled into the GPU profile.

Local automation with n8n

Add --automation to start n8n for webhooks, SaaS integrations, notifications and other business actions:

./bin/etl-workbench https://github.com/example/acme-pipeline.git \
  --automation

Open http://127.0.0.1:18083 and create the first local owner in the n8n UI. The launcher creates a dedicated n8n PostgreSQL database and generates one stable encryption key in the ignored .workbench/n8n.env file with mode 0600. The key must remain stable because n8n uses it to encrypt stored credentials. The n8n-data volume keeps n8n application state that does not belong in PostgreSQL. This local profile cannot be combined with --external-db.

Airflow owns scheduled ETL, dependencies, retries, data quality and large data movement. n8n owns external triggers and business-system actions. Keep one workflow owner for each compute path instead of splitting individual ETL steps between both orchestrators.

The preferred boundary is:

Airflow -> PostgreSQL, ClickHouse or S3 -> n8n -> SaaS, notification or API

Publish compact completed events with a stable event_id; do not make n8n process large tables or partially written objects. Airflow can call an n8n webhook through the local_n8n HTTP Connection. When n8n needs to start a DAG, create an n8n credential for the Airflow API at http://airflow:8080 and keep that credential in n8n rather than in a workflow export.

Verify that n8n is reachable and its PostgreSQL migrations are complete:

scripts/check-n8n-contract.sh

The profile is intended for internal local automation. n8n uses its Sustainable Use License; review the current license before offering n8n itself as a hosted or white-label product.

Local analytics with ClickHouse

Add --analytics to start a dedicated ClickHouse database for pipeline datasets:

./bin/etl-workbench https://github.com/example/acme-pipeline.git \
  --analytics

The HTTP endpoint is http://127.0.0.1:18123. Inside the Compose network, Airflow uses the local_clickhouse Connection and the official clickhouse-connect Python client. The local defaults are database and user analytics, with password analytics-local; override AIRFLOW_CONN_LOCAL_CLICKHOUSE when using non-local credentials.

This ClickHouse is persistent and belongs to pipeline analytics: event facts, large append-heavy datasets, aggregates and reporting marts. PostgreSQL remains the right default for Airflow metadata, transactional state and small relational datasets.

A pipeline can resolve the Airflow Connection without committing credentials:

from airflow.sdk.bases.hook import BaseHook
import clickhouse_connect

connection = BaseHook.get_connection("local_clickhouse")
client = clickhouse_connect.get_client(
    host=connection.host,
    port=connection.port or 8123,
    username=connection.login,
    password=connection.password,
    database=connection.schema,
)

Verify DDL, insert and query access through the Airflow container:

scripts/check-clickhouse-contract.sh

SeaweedFS keeps a free-space reserve before allocating new volumes. If an S3 upload returns InternalError, check docker logs etl-workbench-object-store-1 for No writable volumes and no free volumes left, then reclaim Docker image or build cache space. Do not delete named volumes as a cleanup shortcut.

Local path development

The included example can be mounted read-only without Git:

docker build -t etl-workbench:local .
docker compose -f compose.yaml -f compose.local.yaml \
  --profile local-db --profile local-objects up

Set PIPELINE_ROOT to use another local repository. This fallback expects both dags/ and src/; GitDagBundle is the normal repository integration.

Verify

docker compose config --quiet
docker compose -f compose.yaml -f compose.local.yaml config --quiet
docker build -t etl-workbench:local .
docker compose -f compose.yaml -f compose.local.yaml run --rm airflow python -c \
  'from airflow.models import DagBag; b=DagBag("/opt/airflow/dags"); assert not b.import_errors, b.import_errors'
docker compose -f compose.yaml -f compose.local.yaml \
  --profile local-objects run --rm \
  -v "$PWD/scripts:/opt/workbench/scripts:ro" airflow \
  python /opt/workbench/scripts/check-s3-contract.py
docker compose -f compose.yaml -f compose.local.yaml \
  --profile local-db --profile analytics up -d --wait
scripts/check-clickhouse-contract.sh

The S3 contract check writes only below a unique _workbench_contract/ prefix and removes its objects before returning. It verifies put, metadata, get, list, presigned GET, copy, multipart upload and delete through the same Airflow local_s3 Connection that pipeline tasks use.

Upgrade from MinIO

SeaweedFS uses a new seaweedfs-data volume; it cannot read the MinIO volume format directly. The old minio-data volume is never removed by the upgrade. If it contains objects that must be retained, stop the old stack without deleting volumes and run the one-time copy:

docker compose down --remove-orphans
docker volume inspect etl-workbench_minio-data
docker compose -f compose.yaml -f compose.minio-migration.yaml \
  --profile local-objects --profile migrate-minio \
  up --abort-on-container-exit migrate-minio
docker compose -f compose.yaml -f compose.minio-migration.yaml \
  --profile local-objects --profile migrate-minio down

The migration copies the current contents of ETL_LOCAL_BUCKET and verifies that the complete, sorted object key-and-size inventory has the same SHA-256 digest in both stores. This avoids order-dependent mc diff output; the copy still fails if an object is missing, added only to the target, renamed or has a different size. It does not delete the source volume. After it succeeds, start the workbench normally. To roll back, stop the new stack without --volumes and run the previous workbench release against the preserved minio-data volume.

Stop

Keep local history and data:

docker compose --profile local-db --profile local-objects down

Explicitly delete workbench volumes and generated Git credentials:

docker compose --profile local-db --profile local-objects down --volumes
rm -rf .workbench

Scheduled runs stop when the laptop or Compose stack stops. Shared scheduling, high availability, distributed executors, remote secret management, and untrusted DAG execution are outside this workbench's scope.

License

Apache-2.0. See LICENSE.