Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions astro-airflow-mcp/.claude/skills/provider-registry/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
---
name: provider-registry
description: Airflow Provider Registry CLI for looking up providers, modules, parameters, and connections. Use when working with af registry commands, registry caching, or the registry API endpoints.
---

# Provider Registry CLI

Query the public Airflow Provider Registry from the terminal. No Airflow instance or auth required — hits the static JSON API directly with httpx (not the adapter/instance system).

## Commands

```bash
af registry providers # All providers (id, name, version, lifecycle)
af registry modules <provider_id> # Operators, hooks, sensors, transfers
af registry parameters <provider_id> # Constructor params for all classes
af registry connections <provider_id> # Connection types with fields
```

All commands accept `--version <ver>` to pin a specific release and `--no-cache` to bypass local cache.

## Detailed reference

- [api-endpoints.md](api-endpoints.md) — Registry API endpoints and response shapes
- [caching.md](caching.md) — Cache location, TTLs, and implementation details

## Examples

```bash
# Find all hooks in amazon provider
af registry modules amazon | jq '.modules[] | select(.type == "hook") | .name'

# Get constructor params for FTPHook
af registry parameters ftp | jq '.classes["airflow.providers.ftp.hooks.ftp.FTPHook"]'

# List connection types
af registry connections amazon | jq '.connection_types[] | .connection_type'

# Override registry URL
af registry providers --registry-url https://custom-registry.example.com/registry
```
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# Registry API Endpoints

Base URL: `https://airflow.apache.org/registry` (override with `--registry-url` or `AF_REGISTRY_URL`)

## Endpoints

| Endpoint | CLI command | Description |
|---|---|---|
| `/api/providers.json` | `af registry providers` | All providers |
| `/api/providers/{id}/modules.json` | `af registry modules {id}` | Modules (latest version) |
| `/api/providers/{id}/{ver}/modules.json` | `af registry modules {id} -v {ver}` | Modules (pinned version) |
| `/api/providers/{id}/parameters.json` | `af registry parameters {id}` | Constructor params (latest) |
| `/api/providers/{id}/{ver}/parameters.json` | `af registry parameters {id} -v {ver}` | Constructor params (pinned) |
| `/api/providers/{id}/connections.json` | `af registry connections {id}` | Connection types (latest) |
| `/api/providers/{id}/{ver}/connections.json` | `af registry connections {id} -v {ver}` | Connection types (pinned) |

## Response Shapes

### providers.json

```json
{
"providers": [
{"id": "amazon", "name": "Amazon", "version": "9.22.0", "lifecycle": "production", "description": "..."}
]
}
```

### modules.json

```json
{
"provider_id": "amazon",
"provider_name": "Amazon",
"version": "9.22.0",
"modules": [
{
"id": "amazon-s3-S3Hook",
"name": "S3Hook",
"type": "hook",
"import_path": "airflow.providers.amazon.aws.hooks.s3.S3Hook",
"module_path": "airflow.providers.amazon.aws.hooks.s3",
"short_description": "Interact with Amazon S3.",
"docs_url": "https://airflow.apache.org/docs/...",
"source_url": "https://github.com/apache/airflow/blob/...",
"category": "amazon-web-services",
"provider_id": "amazon",
"provider_name": "Amazon"
}
]
}
```

Module types: `operator`, `hook`, `sensor`, `transfer`

### parameters.json

```json
{
"provider_id": "ftp",
"classes": {
"airflow.providers.ftp.hooks.ftp.FTPHook": {
"name": "FTPHook",
"type": "hook",
"mro": ["FTPHook", "BaseHook"],
"parameters": [
{"name": "ftp_conn_id", "type": "str", "default": "ftp_default"}
]
}
}
}
```

### connections.json

```json
{
"provider_id": "amazon",
"connection_types": [
{
"connection_type": "aws",
"hook_class": "airflow.providers.amazon.aws.hooks.base_aws.AwsBaseHook",
"standard_fields": ["login", "password"],
"custom_fields": [{"name": "region_name", "type": "str"}]
}
]
}
```
39 changes: 39 additions & 0 deletions astro-airflow-mcp/.claude/skills/provider-registry/caching.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Registry Caching

## Location

Cache directory: `~/.af/.registry_cache/`

Each cached response is stored as `{sha256_of_url}.json` containing:

```json
{"_cached_at": 1709400000.0, "_url": "https://...", "_payload": {...}}
```

## TTL Strategy

| Request type | TTL | Rationale |
|---|---|---|
| Unversioned ("latest") | 1 hour (3600s) | Points to latest version, changes on new releases |
| Versioned (pinned) | 30 days (2592000s) | Immutable snapshots, content never changes |

The TTL is determined by whether `--version` was passed to the command. The URL itself encodes this — versioned URLs contain the version segment (e.g. `/providers/amazon/9.22.0/modules.json`).

## Cache Bypass

- `--no-cache` flag skips both read and write
- Delete the directory to clear all: `rm -rf ~/.af/.registry_cache/`

## Implementation

- `_read_cache(url, ttl)` — SHA256 key lookup, TTL check with negative-age guard (handles backward clock jumps)
- `_write_cache(url, payload)` — Creates directory if needed, silently ignores write errors
- `_fetch(url, no_cache, versioned)` — Orchestrates cache read → HTTP fetch → cache write

## Adding New Cached Commands

When adding a new registry command:

1. Call `_build_url()` to construct the URL
2. Pass `versioned=version is not None` to `_fetch()` — this selects the right TTL
3. The caching is automatic from there