fairproxy-api

This project is in its early development stages. Stability is not guaranteed, and documentation is limited. We welcome your feedback and contributions.

Overview

fairproxy-api is a FastAPI service that exposes FAIR-oriented metadata and native payloads for heterogeneous data platforms behind a single API surface.

Supported Platform

Socrata (a.k.a. Data Insights) is the only fully implemented platform at this time. Support for MTNA Rich Data Services in under development.

Socrata

Socrata (also known as Data Insights) is a cloud data publishing platform used by governments and organizations to host open data portals and dataset APIs.

Socrata datasets through both a unified FAIR URI route (/datasets/{uri}) and a Socrata-specific route (/socrata/{host}/{dataset_id}).

Supported Standards

• Croissant — MLCommons metadata format for machine learning datasets.
• DCAT — W3C vocabulary for publishing data catalogs on the web.
• DDI Codebook — XML standard for documenting survey and social science datasets.
• DDI-CDI (via CDIF profile) — semantic model for structured data integration and interoperability.
• Markdown — lightweight plain-text format for human-readable dataset summaries.
• Postman Collection — machine-readable API request collection format for tooling and sharing.
• platform-native metadata via the common /native endpoint

Installation

This project recommends using uv for fast and reliable Python package management.

Setup with uv

To set up the development environment with uv:

# Install dependencies and create virtual environment
uv sync

Run project tasks with uv run:

# Run tests
uv run pytest

# Build documentation
uv run sphinx-build -b html docs/source docs/build/html

Optional: Hatch via uvx

You can also use Hatch directly:

# Run tests
uvx hatch run test

# Enter the default shell
uvx hatch shell

Configuration

The FAIRification API relies on a servers.yaml file to resolve and discover data servers. By default, the application uses an embedded servers.yaml file packaged with the source.

To customize or externalize this configuration (especially in containerized environments), you can configure the file path using the following lookup order:

1. Environment Variable: Set the FAIRPROXY_SERVERS_CONFIG environment variable to the absolute path of your custom YAML file.
2. Container Config: Mount your custom file to /app/config/servers.yaml (automatically detected inside Docker containers).
3. System Config: Place your custom file at /etc/fairproxy/servers.yaml.
4. Embedded Default: Fall back to the packaged servers.yaml.

Docker Compose Configuration

To mount and use a custom servers configuration in Docker Compose, update your docker-compose.yaml service definition:

services:
  api:
    # ...
    volumes:
      - ./my-custom-servers.yaml:/app/config/servers.yaml:ro

Development

Version Management

Versions are managed dynamically in src/dartfx/fairproxy_api/__about__.py.

Secret Management

For local development, create a .env file in the root directory. This file is git-ignored and can be used to store local API keys or configuration. These are automatically loaded by the test suite.

Run API Locally

From this project directory, install dependencies first:

uv sync

Start the API from this project directory:

uv run uvicorn --app-dir src/dartfx fairproxy_api.main:app --host 0.0.0.0 --port 8000 --reload

If you prefer to run it from the monorepo root, use the package-targeted form:

uv run --package dartfx-fairproxy-api uvicorn --app-dir dartfx-fairproxy-api/src/dartfx fairproxy_api.main:app --host 0.0.0.0 --port 8000 --reload

Alternative using PYTHONPATH:

PYTHONPATH=src/dartfx uv run uvicorn fairproxy_api.main:app --host 0.0.0.0 --port 8000 --reload

Quick health check:

curl http://127.0.0.1:8000/status

Quick tests:

# get list of servers
curl http://127.0.0.1:8000/servers


# Get San Francisco 311 Dataset (Socrata)
curl http://127.0.0.1:8000/socrata/data.sfgov.org/wg3w-h783/native
curl http://127.0.0.1:8000/socrata/data.sfgov.org/wg3w-h783/ddi/codebook
curl http://127.0.0.1:8000/socrata/data.sfgov.org/wg3w-h783/ddi/cdi
curl http://127.0.0.1:8000/socrata/data.sfgov.org/wg3w-h783/dcat

# Get San Francisco 311 Dataset (Unified)
curl http://127.0.0.1:8000/datasets/socrata:data.sfgov.org:wg3w-h783/markdown
curl http://127.0.0.1:8000/datasets/socrata:data.sfgov.org:wg3w-h783/postman/collection

Running with Docker

You can also run the application containerized using Docker and Docker Compose:

# Build the container image
docker compose build

# Start the service in the background
docker compose up -d

# Verify it is running
curl http://localhost:8000/status

To stop the container:

docker compose down

Verifying and Publishing the Image

We provide helper scripts to verify and publish the Docker image:

1. Verify: Validate the built image locally to ensure the container starts up and responds to status checks:

   ./verify_images.sh

2. Publish: Build (if not present locally), verify, tag (with latest and the dynamic project version from __about__.py), and push the image to a container registry:

   # Build, verify, tag, and publish to Docker Hub (default namespace 'dartfx')
   ./publish_image.sh
   
   # Publish with a custom namespace or custom tag
   ./publish_image.sh --namespace custom-namespace --tag custom-tag

Running Tests

uv run pytest

Building Documentation

uv run sphinx-build -b html docs/source docs/build/html

Usage

Health Check

curl http://127.0.0.1:8000/status

Current Top-Level Routes

• /status for health status
• /servers for configured data server discovery
• /datasets/{uri} for unified dataset metadata and native payloads
• /socrata/{host}/{dataset_id} for Socrata-specific dataset access

Generic routes like /catalog, /resources, and /vocab are intentionally not exposed in the current build.

Socrata URI Format

The unified datasets endpoint expects Socrata URIs in this format:

socrata:<server>:<dataset-id>

Where:

• socrata is the platform identifier.
• <server> is the Socrata host (for example, data.sfgov.org).
• <dataset-id> is the Socrata dataset identifier (for example, wg3w-h783).

Example URI:

socrata:data.sfgov.org:wg3w-h783

Example Socrata Metadata Endpoints

Unified dataset route using a FAIR URI:

curl "http://127.0.0.1:8000/datasets/socrata:data.sfgov.org:wg3w-h783/ddi/codebook"
curl "http://127.0.0.1:8000/datasets/socrata:data.sfgov.org:wg3w-h783/postman/collection"
curl "http://127.0.0.1:8000/datasets/socrata:data.sfgov.org:wg3w-h783/markdown"
curl "http://127.0.0.1:8000/datasets/socrata:data.sfgov.org:wg3w-h783/native"

Socrata-native route using host and dataset id:

curl "http://127.0.0.1:8000/socrata/data.sfgov.org/wg3w-h783/ddi/codebook"
curl "http://127.0.0.1:8000/socrata/data.sfgov.org/wg3w-h783/native"

Notes

• /datasets/{uri}/native is the platform-agnostic native payload endpoint.
• /socrata/{host}/{dataset_id}/native exposes the same native payload through the Socrata-specific route.
• The older /socrata native suffix has been replaced by /native.

Roadmap

• MTNA Rich Data Service (MTNA RDS) adapter support
• Additional platform adapters aligned with the common DatasetProvider interface

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines and information on how to get started.