Auto-Ingester

Automates the reconciliation of DDEX album submissions with IDAGIO's metadata database. Receives preprocessed DDEX data from LMS, reconciles entities against Alpaca, and ingests matched albums via Alpaca's ingestion API.

Architecture

• Temporal Worker: Executes album reconciliation workflows
• HTTP API (TODO): Accepts workflow requests, exposes workflow state

External Dependencies

• LMS (Label Metadata Service) - Source of preprocessed DDEX submissions
• Alpaca Service - Metadata database and ingestion API
• Temporal - Workflow orchestration

Infrastructure

• Temporal (cloud vs self-hosted TBD)

Code Generation

Alpaca API types are generated from OpenAPI snapshots using go-swagger.

Updating Alpaca Types

1. Export the latest swagger spec from alpaca-service (docs/swagger.yaml)
2. Save it to upstream/alpaca/openapi/snapshots/<epoch-timestamp>.yaml (e.g., 1770155211.yaml)
3. Run make generate-alpaca-types

The Makefile automatically uses the most recent snapshot (sorted numerically). Generated types live in upstream/alpaca/openapi/types/.

Development Setup

Pre-requisites

• GNU Make
• Go >= 1.25
• Docker or local install of Temporal (for local Temporal server)

Environment Setup

cp .env.example .env

Then configure the environment variables in the .env file.

Running Temporal Locally

Local install MacOS:

brew install temporal
temporal server start # Start Temporal server
make worker  # Start Temporal worker

How to run tests?

make test                                    # Run all tests
make test path=internal/temporal/workflows   # Run tests for specific package

How to start the service?

make worker  # Start Temporal worker

API

TBD

CI

TBD

Deployment

TBD

Logging

There are two logging mechanisms, serving different purposes:

Worker logs

Verbose structured logs via activity.GetLogger(ctx), emitted to the worker's stderr. Use for detailed debugging: raw API responses, search results, etc.

Set LOG_LEVEL=debug in .env to see debug-level output.

Activity log (Temporal Memo)

A concise trail of key events stored on the workflow's Temporal Memo under "activity_log", viewable in the Temporal web UI for any workflow run. Designed to give operators a quick overview of the decisions each activity made, especially as activities incorporate LLM-backed reasoning.

al := memo.NewActivityLogger(ctx)
al.Append("label_matched", fmt.Sprintf("matched %q to ID %s", name, id))
return activities.Done(al, result), nil

The workflow unwraps each result and accumulates the log trail:

var fetchResult activities.WithLog[*lms.SubmissionMetadata]
err := workflow.ExecuteActivity(ctx, lmsAct.FetchLMSSubmission, id).Get(ctx, &fetchResult)
trail = append(trail, fetchResult.Log...)
_ = workflow.UpsertMemo(ctx, map[string]interface{}{"activity_log": trail})