Skip to content

videre-project/CardExporter

BranchesTags

Repository files navigation

CardExporter

MTGO

CardExporter keeps Videre's MTGO card database and CDN assets in sync with Magic: The Gathering Online by extracting the MTGO client's local card data into PostgreSQL, card/product renders, and generated files for Cloudflare R2.

Overview

MTGO distributes the data CardExporter needs across several sources:

  • card XML in the ClickOnce data directory;
  • runtime client models used to fill gaps in set metadata;
  • validation-rule files that describe format legalities; and
  • WPF resources and client assemblies that contain symbols and other image assets.

CardExporter turns those sources into three forms of operating state:

  1. PostgreSQL records for cards, sets, products, faces, and legalities.
  2. Pending image work for new or changed card and product catalog IDs.
  3. Manifest files that track MTGO source inputs and generated CDN objects.

Price history is imported separately from GoatBots with sync-prices. It writes catalog price definitions and daily sell-price history into PostgreSQL without starting MTGO or touching R2.

Each scheduled import first fetches Videre's MTGO version manifest and compares its codebase field with manifests/mtgo-version.json. When the codebase is unchanged, the run returns before checking local MTGO files, PostgreSQL, image work, or client assets. When it is new, changed, or temporarily unavailable, CardExporter falls back to the local manifests and database checks below.

The normal flow is:

MTGO source files ──> import ──> PostgreSQL
                          │             │
                          │             └─ new or changed image catalog IDs
                          └─ source-file manifest

catalog IDs ──> sync-images ──> Cloudflare R2 ──> CDN manifest

client assemblies ──> sync-assets ──> Cloudflare R2 ──> CDN manifest

manifests/mtgo-source-files.xml controls incremental import work:

  • card-data changes trigger card, set, and product imports;
  • validation-rule changes update legalities without requiring a full card import; and
  • unchanged tracked inputs do not rewrite the database.

After an import identifies new or changed catalog IDs, image synchronization renders those IDs through the MTGO client. Successful uploads update CDN tracking. Failed uploads do not roll back the database import; the affected image work remains pending for a later sync-images run.

Requirements

  • Docker with Docker Compose
  • Access to a shared mtgo-db PostgreSQL database
  • A valid MTGO account for commands that log into the client, render images, or read live runtime metadata
  • Cloudflare R2 credentials for commands that upload files or list bucket contents

The container provides Wine, Xvfb, and the Windows .NET runtime required by the MTGO and WPF code paths. Linux dotnet can build the solution, but commands that load WPF resources or interact with MTGO should run through the image's wine-run alias.

Quick start

1. Configure the environment

Create .env in the repository root using the settings described in Configuration.

2. Build the solution

docker compose run --rm cardexporter \
  dotnet build Project.slnx

3. Preview the scheduled run

docker compose run --rm cardexporter \
  wine-run CardExporter/CardExporter.csproj import --sync-images --dry-run

The dry run checks the tracked inputs and reports planned work without changing PostgreSQL, R2, or either manifest.

4. Start the scheduler

docker compose up -d

Follow scheduler logs with:

docker logs -f cardexporter

Stop the scheduler with:

docker compose down

To run the import and synchronization flow once instead of waiting for a scheduled poll:

docker compose run --rm cardexporter \
  wine-run CardExporter/CardExporter.csproj import --sync-images

Configuration

Create a .env file in the repository root.

Credentials and service endpoints

# PostgreSQL
CARDEXPORTER_DATABASE_URL=Host=<mtgo-db-host>;Port=5432;Database=mtgo;Username=<user>;Password=<password>
CARDEXPORTER_POSTGRES_HOST=postgres
CARDEXPORTER_POSTGRES_PORT=5432

# Cloudflare R2. Required by upload and bucket-reconciliation commands.
CF_S3_Access_Key_ID=...
CF_S3_Secret_Access_Key=...
R2_BUCKET_NAME=mtgo-cdn
R2_ENDPOINT_URL=https://<account-id>.r2.cloudflarestorage.com
R2_PUBLIC_BASE_URL=https://r2.videreproject.com

CardExporter also understands the POSTGRES_DB, POSTGRES_USER, and POSTGRES_PASSWORD variables from mtgo-db/.env when CARDEXPORTER_DATABASE_URL is unset. In that mode, use CARDEXPORTER_POSTGRES_HOST and CARDEXPORTER_POSTGRES_PORT if the database is not reachable as postgres:5432.

CardExporter does not create or maintain permanent PostgreSQL tables. The catalog schema lives in mtgo-db; CardExporter only writes into those existing tables and creates temporary staging tables for each import transaction.

Optional path and output overrides

The Docker setup already supplies the default paths. Override them only when running against a different filesystem layout.

CARDEXPORTER_SOURCE_MANIFEST_ROOT=/workspace/manifests
CARDEXPORTER_MTGO_VERSION_MANIFEST_URL=https://api.videreproject.com/mtgo/manifest
CARDEXPORTER_CDN_MANIFEST=/workspace/manifests/mtgo-cdn.csv
CARDEXPORTER_MTGO_APP_DIR=/path/to/MTGO/app
CARDEXPORTER_MTGO_CACHE_ROOT=/path/to/MTGO/card/cache

EXPORT_OUTPUT_ROOT=/workspace/output
EXPORT_CARD_HEIGHT=300
CARDEXPORTER_ASSET_OUTPUT_ROOT=/workspace/output/assets

Optional schedule overrides

The Docker service runs schedule by default. The default schedule polls every five minutes during the normal MTGO patch and maintenance windows:

CARDEXPORTER_SCHEDULE_TIME_ZONE=America/Los_Angeles
CARDEXPORTER_SCHEDULE_WINDOWS=Tuesday=08:00-10:00;Wednesday=09:00-12:00
CARDEXPORTER_SCHEDULE_POLL_MINUTES=5

Commands

Command Use it for Primary side effects
schedule Running the recurring patch-window import loop Scheduled import --sync-images polls
import Incremental card, set, product, and legality imports PostgreSQL and source tracking; optional image/CDN work with --sync-images
sync-prices Importing GoatBots card definitions and daily price history PostgreSQL price definition and history tables
sync-images Rendering and uploading missing or pending card/product images Local render output, R2, CDN tracking, and image-work state
export-images Rendering selected card/product images for inspection Local files only
r2-manifest Rebuilding the local CDN manifest from the bucket manifests/mtgo-cdn.csv
r2-upload Uploading already-rendered local PNGs R2 and CDN tracking
sync-assets Extracting and publishing MTGO client assets Generated asset files, R2, and CDN tracking
export-mtgo-assets Extracting MTGO client assets without publishing them Local files only
inspect Investigating MTGO source data and parser behavior Diagnostic output

schedule

Run the patch-window scheduler.

docker compose up -d

The scheduler polls according to CARDEXPORTER_SCHEDULE_WINDOWS. Each poll runs the same import path as:

docker compose run --rm cardexporter \
  wine-run CardExporter/CardExporter.csproj import --sync-images

When the MTGO version manifest is unchanged, the poll returns before starting MTGO or checking local source files. Schedule windows use DAY=HH:mm-HH:mm entries separated by semicolons.

Use --schedule-job sync-prices for the independent GoatBots price-history poller. The compose file includes a separate cardexporter-prices service that runs this job after GoatBots publishes the daily dump:

docker compose up -d cardexporter-prices

For an interactive tmux workflow, use the tmux override file to keep the containers idle and run the schedulers in foreground panes. Build once first so both panes can run the same DLL without racing over bin/obj files:

docker compose -f docker-compose.yml -f docker-compose.tmux.yml up -d cardexporter cardexporter-prices
docker exec cardexporter bash -lc 'cd /workspace && dotnet build CardExporter/CardExporter.csproj'
tmux set-option -g set-titles on \; \
  set-option -g set-titles-string "#S: #W" \; \
  new-session -s "MTGO" -n "Automation" -c /home/csqua/Documents/MTGOBot \
    'printf "\033]2;MTGO Bot\007"; docker exec -it mtgobot wine-run' \; \
  split-window -v -p 35 -c /home/csqua/Documents/CardExporter \
    'printf "\033]2;CardExporter Import\007"; docker exec -it cardexporter bash -lc "export PATH=/opt/wine/bin:\$PATH; cd /workspace/CardExporter/bin/Debug/net10.0-windows && wine cmd /c \"C:\\dotnet\\dotnet.exe CardExporter.dll schedule\""' \; \
  split-window -h -p 50 -c /home/csqua/Documents/CardExporter \
    'printf "\033]2;CardExporter Prices\007"; docker exec -it cardexporter-prices bash -lc "export PATH=/opt/wine/bin:\$PATH; cd /workspace/CardExporter/bin/Debug/net10.0-windows && wine cmd /c \"C:\\dotnet\\dotnet.exe CardExporter.dll schedule --schedule-job sync-prices\""'

import

Import changed MTGO data into PostgreSQL.

docker compose run --rm cardexporter \
  wine-run CardExporter/CardExporter.csproj import

The source-file manifest determines which import stages need to run. Add --sync-images when the same invocation should also process pending card/product image work and client-asset synchronization.

docker compose run --rm cardexporter \
  wine-run CardExporter/CardExporter.csproj import --sync-images

Add --dry-run to report detected changes and planned work without mutating PostgreSQL, R2, or the manifest files.

sync-prices

Import GoatBots card definitions and sell-price history into PostgreSQL. A normal run downloads the latest daily price archive and the current card definition archive:

docker compose run --rm cardexporter \
  wine-run CardExporter/CardExporter.csproj sync-prices

Add --dry-run to parse and stage the archives without committing rows:

docker compose run --rm cardexporter \
  wine-run CardExporter/CardExporter.csproj sync-prices --dry-run

Use --backfill --from-year 2023 to import yearly archives from GoatBots:

docker compose run --rm cardexporter \
  wine-run CardExporter/CardExporter.csproj sync-prices --backfill --from-year 2023

Price sync is catalog-ID based. IDs that GoatBots publishes but CardExporter has not seen in cards, card variants, or products are logged and skipped.

sync-images

Render and upload card or product catalog IDs that PostgreSQL expects but the CDN manifest does not yet track.

Use this command for:

  • an initial image backfill;
  • retrying uploads after a partial failure; or
  • processing image work independently from a card-data import.
docker compose run --rm cardexporter \
  wine-run CardExporter/CardExporter.csproj sync-images

Add --dry-run to list the missing IDs without rendering or uploading them.

export-images

Render card or product images to disk without uploading them. This is useful for checking output quality before a sync.

docker compose run --rm cardexporter \
  wine-run CardExporter/CardExporter.csproj export-images --catalog-ids 57140

Common filters and overrides:

--catalog-ids 57140
--set SOS
--card-height 300
--output-root output

r2-manifest

List the R2 bucket and rebuild manifests/mtgo-cdn.csv from object metadata.

docker compose run --rm cardexporter \
  wine-run CardExporter/CardExporter.csproj r2-manifest

This is a reconciliation and recovery command. Use it after a manual bucket migration or when the local CDN manifest must be reconstructed from R2. It does not render or upload files.

r2-upload

Upload local PNGs from --output-root. The CDN manifest is updated only for successful uploads.

docker compose run --rm cardexporter \
  wine-run CardExporter/CardExporter.csproj r2-upload --output-root output

Use r2-upload when files have already been rendered locally and should be published without running the full image-synchronization flow.

sync-assets

Extract MTGO client assets and upload generated files whose SHA-256 differs from CDN tracking.

docker compose run --rm cardexporter \
  wine-run CardExporter/CardExporter.csproj sync-assets

The command publishes these prefixes:

card-counters/
mana-symbols/
player-counters/
set-symbols/

export-mtgo-assets

Extract MTGO symbols and client image assets to disk without uploading them.

docker compose run --rm cardexporter \
  wine-run CardExporter/CardExporter.csproj export-mtgo-assets

Files are written under output/assets/ unless --output-root or CARDEXPORTER_ASSET_OUTPUT_ROOT selects another directory.

inspect

Inspect the MTGO data directory while developing parsers or investigating individual source records.

docker compose run --rm cardexporter \
  wine-run CardExporter/CardExporter.csproj inspect --find-catalog-id 57140

State and recovery

CardExporter uses separate state for source detection, database work, and CDN publication.

Source-file manifest

manifests/mtgo-source-files.xml records tracked MTGO inputs by size, timestamp, and SHA-256. It is used to determine which import or asset stages need to run.

manifests/mtgo-version.json is the first import gate. It mirrors Videre's public MTGO version manifest, while the skip decision uses the manifest's codebase field. The file is written only after a successful run, so a failed import does not mark a changed MTGO codebase as handled.

Database synchronization

The database records card and product catalog IDs added or changed by an import. R2 uploads for their respective renders are handled outside the database transaction, so a successful database import can be followed by a failed upload. In that case, the image remains pending and can be retried with sync-images or r2-upload in a separate invocation.

CDN manifest

manifests/mtgo-cdn.csv stores CDN object metadata used by synchronization and upload commands, including object keys, public URLs, timestamps, content types, byte counts, and SHA-256 hashes.

Use r2-manifest when this local file must be rebuilt from the actual bucket contents.

Art overrides

manifests/mtgo-art-overrides.xml describes known MTGO art-cache repairs that must be applied before rendering affected card images. Most of these substitute artIDs from their reprints, with a handful of cards using scanned art from the original paper printings.

Generated output

Files under output/ are staging artifacts. Export and synchronization commands recreate the files they need, so the directory can be discarded between runs unless its contents are awaiting a manual r2-upload.

CDN layout

The default public R2 base URL is:

https://r2.videreproject.com

Object keys follow these conventions:

cards/{catalogId}-300px.png
products/{catalogId}-300px.png
card-counters/{slug}.svg
player-counters/{slug}.png
set-symbols/{setCode}-{rarity}.png
mana-symbols/{symbol}.svg

Cards and products can share an MTGO catalog ID, so they use separate CDN prefixes.

Project layout

CardExporter/
├── CardExporter/
│   ├── src/
│   │   ├── CLI/             # Command dispatch and command implementations
│   │   ├── Database/
│   │   │   ├── Postgres/    # Staging tables, copy writers, merge logic, image-work queries
│   │   │   └── R2/          # R2 client, CDN manifest, and object-key helpers
│   │   └── MTGO/
│   │       ├── Files/       # Source-file indexing and manifests
│   │       ├── Parsing/     # MTGO XML parsing and SDK normalization adapters
│   │       ├── Records/     # Card, set, face, legality, and product records
│   │       └── Rendering/   # Card rendering and client-asset extraction
│   └── CardExporter.csproj
├── manifests/               # Source, CDN, and art-override manifests
├── docker-compose.yml       # Scheduled Wine/MTGO runtime service
└── Project.slnx

License

Licensed under Apache-2.0.

This project is not affiliated with Wizards of the Coast or Daybreak Games.

About

CLI for syncing MTGO card data and client assets with PostgreSQL and Cloudflare R2.

Resources

Readme

License

Apache-2.0 license
Activity
Custom properties

Stars

0 stars

Watchers

1 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages