Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 68 additions & 0 deletions .github/workflows/build-distribution-kit.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
name: Build reviewed distribution kit

on:
workflow_dispatch:
inputs:
theme:
description: Native carousel theme
required: true
default: intelligence-dark
type: choice
options:
- intelligence-dark
- editorial-light

permissions:
contents: read

concurrency:
group: distribution-kit-${{ github.ref }}
cancel-in-progress: true

jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Check out source
uses: actions/checkout@v7

- name: Check out published data
uses: actions/checkout@v7
with:
ref: gh-pages
path: published

- uses: actions/setup-go@v6
with:
go-version: "1.25.12"
cache: true

- name: Validate the published evidence bundle
run: go run ./cmd/validator -dir published/data -min-reporters 40

- name: Build the review-pending kit
run: |
go run ./cmd/distributor \
-briefing published/data/briefing.json \
-out "${RUNNER_TEMP}/distribution-kit" \
-base-url https://elecpapaya.github.io/TradeGravity/ \
-theme "${{ inputs.theme }}"

- name: Verify upload assets remain review-gated
shell: bash
run: |
test -f "${RUNNER_TEMP}/distribution-kit/carousel/slide-01.png"
test -f "${RUNNER_TEMP}/distribution-kit/carousel/slide-06.png"
test -f "${RUNNER_TEMP}/distribution-kit/carousel/caption.md"
grep -q '"distribution_status": "review_pending"' "${RUNNER_TEMP}/distribution-kit/manifest.json"
grep -q '"social_publish_authorized": false' "${RUNNER_TEMP}/distribution-kit/manifest.json"
grep -q '"theme": "${{ inputs.theme }}"' "${RUNNER_TEMP}/distribution-kit/manifest.json"
grep -q '"caption_path": "carousel/caption.md"' "${RUNNER_TEMP}/distribution-kit/manifest.json"

- name: Upload review artifact
uses: actions/upload-artifact@v4
with:
name: tradegravity-distribution-kit-${{ github.run_id }}
path: ${{ runner.temp }}/distribution-kit
if-no-files-found: error
retention-days: 14
36 changes: 34 additions & 2 deletions .github/workflows/quality.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,11 +45,43 @@ jobs:
- name: Validate offline sample dataset
run: go run ./cmd/validator -dir examples/sample-data -min-reporters 3

- name: Build offline distribution kit
run: |
go run ./cmd/distributor \
-briefing examples/sample-data/briefing.json \
-out "${RUNNER_TEMP}/distribution-kit" \
-base-url https://example.org/TradeGravity/
test -f "${RUNNER_TEMP}/distribution-kit/email/body.html"
grep -q '{{UNSUBSCRIBE_URL}}' "${RUNNER_TEMP}/distribution-kit/email/body.html"
test -f "${RUNNER_TEMP}/distribution-kit/carousel/slide-06.svg"
test -f "${RUNNER_TEMP}/distribution-kit/carousel/slide-06.png"
test -f "${RUNNER_TEMP}/distribution-kit/carousel/caption.md"
grep -q 'https://example.org/TradeGravity/?tab=semiconductors' "${RUNNER_TEMP}/distribution-kit/carousel/caption.md"
test -f "${RUNNER_TEMP}/distribution-kit/manifest.json"
grep -q '"theme": "intelligence-dark"' "${RUNNER_TEMP}/distribution-kit/manifest.json"
grep -q '"caption_path": "carousel/caption.md"' "${RUNNER_TEMP}/distribution-kit/manifest.json"
go run ./cmd/distribution-approval \
-kit "${RUNNER_TEMP}/distribution-kit" \
-reviewer ci-synthetic-reviewer \
-audience synthetic-internal-pilot \
-channels email,instagram \
-approved-at 2026-07-17T12:00:00Z \
-attest-reviewed
test -f "${RUNNER_TEMP}/distribution-kit/approval.json"
grep -q '"provider_delivery_ready": false' "${RUNNER_TEMP}/distribution-kit/approval.json"
go run ./cmd/instagram-preflight \
-kit "${RUNNER_TEMP}/distribution-kit" \
-out "${RUNNER_TEMP}/instagram-preflight.json" \
-generated-at 2026-07-17T13:00:00Z
grep -q '"manual_upload_required": true' "${RUNNER_TEMP}/instagram-preflight.json"
grep -q '"automatic_publish_authorized": false' "${RUNNER_TEMP}/instagram-preflight.json"
grep -q '"contains_credentials": false' "${RUNNER_TEMP}/instagram-preflight.json"

- name: Check browser JavaScript
run: node --check site/app.js && node --check site/security.js && node --check site/data-tools.js && node --check site/explorer-tools.js && node --check site/intelligence-tools.js && node --check site/semiconductor-tools.js && node --check site/experience-tools.js && node --check site/news-tools.js
run: node --check site/app.js && node --check site/security.js && node --check site/data-tools.js && node --check site/explorer-tools.js && node --check site/intelligence-tools.js && node --check site/semiconductor-tools.js && node --check site/experience-tools.js && node --check site/news-tools.js && node --check site/briefing-tools.js

- name: Test browser data helpers
run: node --test site/security.test.cjs site/data-tools.test.cjs site/explorer-tools.test.cjs site/intelligence-tools.test.cjs site/semiconductor-tools.test.cjs site/experience-tools.test.cjs site/news-tools.test.cjs site/structure.test.cjs
run: node --test site/security.test.cjs site/data-tools.test.cjs site/explorer-tools.test.cjs site/intelligence-tools.test.cjs site/semiconductor-tools.test.cjs site/experience-tools.test.cjs site/news-tools.test.cjs site/briefing-tools.test.cjs site/structure.test.cjs

- name: Validate citation metadata
run: |
Expand Down
10 changes: 10 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,13 @@
/site/data/

tradegravity.db

/private/
/distribution-kit/
/delivery-preflight.json
/instagram-preflight.json
/email-launch-authorization.json
subscriptions.db
subscriptions.db-*
delivery-ledger.db
delivery-ledger.db-*
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,16 @@ All notable changes to TradeGravity will be documented in this file. The project
- Global PNG snapshot, CSV, and Markdown summary-report exports for the active analysis view.
- Always-visible metric, observation-period, scope, and limitation context plus a definitions dialog.
- Current, partial, and degraded publication-health states with recovery guidance.
- A validated `briefing.json` contract that derives three cited semiconductor observations and exposes review-gated email Markdown and 4:5 social-carousel copy without collecting subscribers or publishing automatically.
- An offline distribution-kit CLI and manual read-only Actions workflow that render one-primary-CTA email HTML with an unsubscribe placeholder, six matched 1080×1350 SVG originals and PNG upload assets, alt text, approval gates, and deterministic file hashes without sending or posting.
- A second original `editorial-light` native carousel theme behind the same validated renderer interface; theme choice is recorded in the manifest and therefore bound to editorial approval, with no browser runtime, remote font, image fetch, or arbitrary HTML input.
- Automated palette contrast checks across every gradient stop, including normal text and large bold role labels; the light theme's muted and accent colors were tightened to retain the documented floor.
- A review-pending Instagram `caption.md` derived from the same three validated signals, with comparison period, evidence link, conservative scope note, restrained tags, an editorial length ceiling, and manifest/approval tamper protection.
- An Instagram manual-publish preflight CLI that requires channel approval, verifies every PNG plus caption and alt-text contracts, emits content-free aggregate evidence outside the kit, and explicitly carries no credentials or publish authorization.
- A content-release approval CLI that rejects changed, missing, or untracked kit files and binds the verified manifest to a reviewer, audience label, time, and email/Instagram channel set without claiming delivery readiness.
- A local email preflight CLI that validates double opt-in, audience identity, suppression precedence, timestamps, duplicate addresses, unique opaque HTTPS unsubscribe URLs, and a pilot ceiling while emitting an aggregate plan with no recipient addresses or tokens and no delivery authorization.
- A private SQLite subscription-registry CLI and subscription HTTP service with a default-off double-opt-in signup form, short-lived purpose-separated HMAC confirmation links, read-only scanner-safe GET plus explicit confirmation POST, stable Resend confirmation idempotency, form-encoded one-click unsubscribe, signed raw-body provider feedback, durable global suppression, security headers, and private preflight exports.
- A short-lived email launch-approval contract and Resend pilot CLI that replay consent/suppression checks at send time, render recipient-specific visible and RFC one-click unsubscribe links, use one recipient and one provider idempotency key per request, and prevent automatic duplicate or uncertain retries with a private HMAC-keyed SQLite delivery ledger. A separate reconciliation CLI records provider-confirmed acceptance or non-acceptance without storing recipient PII; only the latter plus a different launch-authorization digest permits retry.

### Changed

Expand Down
17 changes: 15 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,11 @@ TradeGravity is an open-source pipeline and static intelligence dashboard for un
- **Live demo:** https://elecpapaya.github.io/TradeGravity/
- **System design:** [DESIGN.md](DESIGN.md)
- **Published data schema:** [docs/DATA_SCHEMA.md](docs/DATA_SCHEMA.md)
- **Reviewed distribution workflow:** [docs/DISTRIBUTION.md](docs/DISTRIBUTION.md)
- **Email consent and suppression preflight:** [docs/EMAIL_DELIVERY_PREFLIGHT.md](docs/EMAIL_DELIVERY_PREFLIGHT.md)
- **Provider-backed email pilot:** [docs/EMAIL_PROVIDER_PILOT.md](docs/EMAIL_PROVIDER_PILOT.md)
- **Private registry and unsubscribe service:** [docs/UNSUBSCRIBE_SERVICE.md](docs/UNSUBSCRIBE_SERVICE.md)
- **Instagram manual-publish preflight:** [docs/INSTAGRAM_PREFLIGHT.md](docs/INSTAGRAM_PREFLIGHT.md)
- **Semiconductor atlas methodology:** [docs/SEMICONDUCTOR_ATLAS.md](docs/SEMICONDUCTOR_ATLAS.md)
- **Reuse examples:** [docs/USAGE.md](docs/USAGE.md)
- **Data rights and attribution:** [docs/DATA_RIGHTS.md](docs/DATA_RIGHTS.md)
Expand Down Expand Up @@ -70,6 +75,13 @@ The pipeline refresh timestamp indicates when TradeGravity generated the site; i
- HS2 product mix for the selected reporter, kept separate from WITS headline totals.
- Shareable Overview, US–China Lens, Chip Lens, Products, Data & Quality, and Scenario Lab tabs with synchronized filters, country, semiconductor stage/context, product, tariff, and scenario-assumption state.
- A semiconductor Pulse that separates latest month-to-month movement from publish-to-publish coverage and value revisions, with a machine-readable bounded change feed.
- A deterministic `briefing.json` distribution draft with three cited semiconductor observations, review-gated email Markdown, and review-gated 4:5 carousel copy. The static site does not collect subscribers, send email, or publish to social platforms.
- An offline `cmd/distributor` build that turns a ready briefing into email HTML/Markdown, a cited Instagram caption, alt text, and six matched 1080×1350 SVG/PNG cards in either `intelligence-dark` or `editorial-light`; all assets share review gates and deterministic hashes without making a network request.
- An aggregate-only `cmd/instagram-preflight` that requires an unchanged Instagram approval, decodes all six PNGs, validates caption evidence/scope/tags and six alt-text sections, refuses output inside the kit, and keeps credentials and automatic publishing explicitly false.
- A fail-closed `cmd/distribution-approval` step that verifies the complete file set and SHA-256 manifest before recording a channel-specific content approval; provider delivery, subscriber consent, and automatic publishing remain explicitly false.
- A local `cmd/distribution-preflight` gate that validates private double-opt-in and suppression CSVs, approved-audience identity, and unique opaque HTTPS unsubscribe URLs, enforces a pilot ceiling, and writes only aggregate counts and digests—never recipient addresses or tokens—while keeping provider configuration and delivery authorization false.
- A separate SQLite `cmd/subscription-registry` and bounded `cmd/unsubscribe-service` that can collect double-opt-in consent outside the static dashboard, send short-lived Resend confirmations, activate only on explicit confirmation POST, issue HMAC-authenticated links without email/audience claims, keep link-scanner GETs read-only, record idempotent RFC one-click suppressions, verify signed Resend feedback, and export private preflight inputs.
- A fail-closed Resend pilot path that binds a one-hour launch approval to the exact aggregate preflight, sender, audience, and content/input digests; reruns the consent and suppression checks immediately before delivery; adds visible and header one-click unsubscribe links; isolates every recipient in a separate provider request; and records only HMAC recipient keys in a private SQLite ledger. Accepted or uncertain attempts are never sent again automatically; provider-confirmed non-acceptance still requires a recorded reconciliation and a new launch approval.
- Two-anchor position metrics whose formulas are visible: USA share, China share, exposure balance, position shift, dual exposure, and anchor-growth divergence.
- Unadjusted bilateral mirror-reporting diagnostics that compare both countries' reports without choosing either as ground truth or treating the difference as fraud, evasion, rerouting, or an adjusted estimate.
- An illustrative HS6 tariff sensitivity lab that can load a published MFN rate and product import baseline while exposing elasticity, pass-through, fallback, and source assumptions.
Expand Down Expand Up @@ -141,6 +153,7 @@ The public deployment exposes stable machine-readable endpoints:
- `https://elecpapaya.github.io/TradeGravity/data/semiconductors/reference.json`
- `https://elecpapaya.github.io/TradeGravity/data/semiconductors/monthly/index.json`
- `https://elecpapaya.github.io/TradeGravity/data/changes.json`
- `https://elecpapaya.github.io/TradeGravity/data/briefing.json`
- `https://elecpapaya.github.io/TradeGravity/data/tariffs/index.json`
- `https://elecpapaya.github.io/TradeGravity/data/bilateral-matrix/index.json`
- `https://elecpapaya.github.io/TradeGravity/data/mirror/index.json`
Expand Down Expand Up @@ -276,13 +289,13 @@ This repository reads operating-system environment variables and does not load a
## Generated files and deployment

- Local SQLite database: `tradegravity.db`
- Published JSON: `meta.json`, `catalog.json`, `changes.json`, `latest.json`, `series.json`, `quality.json`, `context.json`, `products/`, `strategic-hs6/`, `semiconductors/reference.json`, `semiconductors/monthly/`, `tariffs/`, `bilateral-matrix/`, `mirror/`, and `explanations/` under `site/data/`
- Published JSON: `meta.json`, `catalog.json`, `changes.json`, `briefing.json`, `latest.json`, `series.json`, `quality.json`, `context.json`, `products/`, `strategic-hs6/`, `semiconductors/reference.json`, `semiconductors/monthly/`, `tariffs/`, `bilateral-matrix/`, `mirror/`, and `explanations/` under `site/data/`

Generated data and the local database are intentionally not committed to the default branch. The scheduled or manually dispatched core workflow runs the broad collectors and saves its validated database as a three-day Actions artifact. The staggered semiconductor workflow restores that artifact and the previous `gh-pages` publication, adds annual and monthly chip observations for [`configs/chip_connectors.csv`](configs/chip_connectors.csv), emits a validated publish-to-publish `changes.json`, and deploys `site/` to the `gh-pages` branch. A `main` push uses the latest validated `data/` directory from `gh-pages` and redeploys the site without calling WITS, UN Comtrade, WITS/TRAINS, or World Bank APIs. This keeps code-only deployments fast while the weekly refresh remains the source of new published observations.

The fast deployment intentionally fails if `gh-pages` does not contain `data/latest.json` and `data/meta.json`. Bootstrap or repair the published dataset by manually running **Update TradeGravity core**, then **Update TradeGravity semiconductor**; the second workflow waits out any remaining quota window before it publishes.

Before deployment, `cmd/validator` checks provenance across every artifact, reporter uniqueness, periods, non-negative finite values, totals and shares, matrix availability/count identities, tariff rate identities, product keys, bounded publication-change arithmetic and ordering, context coverage, and explanation evidence references.
Before deployment, `cmd/validator` checks provenance across every artifact, reporter uniqueness, periods, non-negative finite values, totals and shares, matrix availability/count identities, tariff rate identities, product keys, bounded publication-change arithmetic and ordering, briefing arithmetic and mandatory human-review gates, context coverage, and explanation evidence references.

## Maintenance and contributing

Expand Down
Loading