Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions docs/ARCHIVAL-ESCROW.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# Archival Legal-Escrow Arrangement

Joblantern places the annual-archive custody under a legal-escrow
arrangement so the archives outlive the operating entity.

## Parties

- **Grantor**: Joblantern Foundation.
- **Escrow agent**: a neutral, jurisdictionally diverse trust.
- **Beneficiary**: the public, with a named successor entity for operational custody.

## Triggers for custody transfer

- Dissolution of the Joblantern Foundation.
- A material breach of the [council charter](COUNCIL-CHARTER.md).
- A finding by an independent auditor that the Foundation can no
longer maintain the archive.

## Materials held in escrow

- Copies of every annual archive.
- The DataCite DOI prefix authorisation.
- Cryptographic signing keys used for archive integrity.
- Contact lists for the published mirror destinations.

## Successor entity

Named in a sealed annex updated annually. The annex names a list of
fallbacks; the agent opens them in order until one accepts.

## Independence from funders

The escrow agent is independent of any current or recent Joblantern
funder. See [FUNDING](FUNDING.md).
28 changes: 28 additions & 0 deletions docs/ARCHIVAL-RETENTION.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Archival Retention Policy

Joblantern annual archives are intended to survive organisational
change. The policy is binding on any successor entity that takes
over Joblantern's stewardship.

## Retention horizon

- Annual archives: indefinite.
- Monthly deltas: 7 years.
- Operational logs: per [COMPLIANCE](COMPLIANCE/README.md).

## Survivability commitments

- Mirror to at least two independent third-party archives (Internet Archive + Zenodo).
- Maintain a DataCite DOI for each annual archive.
- Hold a legal-escrow arrangement so archives outlive any single corporate form.

## Format obsolescence

Parquet and JSON-LD will be re-encoded into the then-current durable
formats every 15 years. Each re-encoding is itself archived under a
new sub-DOI; the original encoding stays available.

## What happens if Joblantern.org sunsets

The successor entity nominated in the [escrow agreement](ARCHIVAL-ESCROW.md)
takes custody of the archive copies and the DOI prefix.
36 changes: 36 additions & 0 deletions docs/ARCHIVAL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# Long-Term Archival & Historical Record

Recruitment-fraud history is itself an important historical record.
Joblantern publishes annual archives so a researcher in 2045 can
cite a 2026 verdict with confidence.

## Annual archives

- Format: Parquet + JSON-LD ([format](../internal/archival/format/parquet.go)).
- Integrity: SHA-256 per file plus a manifest hash ([integrity](../internal/archival/integrity/hash.go)).
- DOI: assigned through DataCite ([doi](../internal/archival/doi/datacite.go)).
- Landing: BibTeX-shipping HTML page ([landing](../internal/archival/landing/page.go)).
- Mirrors: Internet Archive + Zenodo ([thirdparty](../internal/archival/thirdparty/mirror.go)).

## Anonymisation audit

Every archive runs through [anonaudit](../internal/archival/anonaudit/audit.go)
before publication. Fatal checks block release.

## Time machine

The [time machine](../internal/archival/timemachine/asof.go) lets a
user reconstruct the state of a verdict as of any past date, using
the matching annual archive plus monthly deltas.

## Survivability

- [ARCHIVAL-RETENTION](ARCHIVAL-RETENTION.md)
- [ARCHIVAL-ESCROW](ARCHIVAL-ESCROW.md)

## Citing the archive

```
Joblantern Annual Verdict Archive, 2026,
doi:10.<prefix>/joblantern.2026.
```
32 changes: 32 additions & 0 deletions internal/archival/anonaudit/audit.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
// Package anonaudit reviews each archive for re-identification risk
// before publication.
package anonaudit

// Check is one audit row.
type Check struct {
Name string
Severity string // "fatal" | "warning" | "info"
Passed bool
Notes string
}

// Default returns the canonical battery.
func Default() []Check {
return []Check{
{Name: "k-anonymity-k>=5", Severity: "fatal"},
{Name: "no-raw-contacts", Severity: "fatal"},
{Name: "no-raw-cvs", Severity: "fatal"},
{Name: "stable-pseudonyms-rotated", Severity: "warning"},
{Name: "geo-rounding-tile", Severity: "warning"},
}
}

// Blocking returns true if any fatal check failed.
func Blocking(checks []Check) bool {
for _, c := range checks {
if c.Severity == "fatal" && !c.Passed {
return true
}
}
return false
}
26 changes: 26 additions & 0 deletions internal/archival/doi/datacite.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
// Package doi mints DOIs for each annual archive via DataCite.
package doi

import "fmt"

// Mint is the registered DOI plus the metadata sent to DataCite.
type Mint struct {
DOI string // "10.<prefix>/joblantern.<year>"
Year int
URL string
Title string
Creator string
Publisher string
}

// Build constructs a Mint with a deterministic suffix.
func Build(prefix string, year int, url string) Mint {
return Mint{
DOI: fmt.Sprintf("%s/joblantern.%d", prefix, year),
Year: year,
URL: url,
Title: fmt.Sprintf("Joblantern Annual Verdict Archive, %d", year),
Creator: "Joblantern Foundation",
Publisher: "Joblantern Foundation",
}
}
22 changes: 22 additions & 0 deletions internal/archival/format/parquet.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
// Package format declares the archival file formats. We pick formats
// that survive: Parquet for the columnar data, JSON-LD for the
// schema and the linked-data sidecar.
package format

// File is one archive file specification.
type File struct {
Name string
Format string // "parquet" | "json-ld" | "csv" | "manifest"
Required bool
}

// Defaults returns the canonical file set per archive.
func Defaults() []File {
return []File{
{Name: "verdicts.parquet", Format: "parquet", Required: true},
{Name: "evidence.parquet", Format: "parquet", Required: true},
{Name: "schema.json-ld", Format: "json-ld", Required: true},
{Name: "manifest.json", Format: "manifest", Required: true},
{Name: "README.md", Format: "markdown", Required: true},
}
}
31 changes: 31 additions & 0 deletions internal/archival/integrity/hash.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
// Package integrity computes per-file and per-archive integrity
// hashes.
package integrity

import (
"crypto/sha256"
"encoding/hex"
"io"
)

// Manifest pairs file name with its hash.
type Manifest struct {
Files []FileEntry
}

// FileEntry is one entry in a manifest.
type FileEntry struct {
Name string
Bytes int64
SHA256 string
}

// HashReader streams through r and returns the hex digest plus byte count.
func HashReader(r io.Reader) (string, int64, error) {
h := sha256.New()
n, err := io.Copy(h, r)
if err != nil {
return "", 0, err
}
return hex.EncodeToString(h.Sum(nil)), n, nil
}
30 changes: 30 additions & 0 deletions internal/archival/integrity/hash_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
package integrity

import (
"strings"
"testing"
)

func TestHashReaderDeterministic(t *testing.T) {
a, na, err := HashReader(strings.NewReader("joblantern-2026-archive"))
if err != nil {
t.Fatal(err)
}
b, nb, err := HashReader(strings.NewReader("joblantern-2026-archive"))
if err != nil {
t.Fatal(err)
}
if a != b || na != nb {
t.Fatalf("hash not deterministic: %s != %s", a, b)
}
}

func TestHashReproducibilityAcrossRuns(t *testing.T) {
// Surface a stable expectation so any change to format goes
// through an explicit decision.
got, _, _ := HashReader(strings.NewReader(""))
const want = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
if got != want {
t.Fatalf("empty hash drifted: got %s want %s", got, want)
}
}
42 changes: 42 additions & 0 deletions internal/archival/landing/page.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
// Package landing renders the citable archive landing page. Each
// archive has a stable URL that returns either HTML for browsers or
// BibTeX for citation managers.
package landing

import "fmt"

// Citation is the metadata that drives the landing page.
type Citation struct {
DOI string
Year int
URL string
Title string
Bytes int64
Records int
SHA256 string
}

// BibTeX returns the BibTeX entry.
func (c Citation) BibTeX() string {
return fmt.Sprintf(`@misc{joblantern%d,
title = {%s},
author = {Joblantern Foundation},
year = {%d},
publisher = {Joblantern Foundation},
doi = {%s},
url = {%s}
}`, c.Year, c.Title, c.Year, c.DOI, c.URL)
}

// HTML returns the landing page body (no external assets).
func (c Citation) HTML() string {
return fmt.Sprintf(`<!doctype html><html><head><meta charset="utf-8"><title>%s</title></head>
<body>
<h1>%s</h1>
<p>DOI: <code>%s</code></p>
<p>SHA-256: <code>%s</code></p>
<p>Records: %d. Bytes: %d.</p>
<pre>%s</pre>
<p><a href="%s">Download</a></p>
</body></html>`, c.Title, c.Title, c.DOI, c.SHA256, c.Records, c.Bytes, c.BibTeX(), c.URL)
}
22 changes: 22 additions & 0 deletions internal/archival/snapshot/snapshot.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
// Package snapshot exports annual archive snapshots to long-term
// storage. Snapshots are immutable, content-addressed, and
// integrity-hashed.
package snapshot

import "time"

// Spec drives a snapshot export.
type Spec struct {
Year int
Region string
DestBucket string
StoreClass string // "cold", "deep-archive", "tape"
}

// Result is the produced artifact.
type Result struct {
URI string
SHA256 string
Bytes int64
Snapshotted time.Time
}
19 changes: 19 additions & 0 deletions internal/archival/thirdparty/mirror.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
// Package thirdparty mirrors annual archives to durable third-party
// repositories — the Internet Archive and Zenodo — so the data
// survives even if Joblantern.org sunsets.
package thirdparty

// Target is one mirror destination.
type Target struct {
ID string
Name string
URL string
}

// Defaults captures the seed list.
func Defaults() []Target {
return []Target{
{ID: "ia", Name: "Internet Archive", URL: "https://archive.org"},
{ID: "zenodo", Name: "Zenodo (CERN)", URL: "https://zenodo.org"},
}
}
60 changes: 60 additions & 0 deletions internal/archival/timemachine/asof.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
// Package timemachine serves the "view verdicts as of date D" UI.
// The implementation pulls from the matching annual archive and the
// monthly snapshots that follow it.
package timemachine

import "time"

// Query is the user-facing input.
type Query struct {
AsOf time.Time
Limit int
}

// Resolver maps an AsOf date to the storage paths that cover it.
type Resolver struct {
AnnualArchives map[int]string // year -> archive URL
MonthlyDeltas map[string]string // "2026-05" -> delta URL
}

// Locate returns the archive URLs needed to reconstruct state at AsOf.
func (r Resolver) Locate(q Query) []string {
if q.AsOf.IsZero() {
return nil
}
year := q.AsOf.Year()
out := []string{}
if a, ok := r.AnnualArchives[year-1]; ok {
out = append(out, a)
}
for m := time.January; m <= q.AsOf.Month(); m++ {
key := monthKey(year, m)
if d, ok := r.MonthlyDeltas[key]; ok {
out = append(out, d)
}
}
return out
}

func monthKey(year int, m time.Month) string {
mm := int(m)
mmStr := "0" + string('0'+byte(mm))
if mm >= 10 {
mmStr = string('0'+byte(mm/10)) + string('0'+byte(mm%10))
}
return itoa(year) + "-" + mmStr
}

func itoa(n int) string {
if n == 0 {
return "0"
}
var b [12]byte
i := len(b)
for n > 0 {
i--
b[i] = byte('0' + n%10)
n /= 10
}
return string(b[i:])
}
Loading