Public-Inbox-Stack

Scripts and containers for mirroring and hosting public-inbox mailing list archivers. Batteries included, all in containers.

Supports mirroring only, hosting only, or both.

Complete Architecture

Upstream (public-inbox.example.org, etc.)
        │
        │  grok-pull (git clone/fetch)
        ▼
   ┌───────────┐
   │grokmirror │  profiles: mirroring
   └────┬──────┘
        │  /data/ (git repos)
        ▼
   ┌─────────┐     ┌───────────┐
   │ indexer │ or  │ grok-pi-  │  profiles: manual / indexed hooks
   └────┬────┘     └─────┬─────┘
        │                │
        ▼                ▼
   ┌─────────────────────────┐
   │  public-inbox (httpd,   │  profiles: hosting
   │  nntpd, watch, extindex)│
   └────────────┬────────────┘
                │  port 8080
                ▼
        ┌────────────────┐
        │  Angie/nginx   │  profiles: hosting
        │  (proxy+SSL)   │
        └────────────────┘
                │
                ▼  http://your-host:8088

Requirements

podman + podman-compose >= 1.1.0 or
docker + docker-compose >= 1.28.0 (v2 plugin preferred)

Runtime is auto-detected. See Container Runtime for details.

Quick Start

This repository relies on git submodules. It must be clone recursively, or run git submodule update --init --recursive after clonning.

# 1. Create your environment configuration
cp example.env .env

# 2. Edit .env with your values (see Configuration section)
vim .env

# 3. Generate configuration files from templates
make setup

# 4. Start mirroring (clones repos from upstream)
make run-mirroring

# 5. Index the cloned repos
make run-indexer

# 6. Start hosting (public-inbox + nginx)
make run-hosting

Visit http://localhost:8088 (or your configured port) to see your instance.

Or run everything in one command:

make run-all

Configuration

How Configuration Works

All configuration flows from a single .env file:

Copy example.env to .env and edit your values
Run make setup — this executes scripts/setup.sh which:
- Reads .env and validates required variables
- Processes templates in config_template/ using two mechanisms:
  - Conditional blocks: {{#VAR}}...{{/VAR}} — kept if VAR=true, removed entirely if VAR=false
  - Variable substitution: {{VAR}} — replaced with the value from .env
- Writes rendered configs to configs/ (git-ignored, regenerated each time)
Mount — compose.yaml mounts configs/ into the containers at runtime

.env  ──setup.sh──►  config_template/  ──►  configs/
(user edit)        (source templates)     (rendered output, mounted into containers)

Environment Variables

Edit .env to configure your instance. All variables are documented below:

Mirroring

Variable	Description	Default	Example
`MIRROR_UPSTREAM_HOST`	Grokmirror source URL (where you mirror from)	(required)	`https://public-inbox.example.org`
`GROKMIRROR_MODE`	Mirroring mode: `clone` (just git) or `indexed` (git + auto-indexing via hooks)	`clone`	`clone`
`GROKMIRROR_EXTINDEX_ENABLED`	In indexed mode, run cross-inbox extindex after each pull cycle	`true`	`false`

Hosting

Variable	Description	Default	Example
`SERVE_HOST`	Domain where your instance will be accessible	`localhost`	`lists.example.com`
`SERVE_HTTP_PORT`	HTTP port exposed on the host	`80`	`8088`
`SERVE_HTTPS_PORT`	HTTPS port exposed on the host	`443`	`8443`
`SERVE_NNTP_PORT`	NNTP port exposed on the host	`119`	`1119`
`PI_HTTP_ENABLE`	Enable the public-inbox HTTP daemon	`true`	`true`
`PI_NNTP_ENABLE`	Enable the public-inbox NNTP daemon	`false`	`true`
`PI_INDEXING_ENABLE`	Run extindex on container startup	`false`	`true`
`PI_INDEX_LEVEL`	Xapian index level: `basic` (no full-text search), `medium` (no phrase search), `full` (default)	`full`	`medium`
`EXTINDEX_DESCRIPTION`	Description text shown on the extindex main page header	`All Lore`	`My Lore Mirror`
`ANUBIS_ENABLED`	Enable Anubis proof-of-work bot protection	`false`	`true`
`ANUBIS_PORT`	Port Anubis binds to (proxies to nginx)	`3000`	`3000`

IMAP Watch (for hosting your own lists)

Variable	Description	Default	Example
`PI_IMAP_ENABLED`	Enable IMAP watcher to ingest mail from an IMAP mailbox	`false`	`true`
`PI_IMAP_LIST_NAME`	Inbox name/identifier	`list.name`	`my-list`
`PI_IMAP_LIST_ADDRESS`	Email address for the list	`name@lists.domain.tld`	`discuss@lists.example.com`
`PI_IMAP_AUTH_URL`	IMAP connection string with credentials	(none)	`imaps://user:pass@imap.server.tld:993/INBOX`

Spam Checking

Variable	Description	Default	Example
`SPAMCHECK_ENABLED`	Enable SpamAssassin spam checking via `spamc`	`false`	`true`

SSL/TLS

Variable	Description	Default	Example
`ACME_ENABLED`	Enable automatic SSL via Let's Encrypt (ACME)	`false`	`true`
`ACME_EMAIL`	Email for ACME certificate registration	`admin@example.com`	`admin@example.com`

Understanding MIRROR_UPSTREAM_HOST vs SERVE_HOST

MIRROR_UPSTREAM_HOST: The source you're cloning from. This is the grokmirror manifest host (e.g., https://public-inbox.example.org). Grokmirror uses this to fetch the manifest and git repos.
SERVE_HOST: The domain where your instance will be accessible. Used in nginx server_name, public-inbox URLs, and the extindex configuration.

Grokmirror Modes

Clone Mode (`GROKMIRROR_MODE=clone`)

Pure git cloning/fetching. No indexing happens during mirroring. You must run indexing separately with make run-indexer.

Use when: You want full control over when indexing happens, or you're only interested in the git repos.

Indexed Mode (`GROKMIRROR_MODE=indexed`)

Clones repos AND automatically indexes them via grok-pi-indexer hooks:

post_update_hook — runs after each pull to update the index
post_clone_complete_hook — runs after cloning a new repo to initialize it
post_work_complete_hook — runs after all work is done to update the extindex

Use when: You want a set-and-forget mirror that stays indexed automatically.

Conditional Configuration

The template system supports conditional blocks. Setting a variable to true in .env enables the corresponding feature:

Variable	What it enables
`ACME_ENABLED=true`	HTTPS server block with Let's Encrypt certificates in nginx
`ANUBIS_ENABLED=true`	Anubis PoW bot protection integrated into nginx
`PI_IMAP_ENABLED=true`	IMAP inbox definition in public-inbox config
`SPAMCHECK_ENABLED=true`	SpamAssassin integration via `spamc` in public-inbox-watch

Makefile Targets

Setup

Target	Description
`make setup`	Generate configs from templates (reads `.env`, writes to `configs/`)
`make setup-dry-run`	Preview what would be generated without writing files

Running Services

Target	Description
`make run-hosting`	Start public-inbox + nginx (hosting profile)
`make watch-hosting`	Same as `run-hosting` but in foreground (see logs in terminal)
`make run-mirroring`	Start grokmirror daemon (mirroring profile, detached, pulls every 2 hours)
`make pull-mirror`	Run grokmirror once, pull and exit (mirroring profile)
`make run-mirroring-indexed`	Start grokmirror in indexed mode (auto-indexing enabled)
`make run-indexer`	One-shot manual indexing of cloned repos (manual profile)
`make purge-indexing`	Purge public-inbox indexing data (preserves grokmirror git clones)
`make run-all`	Full pipeline: `setup` → `run-mirroring` → `run-hosting`

Utilities

Target	Description
`make logs`	Follow logs for all services
`make logs-hosting`	Follow logs for hosting services only
`make logs-mirroring`	Follow logs for mirroring services only
`make obfuscate`	Add `obfuscate = true` to all inbox configs missing it
`make stop`	Stop all services
`make stop-hosting`	Stop hosting services only
`make stop-mirroring`	Stop mirroring services only
`make clean`	Remove generated `build/` directory
`make help`	Show all available targets

Container Runtime

The project auto-detects your container runtime with this priority:

podman + podman-compose
docker compose (v2 plugin)
docker-compose (v1 standalone)

Override the detected runtime:

# Use nerdctl
make run-hosting CONTAINER=nerdctl COMPOSE="nerdctl compose"

# Force docker-compose v1
make run-hosting COMPOSE=docker-compose

When running with sudo, the real user/group ID is preserved so files in data/ are owned by your user, not root.

Compose Profiles

Profile	Services	Description
`hosting`	`public-inbox`, `nginx`	Serves the web interface, NNTP, and HTTP
`mirroring`	`grokmirror`	Clones and fetches repos from upstream
`manual`	`indexer`	One-shot indexing service (run with `run --rm`)
`anubis`	`anubis`	Proof-of-work bot protection (auto-enabled with `ANUBIS_ENABLED=true`)

Workflows

Clone Mode (Manual Indexing)

make setup
make run-mirroring        # starts daemon, clones/fetches repos every 2 hours in background
# or
make pull-mirror          # one-shot: clones/fetches repos once and exits
make run-indexer          # initializes and indexes all cloned repos
make run-hosting          # serves the indexed repos

Re-run make pull-mirror followed by make run-indexer periodically to fetch new messages and update the index.

Indexed Mode (Automatic Indexing)

make setup
make run-mirroring-indexed  # clones AND indexes automatically
make run-hosting            # serves the indexed repos

No manual indexing needed — hooks handle it after every pull.

Hosting Only (No Mirroring)

If you have your own mailing lists (not mirrored):

Add inbox definitions to config_template/pi-configs/config.template
Place your git repos in data/<inbox-name>/git/
make setup && make run-hosting

Adding Your Own Mailing List via IMAP

Set PI_IMAP_ENABLED=true in .env
Configure PI_IMAP_LIST_NAME, PI_IMAP_LIST_ADDRESS, and PI_IMAP_AUTH_URL
make setup && make run-hosting

The public-inbox-watch daemon will poll the IMAP mailbox and ingest new messages.

Pinning Inboxes to the Top

To control the order inboxes appear on the main listing page, add a sortorder key to the inbox config. Lower values appear first:

[publicinbox "my-important-list"]
	address = "important@lists.example.com"
	url = https://lists.example.com/my-important-list
	inboxdir = /data/my-important-list
	sortorder = 1

[publicinbox "another-list"]
	address = "another@lists.example.com"
	url = https://lists.example.com/another-list
	inboxdir = /data/another-list
	sortorder = 2

Inboxes without sortorder appear below all sorted inboxes, ordered by last modification time (newest first).

Services Detail

grokmirror

Runs grok-pull to clone/fetch git repos from an upstream grokmirror manifest. Configuration is generated from config_template/grokmirror/clone.conf.template or indexed.conf.template depending on GROKMIRROR_MODE.

indexer

Runs scripts/index-cloned-repos.sh to scan /data/ for cloned v2 inboxes, initialize them with public-inbox-init -V2, index with public-inbox-index, and update the external index with public-inbox-extindex --all.

public-inbox

Runs scripts/start-public-inbox.sh which starts the appropriate daemons based on .env flags:

public-inbox-httpd — HTTP/PSGI web interface
public-inbox-nntpd — NNTP server
public-inbox-watch — IMAP watcher (when PI_IMAP_ENABLED=true)
spamd — SpamAssassin daemon (when SPAMCHECK_ENABLED=true)
public-inbox-extindex — External indexing (when PI_INDEXING_ENABLE=true)

If the data directory is empty on startup, reinit-from-config.sh runs automatically to initialize inboxes from the public-inbox config file.

nginx (Angie)

Angie — an nginx fork with built-in ACME support. Proxies HTTP/HTTPS to public-inbox:8080 and streams NNTP (TCP) to public-inbox:119.

Anubis

When ANUBIS_ENABLED=true, Anubis is integrated with the Nginx proxy to protect against aggressive bots. It requires visitors to solve a proof-of-work challenge before accessing the site. Traffic will reach Nginx first (for TLS termination), then Anubis listens on ANUBIS_PORT (should be private), and proxies back to Nginx via a Unix socket.

SSL/TLS Configuration

To enable automatic SSL with Let's Encrypt:

Set ACME_ENABLED=true and ACME_EMAIL=your@email.com in .env
Ensure SERVE_HOST is a real domain that resolves to your server
Run make setup to regenerate nginx config with ACME blocks
Start hosting: make run-hosting

Angie will automatically obtain and renew certificates. The acme/ directory on the host stores certificate state.

Email Obfuscation

To hide email addresses from scrapers in the web UI, run:

make obfuscate

This adds obfuscate = true to every [publicinbox "..."] section in configs/pi-configs/config that doesn't already have it.

Note: Running make setup overwrites configs/pi-configs/config. The script creates a backup (config.bak) and reminds you to re-run make obfuscate.

Troubleshooting

Config generation fails

[ERROR] .env not found

Run cp example.env .env first, then edit .env with your values.

No v2 inboxes found

The indexer scans /data/ for directories with git/N.git structure. If grokmirror hasn't finished cloning yet, run make run-mirroring first and wait for it to complete.

Indexing is slow or hangs

Large inboxes (10,000+ messages) may trigger a hint about --split-shards. This flag is not recommended in containerized environments — it requires 2-3x temporary disk space and is prone to Xapian database corruption on interruption. The script does not use it by default.

Container permission errors

When using sudo make, the real user ID should be preserved. If you see permission errors in data/, check ownership:

ls -la data/

Fix with: sudo chown -R $(id -u):$(id -g) data/

Reinitializing from scratch

If your data directory is corrupted or you want to start fresh:

make stop
rm -rf data/*
make setup          # regenerates configs
make run-hosting    # auto-runs reinit-from-config.sh on empty data dir
make run-indexer    # re-indexes all repos

Purging Indexing Data

To remove public-inbox indexing artifacts (Xapian, msgmap, over.sqlite3, all.git, external index) while preserving grokmirror git clones:

make purge-indexing
# or with dry-run to preview:
./scripts/purge-indexing.sh -d /data -n

Use this when you want to re-index from scratch without re-cloning all repos from upstream. After purging, run make run-indexer to rebuild the indexes.

Note: This also removes the external index (/data/all/) which may be corrupted if you see dir_end invalid errors during public-inbox-extindex.

Viewing generated configs

Before applying, preview what make setup will generate:

make setup-dry-run

Or inspect the templates directly in config_template/ to understand what each variable controls.

Name		Name	Last commit message	Last commit date
Latest commit History 65 Commits
config_template		config_template
configs		configs
grokmirror @ 41daf7b		grokmirror @ 41daf7b
public-inbox @ 7c2cdb6		public-inbox @ 7c2cdb6
scripts		scripts
.gitignore		.gitignore
.gitmodules		.gitmodules
Containerfile		Containerfile
Makefile		Makefile
README.md		README.md
compose.yaml		compose.yaml
containers.mk		containers.mk
example.env		example.env

Uh oh!

Folders and files

Latest commit

History

Repository files navigation