Distributed Trading Post

A distributed marketplace where buyers and sellers trade through trader services over TCP. Built for UMass COMPSCI 677 (Distributed and Operating Systems) across three successive phases — from flood-routed P2P trades, to a single elected coordinator, to replicated traders backed by a shared warehouse.

Each component runs as its own OS process and talks only over the network, so concurrency, failures, and consistency show up the way they do in real multi-service systems.

What each phase covers

Phase	Focus	Highlights
phase1-bazaar	Flood-routed P2P marketplace	Structured overlay (ring + chords), lookup flood / path reply, direct buy, lock-protected stock
phase2-coordinator	Single elected trader, ordering & recovery	Bully leader election, vector clocks with deterministic tie-breaking, disk snapshot + journal so a new coordinator can resume mid-queue (~0.9 ms steady-state buy latency; ~10.6 ms average during re-election)
phase3-replication	Replication, caching & fault tolerance	Shared warehouse + replicated traders; eventually consistent ledger cache cut warehouse RPC traffic 5.8× vs cacheless; killing a trader left throughput flat at ~40 goods/s via heartbeats and idempotent retries

Design write-ups (PDF reports) are under reports/.

Repository layout

distributed-trading-post/
├── README.md
├── reports/                      # design + experiment write-ups
│   ├── phase1-bazaar-report.pdf
│   ├── phase2-coordinator-report.pdf
│   ├── phase3-replication-report.pdf
│   └── assignments/              # original homework prompts
├── phase1-bazaar/                # flood-routed P2P marketplace (HW3)
├── phase2-coordinator/           # elected trader + vector clocks + snapshots (HW4)
└── phase3-replication/           # warehouse + replicated traders + cache + FT (HW5)

Phases are independent runnable systems (not one merged binary). Start with Phase 3 for the multi-trader / warehouse architecture; Phase 2 for the coordinator story; Phase 1 for the earlier flooding bazaar.

Architecture evolution

flowchart LR
  subgraph P1["Phase 1 — Flood bazaar"]
    B0[Buyer] -->|lookup flood| N[Neighbors]
    N --> S0[Seller]
    B0 -->|direct buy| S0
  end

  subgraph P2["Phase 2 — Single coordinator"]
    B1[Buyers] --> T[Elected trader]
    S1[Sellers] -->|deposit| T
    T -->|snapshot / journal| Disk[(Disk)]
  end

  subgraph P3["Phase 3 — Replicated traders"]
    B2[Buyers] --> T0[Trader]
    B2 --> T1[Trader]
    S2[Sellers] --> T0
    S2 --> T1
    T0 --> WH[(Warehouse)]
    T1 --> WH
    T0 -.->|heartbeat| T1
    WH -.->|invalidate| T0
    WH -.->|invalidate| T1
  end

Loading

Phase 1 — Asterix and the Bazaar

Peers form a structured overlay (ring + chords, degree ≤ 3). Buyers flood lookup messages; sellers reply along the reverse path; buys complete over a direct TCP session. Stock updates are lock-protected.

Phase 2 — Single trading-post coordinator

Peers elect one trader via bully election. Sellers deposit inventory; buyers submit buy requests ordered with vector clocks and a deterministic tie-break. Trader state is snapshotted + journaled so a resigning/re-elected coordinator can resume without losing accounts or stock.

Phase 3 — Multi-trader replication

A central warehouse owns authoritative inventory. Multiple trader processes front it. Two modes:

• cacheless — every buy/sell is a warehouse RPC
• cache — traders keep an eventually consistent ledger-backed replica; warehouse pushes invalidations

Traders exchange heartbeats; on failure the survivor broadcasts SOLE_TRADER and peers retry with the same request_id so the warehouse stays idempotent.

Quick start

Requires Python 3.10+. Core runs use the standard library only. Optional plots need matplotlib (pip install -r requirements.txt inside a phase directory).

Phase 3 — caching + fault tolerance

cd phase3-replication

# Cache mode demo
python3 launcher.py --mode cache --nt 2 --nb 4 --ns 4 --n-buy-requests 50 \
  --tg 0.5 --ng 5 --buyer-warmup-s 1.5 --buyer-think-ms 50

# Kill trader 0 mid-run
python3 launcher.py --mode cache --nt 2 --nb 4 --ns 4 --n-buy-requests 80 \
  --tg 0.5 --ng 5 --buyer-warmup-s 1.5 --buyer-think-ms 80 \
  --fault-kill-after-s 4.0 --fault-kill-target 0

# Reproduce Section B numbers (~3–5 min full; or --quick)
python3 measure.py --full

Phase 2 — election, ordering, recovery

cd phase2-coordinator

python3 launcher.py 6 --requests 200 --resign-after 2
python3 measure.py --report

Phase 1 — flood-routed bazaar

cd phase1-bazaar

python3 launcher.py 6
python3 measure.py --report

More detail, flags, and design notes: each phase has its own README.md.

Key results (from checked-in experiment output)

Caching (Phase 3) — warehouse RPCs fall from ~188/s (cacheless path pressure) to ~32/s with the ledger cache, a 5.8× offload, while shipped goods/s stay comparable (~37 vs ~40). Oversell rises when buyer pressure outruns invalidation freshness.

Fault tolerance (Phase 3) — after killing a trader, throughput stays ~40 goods/s (pre 39.97/s → post 40.00/s in the recorded run).

Ordering & recovery (Phase 2) — steady-state buy latency ~0.9 ms; with trader resignation / re-election in the middle of the run, average client latency rises to ~10.6 ms.

Plots and CSVs live under each phase’s output/ directory.

Tech stack

• Python 3, subprocess process isolation (no shared memory across peers/traders/warehouse)
• Raw TCP + newline-delimited JSON messaging
• Threading within a process for concurrent connections / heartbeats
• Vector clocks (Phase 2), Lamport-ordered ledger + push invalidation (Phase 3)
• Disk snapshots / journals for recovery