infra+dotnet: replace Kafka with in-process Channel<T> message bus (HOL-23)

[BrewingCoder]

Summary

Two infra containers (kafka + zookeeper) and a JVM stack went away in service of the same producer/consumer call shape. The .NET backend already runs in all-in-one mode where producer and worker share a process; the only thing Kafka was buying us at hobby scale was a JVM broker, two extra containers, and ~750 MiB of resident memory.

	RAM (idle)	Containers
Before HOL-23	~2 GiB	6
After HOL-23	~910 MiB	4
Original (pre-HOL-17)	12+ GiB	9

New abstraction (HoldFast.Shared.Messaging)

• IMessageBus — PublishAsync<T>(topic, key, value, ct) + SubscribeAsync(topic, ct)
• InProcessMessageBus — singleton with one Channel<(string, string)> per topic, lazily created. Unbounded (queue depth stays trivial at hobby scale; observed growth is the signal to swap in a Kafka-backed implementation).
• MessageConsumerBase<T> — replaces KafkaConsumerService<T>; reads channel, JSON-deserializes, dispatches to ProcessAsync.

Consumer mechanical refactor (6 files)

All six consumers (SessionEvents, ErrorGrouping, FrontendErrors, Logs, Metrics, Traces) re-based from KafkaConsumerService<T> to MessageConsumerBase<T>. Only the constructor signature changed (IOptions<KafkaOptions> → IMessageBus); ProcessAsync bodies unchanged.

Removed

• HoldFast.Shared/Kafka/{KafkaConsumerService, KafkaProducerService, KafkaTopicBootstrapService}.cs
• Confluent.Kafka package reference (HoldFast.Shared.csproj)
• Program.cs: Kafka DI block + KafkaTopicBootstrap registration
• compose.yml: kafka + zookeeper services + kafka-data/zoo-* volumes
• compose.hobby-dotnet.yml: Kafka__BootstrapServers env
• .env.example: KAFKA_*, ZOOKEEPER_IMAGE_NAME
• env.sh: KAFKA exports
• start-infra.sh: kafka+zookeeper from SERVICES list
• HoldFast.Shared.Tests/Kafka/KafkaOptionsTests.cs (class deleted)

Tradeoffs

• No durability — messages live in memory; backend restart drops the in-flight queue. Acceptable at hobby scale; pushPayload still returns 200, SDK retry covers transient loss.
• Single-process only — producer and consumer must be in the same .NET host (matches our all-in-one runtime mode).
• No replay / consumer groups / rebalancing.
• If you outgrow this, swap an alternative IMessageBus implementation in (Kafka, Redis Streams, RabbitMQ) without touching consumers.

Verified

• All 3,027 .NET tests pass (was 3,037; -10 from deleted KafkaOptionsTests for a class that no longer exists)
• Smoke test (./infra/docker/smoke-test-ingest.sh) passes end-to-end against the lean stack
• docker compose ps now shows 4 HoldFast containers (backend, clickhouse, frontend, postgres)

Test plan

• docker compose down && docker compose up -d → 4 containers come up
• Smoke test passes
• No Connection refused localhost:9092 errors anywhere
• Backend logs show In-process consumer started for topic ... for each of session-events, backend-errors, frontend-errors, logs, metrics, traces

Stacks on #101 (HOL-22). Subtask of HOL-17. Closes HOL-23.

🤖 Generated with Claude Code