Repository navigation
docs: document ClickHouse Kafka-engine consumer for enriched flows - #402
Merged
Merged
Conversation
The deltav-flows consumer is ClickHouse itself (flows_kafka Kafka-engine table + flows_ingest MV into flows_raw), declared as init SQL rather than a deployable persister service. Covers the consumer-group/format settings, the three-way schema sync (proto / flows_kafka / positional MV SELECT), and the kafka_num_consumers scaling knob. Claude-Session: https://claude.ai/code/session_01E1cRKma4wRcjD6FmtVriA3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extends
docs/architecture/deltav-telemetryd-architecture.mdwith a new pipeline stage 4 describing how enriched flows reach ClickHouse:deltav-flowsconsumer is ClickHouse itself — theflows_kafkaENGINE = Kafkatable (groupdeltav-clickhouse-persister,ProtobufSingleagainstdeltav-flows.proto:FlowDocument,kafka_skip_broken_messages) plus theflows_ingestmaterialized view that drives consumption intoflows_raw, all declared as init SQL incomponents/clickhouse/init/— not a deployable Java persister.flows_kafkacolumns / positional MVSELECT) and the DROP+recreate migration pattern.kafka_num_consumersscaling knob to the scaling-ceilings section and routes the mermaid diagram's ClickHouse edge through Kafka.Documentation only — no code changes.
https://claude.ai/code/session_01E1cRKma4wRcjD6FmtVriA3