docs: re-center messaging on the Kafka→ClickHouse ingestion flow#36
Merged
Conversation
The kafka command (full streams→raw→datalake pipeline + materialized views) is the tool's original purpose — the create_ddl_for_kafka.sh successor — but the taglines and README had drifted to a generic JSON→migration framing. - README: lead with the Kafka ingestion flow (diagram + kafka quickstart up top), new headline kafka demo GIF, and group scan/table/diff under Helper commands (scan GIF kept there as a helper demo). - cli.rs about/long_about/after_help and Cargo.toml description: foreground the pipeline; kafka-first tips instead of scan-first. - No CLI behavior change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
clickforge's core purpose is generating the Kafka→ClickHouse ingestion pipeline (
streams→raw→datalake+ two materialized views) — the direct successor tocreate_ddl_for_kafka.sh. The taglines and README had drifted toward a generic "JSON → migration SQL" framing that treatedkafkaas just one of four equal commands. This re-centers the messaging so the ingestion flow is the headline andscan/table/diffread as helpers.Changes
assets/kafka.gif) showingclickforge kafkagenerating thestreams.* ENGINE = Kafka(...)→raw.*→datalake.*DDL.clickforge kafka video_events.json.kafkaprimary, the rest helper;scan/table/diffgrouped under a Helper commands heading. The existing scan GIF (assets/demo.gif) now lives in thescanhelper section.src/cli.rs—about,long_about, and the top-levelafter_helpexamples lead withkafka; tips changed from "start with scan" to "runclickforge kafka <file>".Cargo.toml—descriptionreframed to the ingestion pipeline.assets/kafka.tape— VHS script for the new GIF (regenerable withvhs assets/kafka.tape).Notes
scan/table/diffare unchanged and still documented.cargo build,cargo fmt --check, andcargo test(26 passed) all green.