178 hive sentinel clickhouse pixie

[mebegu]

• Added ClickHouse table initialization SQL for ETL pipelines (kubescape, tetragon, http, dns, STIX) via init configuration.

• Configured Vector to sink Kubescape and Tetragon data into ClickHouse automatically.

• Configured Kubescape and Tetragon STIX ETLs to auto-start in Flask on boot, so data inserted into ClickHouse via Vector is automatically transformed into STIX.

• Added Flask endpoints to control Pixie ETL lifecycle (start, stop, status). Implemented ETLs for dns_events and http_events tables from Pixie.

• Implemented Pixie STIX transformation and integrated it into the Pixie ETL, allowing Pixie data to be transformed into STIX while storing it in ClickHouse.

• Added unit tests for Hive Sentinel, covering Pixie ETL, STIX ETL, and endpoint behavior.

• Adjusted GitHub Actions to tag Hive Sentinel images by branch.

• Added a Make command to deploy Hive Sentinel to the cluster, allowing direct ClickHouse DB access through cluster DNS without requiring port forwarding.

• Added endpoints to fetch and filter the contents of clickhouse database tables. Currently, it is just to see content of the database.

[mebegu]

I have worked on my lab, I can see the kubescape and tetragon logs on pixie. And also deployed the hive_sentinel to the cluster.

[mebegu]

I also added endpoints to fetch and filter the contents of clickhouse database tables. Currently, it is just to see content of the database. But we can extend

[mebegu]

Pixie ETL is also there, in the attach screenshots, it shows we fetch http_events with webapp namespace into its table, and also do stix transformation for it into http_stix table

[mebegu]

Please find my lab script at
https://github.com/k8sstormcenter/ixi/blob/main/socclickhouse-32f67ac6/index.md

I merged all into a single PR with this one. so, if we are happy with this version, we can abandon the other 3 PRs (#185, #190, #191)

I went through all the commands on the lab, did not encounter any runtime errors or empty returns. And generally I observed the expected results
I will spend some more time to verify the content matches, and double check if the screenshots are aligned

The Hive Sentinel text on the lab might not be perfect, but we can improve on that later

[entlein]

I read and tested the code and lab. THANK YOU, this is great!

Merge: YES - Reason: Since all labs that have dependencies are build off feature branches, merging the vector/soc.yaml wont break anything.
Tomorrow: We will together review the pattern-matcher and determine any next urgent steps.

All in all: this is now going in exactly the right direction, looking forward to see part 2 tomorrow.
I assume that the vector/soc.yaml will be fixed with the next PR