LOTC-646: [TrafficPeak] Bot Insights CDN#175
Open
kevinborkman-hub wants to merge 56 commits into
Open
Conversation
Reorganized transformations into provider subdirectories (akamai, cloudflare, cloudfront_firehose, default, fastly), cleaned metadata fields, extracted sample data, and updated SQL prefixes from reference_ to akamai_ for trafficpeak bundle. Created bundle.json with multi_stream method configuration, updated summary SQL files with template variables, and fixed dashboard structure with proper template variable patterns for primary dashboard. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…ment-templates into LOTC-646-traffic-peak-bot-insights-cdn
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ydrolix/integration-deployment-templates into LOTC-646-traffic-peak-bot-insights-cdn
…ub.com/hydrolix/integration-deployment-templates into LOTC-646-traffic-peak-bot-insights-cdn
…-bot-insights-cdn
Fix originals with manual corrections from branch history: - ai_category → user_agent_category in dashboard and summaries - cacheStatus suppress: true → false in transform - Summary field names updated to match transform output columns - Remove bundle.json so CI regenerates it from bundle-config.json Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Update primary_url to Akamai techdocs URL - Remove old portables/bot_insights_cdn/ (CI generates portables/trafficpeak/) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Shift reqTimeSec values to Apr 1 2026 UTC to pass the 183-day freshness threshold. The pipeline's _shift_stale_timestamps() doesn't handle transforms where the raw JSON key differs from the output column name (reqTimeSec vs timestamp) — bug ticket to follow. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Previous commit only patched akamai.json on top of processed files. This properly replaces the bundle dir with raw originals including all fixes (ai_category, cacheStatus suppress, summary field names, fresh timestamps) and removes stale portables for full pipeline re-run. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Resolve conflicts from prior CI run by keeping deletions — bundle dir has only raw originals with all fixes (ai_category, cacheStatus suppress, summary field names, fresh timestamps) plus bundle-config.json. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…into LOTC-646-traffic-peak-bot-insights-cdn
Restore original stale reqTimeSec values (188-510 days old) in originals and bundle dir to verify the _shift_stale_timestamps() fix from LOTC-1412 can now resolve from_json_pointers and freshen them automatically. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…-bot-insights-cdn
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Replace ${VAR_SUMMARY_HOUR/DAY/MONTH} with direct __PROJECT_NAME__.bot_summary_* refs to bypass the configurator's self-reference misrouting (tracked in LOTC-1435)
- Replace `count()` backticked column name with cnt_all in 3 panel queries (the summary's count() is aliased as cnt_all)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The summary column is stored as AggregateFunction(count) and when referenced by its alias cnt_all, ClickHouse resolves it back to countMerge(`count()`), which nests inside the outer countMergeIf() and fails with error 184. Using `count()` directly (matching aws/bot-insights) avoids the resolution round-trip. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…-bot-insights-cdn
…eline re-run CI Stage 2 will auto-freshen stale primary timestamps via transform_organizer._shift_stale_timestamps. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
kcorbett-hdx
requested changes
Apr 28, 2026
| __extend__: dashboards/bot_insights_cdn.json | ||
| folderUid: hdx-security-folder | ||
| inputs: | ||
| DS_HYDROLIX-HYDROLIX-DATASOURCE: hdx-hydrolix-datasource |
Collaborator
There was a problem hiding this comment.
summary tables are missing...was that not automatically generated?
Collaborator
There was a problem hiding this comment.
You can delete this file since this uses summary tables explicitly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
trafficpeak/bot-insights-cdn/).main.Contents
Bot Insights (CDN).json), 1 transform (akamai.json), 3 summaries (bot_summary_day/hour/month.sql), 14 UDFs underfunctions/..originals/trafficpeak/bot-insights-cdn/contains the preserved raw assets for pipeline re-runs.bundle-config.json:data_category: security,table_name: logs,version: 1.0.0.CI expectations
scripts/detect_track.py.bundle.json,sample_data.json, rename the transform, inject template variables into the dashboard/summaries, and produce portables atportables/trafficpeak/bot_insights_cdn/1.0.0/.Links
Test plan
portables/trafficpeak/bot_insights_cdn/1.0.0/Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com