Repository navigation
Unblock publishing: reset + cap oversized history.jsonl - #82
Conversation
The "Publish Report to GitHub Pages" workflow failed on every run: its "Commit & push updated history" step pushes history.jsonl back to master, and the file had grown to ~101 MB (83 run snapshots), over GitHub's 100 MB per-file push limit, so every push was rejected and no report deployed (qa.meshery.io reports went stale; /connections/ 404'd). Root cause: allure generate (historyPath: ./history.jsonl in allurerc.mjs) appends one run snapshot per line and never bounds the file. Each snapshot also carries the full knownTestCaseIds set, so it grows both per run and per test. Fix (captain-approved reset + cap; no test results affected, only trend charts): - Remove the oversized committed history.jsonl. The next publish regenerates a fresh, small one (the seed step already handles a missing file as "first run"). - Add a "Cap Allure trend history" step before the commit step that keeps only the most recent runs, bounded by BOTH a run count (25) and a hard byte budget (80 MB) so it can never approach the 100 MB limit again - the byte budget also guards against a single snapshot ballooning as the suite grows. Past large blobs remain in history (a separate, optional cleanup) but do not block publishing. No git history rewrite. Signed-off-by: marblom007 <158522975+marblom007@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Unblocks the GitHub Pages publish workflow by preventing history.jsonl from exceeding GitHub’s per-file push limit, via a pre-commit capping step.
Changes:
- Add a workflow step to cap
history.jsonlby maximum run count and maximum byte size before committing/pushing. - Document the rationale and constraints directly in the workflow for future maintainers.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| while [ "$(wc -c < "$HIST.capped")" -gt "$MAX_BYTES" ] && [ "$(wc -l < "$HIST.capped")" -gt 1 ]; do | ||
| tail -n +2 "$HIST.capped" > "$HIST.capped.tmp" | ||
| mv "$HIST.capped.tmp" "$HIST.capped" | ||
| done |
| MAX_RUNS=25 | ||
| MAX_BYTES=$((80 * 1024 * 1024)) |
|
Warning Review limit reached
Next review available in: 34 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe publishing workflow now bounds ChangesAllure history publishing
Estimated code review effort: 2 (Simple) | ~10 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/publish-allure-report.yml:
- Around line 73-76: Update the history-capping loop around HIST.capped so
MAX_BYTES is enforced even when the remaining record is a single oversized line.
After trimming records, validate the final file size before moving it into
place; remove the oversized final record or explicitly fail the job instead of
allowing an oversized file to be written.
- Around line 59-60: Update the workflow steps around “Cap Allure trend history”
and make report-build so history.jsonl is capped before make report-build runs,
ensuring the generated allure-report uses bounded history; retain the final byte
cap if it is still required.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 48765cd6-a4b9-45d8-8e58-dc76372f0cd9
📒 Files selected for processing (2)
.github/workflows/publish-allure-report.ymlhistory.jsonl
Review on meshery#82 (CodeRabbit + Copilot): - CodeRabbit (major): `make report-build` generated the deployed report BEFORE the cap ran, so the report could embed uncapped history. Now the cap runs on BOTH sides of report-build - before (report built from a bounded seed) and after (the file committed/pushed is re-bounded after allure appends this run). - CodeRabbit/Copilot: the size loop could stop at one line still over budget and push an oversized file. Added a hard-limit failsafe that resets to a fresh history if a single retained snapshot still exceeds HISTORY_HARD_BYTES (95 MiB, < GitHub's 100 MB limit) - losing one pathological snapshot beats blocking every deploy. - Copilot: lifted the caps to job-level env (HISTORY_MAX_RUNS / MAX_BYTES / HARD_BYTES) and extracted the logic into cap-history.sh so tuning needs no script edits and the before/after steps share one implementation. Verified all paths: normal 83->25 runs, missing-file no-op, single-oversized- line failsafe reset to empty. Signed-off-by: marblom007 <158522975+marblom007@users.noreply.github.com>
Review on meshery#82 (CodeRabbit + Copilot): - CodeRabbit (major): `make report-build` generated the deployed report BEFORE the cap ran, so the report could embed uncapped history. Now the cap runs on BOTH sides of report-build - before (report built from a bounded seed) and after (the file committed/pushed is re-bounded after allure appends this run). - CodeRabbit/Copilot: the size loop could stop at one line still over budget and push an oversized file. Added a hard-limit failsafe that resets to a fresh history if a single retained snapshot still exceeds HISTORY_HARD_BYTES (95 MiB, < GitHub's 100 MB limit) - losing one pathological snapshot beats blocking every deploy. - Copilot: lifted the caps to job-level env (HISTORY_MAX_RUNS / MAX_BYTES / HARD_BYTES) and extracted the logic into cap-history.sh so tuning needs no script edits and the before/after steps share one implementation. Verified all paths: normal 83->25 runs, missing-file no-op, single-oversized- line failsafe reset to empty. Signed-off-by: marblom007 <158522975+marblom007@users.noreply.github.com>
2f33fca to
abc9ecb
Compare
|
Thanks @coderabbitai and @copilot - all four points addressed in abc9ecb:
Tested all paths locally against the real 83-run/105 MB file: normal cap -> 25 runs/32 MB, missing-file no-op, and the single-oversized-line failsafe resets to empty. |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
git show --stat --oneline abc9ecb
git show --format= --name-only abc9ecb
printf '\n--- Candidate files ---\n'
fd -HI '^(publish-allure-report\.yml|cap-history\.sh)$' . || true
printf '\n--- Workflow outline ---\n'
workflow="$(fd -HI '^publish-allure-report\.yml$' . | head -n1)"
if [ -n "$workflow" ]; then
ast-grep outline "$workflow" --items all || true
printf '\n--- Workflow relevant sections ---\n'
rg -n -C 8 'HISTORY_(MAX_RUNS|MAX_BYTES|HARD_BYTES)|cap-history\.sh|report-build|Cap Allure trend history' "$workflow" || true
fi
printf '\n--- Script content ---\n'
script="$(fd -HI '^cap-history\.sh$' . | head -n1)"
if [ -n "$script" ]; then
nl -ba "$script"
fi
printf '\n--- Commit patch ---\n'
git show --format= --find-renames abc9ecb -- \
'.github/workflows/publish-allure-report.yml' \
':(glob)**/cap-history.sh'Length of output: 3565 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '--- cap-history.sh (numbered) ---'
awk '{ printf "%6d %s\n", NR, $0 }' cap-history.sh
printf '%s\n' '--- cap-history.sh patch in abc9ecb ---'
git show --format= --find-renames abc9ecb -- cap-history.sh
printf '%s\n' '--- Full relevant workflow block ---'
sed -n '30,80p' .github/workflows/publish-allure-report.yml
printf '%s\n' '--- Path and call-site checks ---'
rg -n -C 3 'history\.jsonl|cap-history\.sh|HISTORY_(MAX_RUNS|MAX_BYTES|HARD_BYTES)' \
.github/workflows/publish-allure-report.yml cap-history.shLength of output: 13776
The seed cap runs before
The job-level environment provides the three cap values. Both workflow steps use the same script. The four review findings are addressed. 🐇 ✏️ Learnings added
You are interacting with an AI system. |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@cap-history.sh`:
- Around line 46-57: The failsafe in cap-history must enforce MAX_BYTES for the
final retained snapshot, not only HARD_BYTES. Update the post-loop check after
the tail/mv loop to reset or fail whenever HIST.capped exceeds MAX_BYTES, and
adjust the warning to reference the configured maximum; preserve the existing
behavior for snapshots within the limit.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: acac08e3-9c7e-4057-8a0e-c499af09628a
📒 Files selected for processing (2)
.github/workflows/publish-allure-report.ymlcap-history.sh
…e ceiling Review on meshery#82 (CodeRabbit + Copilot): - CodeRabbit (major): `make report-build` generated the deployed report BEFORE the cap ran, so the report could embed uncapped history. The cap now runs on BOTH sides of report-build - before (report built from a bounded seed) and after (the file committed/pushed is re-bounded after allure appends this run). - CodeRabbit/Copilot: the size cap could retain a single snapshot still over budget. HISTORY_MAX_BYTES is now a STRICT ceiling: a single-pass, byte-accurate (LC_ALL=C) awk keeps only the newest complete lines whose cumulative bytes fit, and if even the newest snapshot alone exceeds it the file resets to fresh - the file written back is always <= the ceiling (well under GitHub's 100 MB). - Copilot: lifted the caps to job-level env (HISTORY_MAX_RUNS / MAX_BYTES) and extracted the logic into cap-history.sh shared by the before/after steps. Tested: 83->25 runs normal; byte cap trims to newest lines under budget in one fast pass; single-oversized-snapshot resets to empty; missing file no-op. Signed-off-by: marblom007 <158522975+marblom007@users.noreply.github.com>
abc9ecb to
0f1fbfb
Compare
|
Addressed in 0f1fbfb. |
Problem
The Publish Report to GitHub Pages workflow has been failing on every run. Its "Commit & push updated history" step pushes
history.jsonlback tomaster, but the file had grown to ~101 MB (83 run snapshots) - over GitHub's 100 MB per-file push limit - so every push was rejected (pre-receive hook declined), the build job failed, anddeploywas skipped. No report has deployed for hours; every dashboard view (including the new Connection Lifecycle report) is stale andhttps://qa.meshery.io/connections/404s.Root cause
allure generate(historyPath: "./history.jsonl"inallurerc.mjs) appends one run snapshot per line and never bounds the file. Each snapshot also embeds the fullknownTestCaseIdsset, so it grows both per run and per test (83 lines x ~1.26 MB).Fix (reset + cap)
history.jsonl. The next publish regenerates a fresh, small one -report-buildalready treats a missing file as "first run". Only historical trend charts are lost; no test results are affected.Cap Allure trend historystep before the commit step that keeps only the most recent runs, bounded by both a run count (MAX_RUNS=25) and a hard byte budget (MAX_BYTES=80 MB), so the file can never approach the 100 MB limit again. The byte budget also guards against a single snapshot ballooning as the suite grows.Verified the cap on the real 83-run/105 MB file:
-> 25 runs / 32 MB, every retained line valid JSON, newest snapshot preserved.Minimal and qa-scoped; no git-history rewrite (past large blobs are a separate, optional cleanup that does not block publishing).
Post-merge
The publish workflow does not trigger on
.github/**changes, so I will dispatch aPublish Report to GitHub Pagesrun after merge and confirm it succeeds and thathttps://qa.meshery.io/connections/returns 200 with the Connection Lifecycle report.Summary by CodeRabbit