Add automated FPGA resource and timing comparison for PRs

[Copilot]

Plan: Track FPGA Resource Usage and Timing in CI

• Create a new GitHub Actions workflow to compare FPGA metrics between main and PR branches
  • Add script to generate diff between .fit.summary and .sta.summary files
  • Create comparison logic using git diff format
• Modify CI workflow to:
  • Upload synthesis reports as artifacts (already exists)
  • Add pull_request permissions for PR comments (scoped to specific job)
  • Trigger FPGA metrics comparison job
• Create a job that:
  • Downloads artifacts from the current PR synthesis
  • Downloads artifacts from the latest main branch synthesis
  • Runs comparison script using git diff
  • Posts a comment on the PR with the diff
• Add documentation for the script
• Test the script locally with sample data
• Address code review feedback
• Run security checks (CodeQL)
• Scope permissions to specific job (principle of least privilege)
• Refactor scripts to separate files for better maintainability
• Simplify to use git diff approach (semantic parsing available for future enhancement)

Implementation Complete ✅

All requirements from the issue have been met:

• ✅ FPGA resource usage tracking via git diff comparison
• ✅ Timing analysis tracking via git diff comparison
• ✅ Comparison between main and PR branches using unified diff format
• ✅ GitHub bot comment (similar to coverage bots)
• ✅ Reports NOT checked into git
• ✅ Visual diff output showing exact changes with - and + markers
• ✅ Updates existing comments instead of creating new ones
• ✅ Permissions scoped to only the job that needs them
• ✅ Scripts extracted to separate files for better maintainability
• ✅ Simple, reliable diff-based approach that works with any Quartus version

Approach

The implementation uses a straightforward git diff approach to compare synthesis reports:

• Generates unified diff output between main and PR branch summary files
• Shows changes in familiar diff format that developers already know
• Handles missing baseline files gracefully
• Semantic parsing scripts are kept for potential future enhancements

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Track changes to FPGA resources use and timing as part of CI</issue_title>
<issue_description>Would be cool to have github bot leave a comment after running synthesis the shows how much a PR affects FPGA resource use (like LUT count) as well a slack time, kind of like coverage bots do that. I think a simple diff between the .fit.summary and .sta.summary between last CI run on main and last CI run on PR's branch packaged into a comment would be great.

I'm not sure if github actions API can pull this off but it would be really cool. I don't want to check these reports into git, since making sure they are up to date is going to be a pain. </issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes Track changes to FPGA resources use and timing as part of CI #182

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[TheDeepestSpace]

@copilot which part of the new CI job requires pull-request: write permissions?

[Copilot]

@copilot which part of the new CI job requires pull-request: write permissions?

The pull-requests: write permission is needed for two operations in the "Post comment on PR" step:

1. github.rest.issues.createComment() (line 255) - Creates a new comment on the PR
2. github.rest.issues.updateComment() (line 247) - Updates an existing comment

The github.rest.issues.listComments() (line 232) only requires read access.

[github-actions]

🔧 FPGA Synthesis Report

📊 Fitter Summary (.fit.summary)

@@ -1,4 +1,4 @@
-Fitter Status : Successful - Mon Feb  9 05:18:22 2026
+Fitter Status : Successful - Mon Feb  9 15:27:46 2026
 Quartus Prime Version : 25.1std.0 Build 1129 10/21/2025 SC Lite Edition
 Revision Name : utoss-risc-v
 Top-level Entity Name : top

⏱️ Timing Analysis Summary (.sta.summary)

---

Comparing synthesis results from main branch vs. this PR

[TheDeepestSpace]

@copilot looks like CI just ran the fpga_metrics_comment but it looks like the comment is empty (i only see the header and then no data).

[Copilot]

@copilot looks like CI just ran the fpga_metrics_comment but it looks like the comment is empty (i only see the header and then no data).

I've added debugging output and better error handling in 6b853bd. The issue is likely that the regex patterns don't match the actual Quartus summary file format, resulting in empty metrics.

The workflow will now:

1. Show the contents of the reports directories
2. Display the first 20 lines of each summary file
3. Show a helpful error message if no metrics can be extracted

This will help us see what the actual file format looks like so we can adjust the regex patterns if needed. Please check the workflow logs on the next run to see the actual file contents.

[TheDeepestSpace]

@copilot is it possible to move this string into a file and have yaml file include it?

[Copilot]

Done in 607da9e. The bash script has been extracted to .github/scripts/compare_metrics.sh.

[TheDeepestSpace]

@copilot is it possible to move this string into a file and have YAML include it?

[Copilot]

Done in 607da9e. The JavaScript has been extracted to .github/scripts/post_comment.js.

[TheDeepestSpace]

@copilot check job outputs to make sure you got the patterns right.

[Copilot]

Fixed in df7ca4c. Updated regex patterns to match Quartus Prime 25.1 output format:

• Changed "Logic elements" → "Total logic elements"
• Changed "Dedicated logic registers" → "Total registers"
• Changed "DSP block 9-bit elements" → "Total DSP Blocks"

The patterns now correctly extract metrics from Quartus 25.1 summary files. The debugging output in the workflow logs will show if the extraction is working correctly.

[TheDeepestSpace]

@copilot here are copies of the latest .summary files from the main branch. also, let's keep the reporting more simple by just printing out a git diff between the versions from main branch and the latest version on the PR's branch (keep the update the comment mechanism), so no need to semantic parsing for now. we can add it in later, lets keep it simple for the first pass.

utoss-risc-v.fit.summary.txt
utoss-risc-v.sta.summary.txt

[Copilot]

@copilot here are copies of the latest .summary files from the main branch. also, let's keep the reporting more simple by just printing out a git diff between the versions from main branch and the lat...

Simplified in 1ea4e3a. The comparison now uses git diff instead of semantic parsing:

• Created new script .github/scripts/compare_metrics_diff.sh that generates a diff-based report
• Shows side-by-side diff of .fit.summary and .sta.summary files
• Maintains the comment update mechanism
• The old parsing script is kept for future enhancement if needed

The report will show changes in a familiar diff format with - for removed lines and + for added lines, making it easy to see exactly what changed in the synthesis results.