SQL queries for dashboard data

[miguelgonrod]

Part of https://github.com/osrf/buildfarm-tools-private/issues/90

[Crola1702]

Can you add example runs of each of the queries?

[Crola1702]

I think these changes are due to an sqlite3 version difference, right? Which version are you testing with?

[miguelgonrod]

Yes, Ubuntu's 24.04 sqlite3 version is 3.45.1 in which already has a warning of not using double quotes for table columns, jajaj for some reasons I tried to run this command with version 3.53.2 and in this version is fully prohibited using double quotes for this type of calls. I don't know if we have the same problem with other sql scripts, but I found that one

[Crola1702]

Okay, some comments to start:

• The queries are really big, and they can be pretty complex and also really hard to maintain.
• We want to prioritize database size (we don't want to grow the database a lot), and maintainability of the queries.

My suggestion is to reuse Common table expressions into Views, so we can have stored queries there

Let's create a active_failures view, which identifies persistently broken tests across the buildfarms.

Let's see if there are other suggestions to extra views we can leverage.

Also, please fill the PR description with the intended usage and purpose of each of the queries, as well as real examples of their usage (in the PR comments is okay)

[Crola1702]

Oookay, I think this might be a bottleneck of our tooling right now.

I suggest adding (and updating) these rules to the server_status table. Which already contains some of this information:

nightly_linux_debug | Active | x86_64_Ubuntu | x86_64_Ubuntu | ROS | https://ci.ros2.org
nightly_linux_release | Active | x86_64_Ubuntu | x86_64_Ubuntu | ROS | https://ci.ros2.org
nightly_linux_repeated | Active | x86_64_Ubuntu | x86_64_Ubuntu | ROS | https://ci.ros2.org
packaging_linux | Active | x86_64_Ubuntu | x86_64_Ubuntu | ROS | https://ci.ros2.org
nightly_linux-aarch64_debug | Active | aarch64_Ubuntu | aarch64_Ubuntu | ROS | https://ci.ros2.org
nightly_linux-aarch64_release | Active | aarch64_Ubuntu | aarch64_Ubuntu | ROS | https://ci.ros2.org
nightly_linux-aarch64_repeated | Active | aarch64_Ubuntu | aarch64_Ubuntu | ROS | https://ci.ros2.org
packaging_linux-aarch64 | Active | aarch64_Ubuntu | aarch64_Ubuntu | ROS | https://ci.ros2.org
nightly_linux-rhel_debug | Active | linux-rhel | linux-rhel | ROS | https://ci.ros2.org
nightly_linux-rhel_release | Active | linux-rhel | linux-rhel | ROS | https://ci.ros2.org
gz_cmake-ci-main-homebrew-arm64 | Active | rotary | Brew Arm64 | GAZEBO | https://build.osrfoundation.org
gz_cmake-ci-main-noble-amd64 | Active | rotary | Linux Noble | GAZEBO | https://build.osrfoundation.org
gz_cmake-ci-main-resolute-amd64 | Active | rotary | Linux | GAZEBO | https://build.osrfoundation.org
gz_cmake-main-cnlwin | Active | rotary | Windows | GAZEBO | https://build.osrfoundation.org
gz_common-ci-main-homebrew-amd64 | Active | rotary | Brew Amd64 | GAZEBO | https://build.osrfoundation.org
gz_common-ci-main-homebrew-arm64 | Active | rotary | Brew Arm64 | GAZEBO | https://build.osrfoundation.org

This implies a change to the database schema and a updates to databaseFetcher.p. Please do that in a different PR.

[Crola1702]

The query is pretty big. Not a bad comment, but it might be less understandable. If you can add some extra comments of what every part is doing, I think it would increase the maintainability

[miguelgonrod]

Ready, added comments to queries

[miguelgonrod]

Dashboard SQL queries: purpose, usage and examples

Summary

This PR adds SQL queries used by the dashboard generator and documents intended usage and real examples for each query. All examples use the project's wrapper script located at sql_run.sh, which runs SQL against the repository database buildfarmer.db.

active_test_regressions

• Purpose: Identify tests that are currently failing consistently and have not passed in the last 3 days. Used to populate the "Active regressions" section of the dashboard and prioritize triage.
• File: active_test_regressions.sql
• Intended usage: Run periodically (or on-demand) to produce a list of active regressions grouped by test/package, with first-seen date, consecutive failure streak and affected platforms.
• Example (from repository root):

cd database/scripts
./sql_run.sh active_test_regressions.sql

• Sample output (first row):

topic_monitor.topic_monitor.test.test_mypy.test_mypy | topic_monitor | 2026-05-29 | 34 | [{"os":"linux","arch":"unknown"}] | https://github.com/ros2/geometry2/issues/941

flaky_tests

• Purpose: List flaky tests observed in a ~20-day window (excludes tests already present in active_test_regressions). Returns failure_count, total_runs, failure percentage, affected platforms and linked issue if any.
• File: flaky_tests.sql
• Intended usage: Feed the dashboard's flaky-tests section and help decide which tests require investigation or labeling as known issues.
• Example:

cd database/scripts
./sql_run.sh flaky_tests.sql

• Sample output (first row):

1 cmake warnings | cmake | 90 | 145 | 62.07 | [{"os":"linux","arch":"aarch64"},...] | https://github.com/ros2/rviz/issues/1750

single_test_history

• Purpose: Produce a 90-day daily time series for a single test_name. Each day is 1 (pass), 0 (fail) or NULL (not run). Used by the dashboard when a user inspects a single test's history.
• File: single_test_history.sql
• Parameters: The query expects a single parameter @param1@ for the test_name. Pass it as the script argument to sql_run.sh
• Example:

cd database/scripts
./sql_run.sh single_test_history.sql "topic_monitor.topic_monitor.test.test_mypy.test_mypy"

• Sample output (snippet):

2026-03-06 | 0
2026-03-07 | 1
2026-03-08 | NULL

known_issues_summary

• Purpose: Return a summary of issues tracked in test_fail_issues, including open issues and recently-seen closed issues (last 30 days). Includes derived title, status, priority, assignee, last_activity_date, last_seen_in_build_date and stale flag.
• File: known_issues_summary.sql
• Intended usage: Populate the dashboard view for known issues and provide direct links to GitHub issues for triage.
• Example:

cd database/scripts
./sql_run.sh known_issues_summary.sql

• Sample output (first row):

https://github.com/gazebosim/gz-plugin/issues/210 | gazebosim/gz-plugin#210 | NOT ASSIGNED | 0.175 | Not Assigned | NULL | 2026-05-17 | 0

Operational notes

• Use sql_run.sh as the canonical way to run these queries; the script reads the SQL file, substitutes @param1@/@param2@ placeholders when arguments are provided, and runs the final SQL against buildfarmer.db.
• Run commands from scripts (or adjust the path to sql_run.sh ) so the script finds buildfarmer.db.

[miguelgonrod]

Okay, some comments to start:

• The queries are really big, and they can be pretty complex and also really hard to maintain.
• We want to prioritize database size (we don't want to grow the database a lot), and maintainability of the queries.

My suggestion is to reuse Common table expressions into Views, so we can have stored queries there

Let's create a active_failures view, which identifies persistently broken tests across the buildfarms.

Let's see if there are other suggestions to extra views we can leverage.

Also, please fill the PR description with the intended usage and purpose of each of the queries, as well as real examples of their usage (in the PR comments is okay)

Ready, added views. Please check if that was what you meant. The check of the OS is still pending, I'm checking that

[miguelgonrod]

Made the respective migration, but found an "error"

This are the available platforms

platform_os	COUNT(*)
homebrew	154
linux	163
unknown	15
windows	83

And as we can see there are 15 in unknown, these are

job_name	platform_os	platform_arch
gz-rotary-cmake-debbuilder	unknown	unknown
gz-rotary-common-debbuilder	unknown	unknown
gz-rotary-fuel-tools-debbuilder	unknown	unknown
gz-rotary-gui-debbuilder	unknown	unknown
gz-rotary-math-debbuilder	unknown	unknown
gz-rotary-msgs-debbuilder	unknown	unknown
gz-rotary-physics-debbuilder	unknown	unknown
gz-rotary-plugin-debbuilder	unknown	unknown
gz-rotary-rendering-debbuilder	unknown	unknown
gz-rotary-sdformat-debbuilder	unknown	unknown
gz-rotary-sensors-debbuilder	unknown	unknown
gz-rotary-sim-debbuilder	unknown	unknown
gz-rotary-tools-debbuilder	unknown	unknown
gz-rotary-transport-debbuilder	unknown	unknown
gz-rotary-utils-debbuilder	unknown	unknown

There are no rule for debbuilder, they are homebrew? or should I create a new category?

[Crola1702]

There are no rule for debbuilder, they are homebrew? or should I create a new category?

Debbuilders should be ubuntu (may be resolute or noble, that's hard to handle actually, so we can keep them as ubuntu), and arch is amd64