[RUM-9617] - Base Benchmark app with support for Test Scenarios and external config by sbarrio · Pull Request #881 · DataDog/dd-sdk-reactnative

sbarrio · 2025-05-19T15:25:19Z

What does this PR do?

This PR implements a base implementation for an app that can be internally used as a benchmark tester of different scenarios. It lays out the groundwork needed to be able to set up different kinds of builds, tests and related configurations so test scenarios can be run under different environments and instrumentation conditions. It allows for new scenarios to be added easily, allowing them to be instrumented with the Datadog SDK. It also enables a set of vitals tracking systems at the native level so metrics on CPU/Memory/FPS are tracked and reported too.

Motivation

We want to have a set of automatic benchmark tests run every time we merge code to develop.

Additional Notes

This PR on itself does not yet implement any real test scenario nor sets up a CI pipeline for tests. That will come on the next PRs right after this one. This is just laying the groundwork needed for those to work.

Review checklist (to be filled by reviewers)

Feature or bugfix MUST have appropriate tests
Make sure you discussed the feature or bugfix with the maintaining team in an Issue
Make sure each commit and the PR mention the Issue number (cf the CONTRIBUTING doc)
If this PR is auto-generated, please make sure also to manually update the code related to the change

datadog-datadog-prod-us1 · 2025-05-20T08:10:08Z

Datadog Report

Branch report: sbarrio/RUM-9617/base-benchmark-test-app-with-scenario-support
Commit report: a0f8388
Test service: dd-sdk-reactnative

✅ 0 Failed, 661 Passed, 1 Skipped, 3.76s Total Time

sbarrio · 2025-05-26T08:06:06Z

+  implementation "com.datadoghq:dd-sdk-android-rum:2.21.0"
+  implementation "com.datadoghq:dd-sdk-android-logs:2.21.0"
+  implementation "com.datadoghq:dd-sdk-android-trace:2.21.0"
+  implementation "com.datadoghq:dd-sdk-android-webview:2.21.0"


This PR needs Android SDK 2.21.0 as the Vitals tracking module was compiled with that version (it relies on Metrics improvements done on that version).

For this, we'll need to merge this PR first.

We now include the benchmark tools directly from Maven, but I still want to keep them at 2.21.0 since there were important changes included on that version that our Vitals tracking native modules rely on.

sbarrio · 2025-05-26T08:08:10Z

+    // From the release branch of dd-sdk-android, run:
+    //      ./gradlew :tools:benchmark:assembleRelease
+    // Then copy the resulting .aar into benchmarks/android/app/libs
+    implementation(name: 'benchmark-release', ext: 'aar')


The benchmark app relies on the tools/benchmarks vitals tracking code on dd-sdk-android, however, that code is not published as a standalone dependency that can be imported into different projects, so at least for now, I opted for including it as a precompiled .aar into this package. This means that we need to recompile it whenever we bump the Android version.

On next iterations we can replace this with an external dependency if we decide to make the benchmarks project public.

I believe this and the iOS code duplication are the only critical issues to address, as manually assembling the AAR for each bump of the Android SDK version would be too complex, and maintaining many duplicate iOS classes is not ideal.

you can already include the tools:benchmark module like this implementation "com.datadoghq:benchmark:2.19.2". It is published.

Thanks @aleksandr-gringauz, let me give it a try. I assume 2.21.0 is also available?

It does indeed work. Thanks for raising this 🙇

@marco-saia-datadog I'll ask the iOS SDK team for support on this in case there's a similar solution possible on their end.

sbarrio · 2025-05-26T08:14:52Z

+import OpenTelemetryApi
+
+internal final class Meter: DatadogInternal.BenchmarkMeter {
+    let meter: OpenTelemetryApi.Meter


These classes are a 1:1 match with the ones found on the iOS Benchmark app code.

Ideally, we should be able to only import this as an external dependency, but SPM does not allow for specific folder imports from Github, just whole projects.

cdn34dd · 2025-05-27T15:52:08Z

If this PR does not yet implement any test scenarios, should this not be merged onto a feature brach, and only push to develop once the full behavior is implemented ?

marco-saia-datadog

@sbarrio Amazing work here! 🙌 🎉

As @cdn34dd suggested, let's change the target to a feature branch, and let's only merge it to develop when we have at least one real test scenario, and a CI job to run it.

sbarrio · 2025-05-28T08:05:07Z

If this PR does not yet implement any test scenarios, should this not be merged onto a feature brach, and only push to develop once the full behavior is implemented ?

@sbarrio Amazing work here! 🙌 🎉

As @cdn34dd suggested, let's change the target to a feature branch, and let's only merge it to develop when we have at least one real test scenario, and a CI job to run it.

Thanks for the reviews 🙇

I'll create a feature branch as suggested, so things are cleaner 👍

sbarrio · 2025-05-28T08:32:07Z

@janine-c Thanks a ton for your thorough review. I have updated the README.md with your suggestions and have clarified some parts 🙇

cdn34dd

Looks good, well done.

janine-c

Looks great, thank you for applying those edits so quickly! 🚀

sbarrio force-pushed the sbarrio/RUM-9617/base-benchmark-test-app-with-scenario-support branch from 094683c to a348fb9 Compare May 20, 2025 08:01

sbarrio force-pushed the sbarrio/RUM-9617/base-benchmark-test-app-with-scenario-support branch from a41dfe2 to 5b8bda0 Compare May 26, 2025 07:58

sbarrio commented May 26, 2025

View reviewed changes

Comment thread benchmarks/android/app/src/main/java/com/benchmarkrunner/MainApplication.kt

sbarrio commented May 26, 2025

View reviewed changes

Comment thread benchmarks/.env.alternate

sbarrio commented May 26, 2025

View reviewed changes

Comment thread benchmarks/ios/BenchmarkVitals/BenchmarkVitals.mm

sbarrio commented May 26, 2025

View reviewed changes

sbarrio marked this pull request as ready for review May 26, 2025 08:35

sbarrio requested review from a team as code owners May 26, 2025 08:35

datadog-datadog-prod-us1 Bot reviewed May 26, 2025

View reviewed changes

Comment thread benchmarks/src/testSetup/testUtils.ts Outdated

janine-c previously approved these changes May 26, 2025

View reviewed changes

sbarrio self-assigned this May 27, 2025

marco-saia-datadog previously approved these changes May 28, 2025

View reviewed changes

sbarrio dismissed stale reviews from marco-saia-datadog and janine-c via 1c4993c May 28, 2025 08:16

sbarrio force-pushed the sbarrio/RUM-9617/base-benchmark-test-app-with-scenario-support branch from 5b8bda0 to 1c4993c Compare May 28, 2025 08:16

sbarrio changed the base branch from develop to sbarrio/RUM-9457/automated-regression-detecting-benchmark-system May 28, 2025 08:16

datadog-datadog-prod-us1 Bot reviewed May 28, 2025

View reviewed changes

sbarrio force-pushed the sbarrio/RUM-9617/base-benchmark-test-app-with-scenario-support branch from 1c4993c to 1f907df Compare May 28, 2025 08:31

sbarrio force-pushed the sbarrio/RUM-9617/base-benchmark-test-app-with-scenario-support branch from 1f907df to 8403f09 Compare May 28, 2025 08:45

sbarrio requested review from janine-c and marco-saia-datadog May 28, 2025 08:46

sbarrio force-pushed the sbarrio/RUM-9617/base-benchmark-test-app-with-scenario-support branch from 8403f09 to ad1921f Compare May 28, 2025 09:29

Base benchmark app with scenario support

a0f8388

sbarrio force-pushed the sbarrio/RUM-9617/base-benchmark-test-app-with-scenario-support branch from ad1921f to a0f8388 Compare May 28, 2025 10:17

cdn34dd approved these changes May 28, 2025

View reviewed changes

janine-c approved these changes May 28, 2025

View reviewed changes

sbarrio merged commit 70bc798 into sbarrio/RUM-9457/automated-regression-detecting-benchmark-system Jun 2, 2025
6 checks passed

sbarrio deleted the sbarrio/RUM-9617/base-benchmark-test-app-with-scenario-support branch June 2, 2025 12:48

sbarrio mentioned this pull request Jul 30, 2025

[RUM-9457] - Automated regression detecting benchmark system #949

Merged

4 tasks

Conversation

sbarrio commented May 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Additional Notes

Review checklist (to be filled by reviewers)

Uh oh!

datadog-datadog-prod-us1 Bot commented May 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Datadog Report

Uh oh!

sbarrio May 26, 2025

Choose a reason for hiding this comment

Uh oh!

sbarrio May 28, 2025

Choose a reason for hiding this comment

Uh oh!

sbarrio May 26, 2025

Choose a reason for hiding this comment

Uh oh!

marco-saia-datadog May 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aleksandr-gringauz May 28, 2025

Choose a reason for hiding this comment

Uh oh!

sbarrio May 28, 2025

Choose a reason for hiding this comment

Uh oh!

sbarrio May 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sbarrio May 26, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cdn34dd commented May 27, 2025

Uh oh!

marco-saia-datadog left a comment

Choose a reason for hiding this comment

Uh oh!

sbarrio commented May 28, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sbarrio commented May 28, 2025

Uh oh!

cdn34dd left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

janine-c left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

sbarrio commented May 19, 2025 •

edited

Loading

datadog-datadog-prod-us1 Bot commented May 20, 2025 •

edited

Loading

marco-saia-datadog May 28, 2025 •

edited

Loading

cdn34dd left a comment •

edited

Loading