Welcome to DBMS-Benchmarker Discussions!

[perdelt]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we
  build together 💪.

To get started, comment below with an introduction of yourself and tell us about what you do with this community.

[Korg95]

Hi @perdelt ,

Nice to meet you. Enjoying your tool so far, its been a blast to use! Thank you for your work :)

[perdelt]

Hi @Korg95 !

Welcome and nice to meet you!

Thanks for the kind words. Could you tell us something about what your use case is? Which database system are you interested in? What is the workload? Are there any options or parameters that you particularly use (or miss)?

Thank you and welcome again!

[Korg95]

Hi @perdelt ,

Apologies for the late reply!

Currently I am using the tool to test online analytical workloads across 3 databases. Vertica, Druid and Clickhouse.
It's a bit interesting compared to what classical workloads go through this tool in the sense that I have to test for low latencies < 1-2 seconds and around 60-100 QPS with randomized query execution to simulate daily users executing a variety of tasks in a concurrent manner. This I hope showcases the effect of noisy neighbors if any.

So far comparing the databases is a breeze when working with lower amounts of threads, I however run into issues when running 60 threads or more with JVM OOM issues in relation to the variety of JDBC drivers that using. It hopefully should be solvable and I can reach back with more info on that side once I get more time.

What I've found more interesting is figuring out how to simplify the merging of results. When testing with parallelism it results in many subfolders being generated with results that have connections renamed according to numbered suffixes assigned. I can't seem to merge those results into a single result. This means that when comparing results each of these connections is considered a different DBMS. It is fairly easy myself to merge these adhoc but I was wondering if this is a case of user error or some misunderstanding of how the tooling works.

Here is my workflow:

python .\DBMS-Benchmarker\benchmark.py run -b -d -w connection -f ./config -r ./results -p 10 -pp -ssh 1
python .\DBMS-Benchmarker\benchmark.py read -r ./results/1721340615 -e yes
python .\DBMS-Benchmarker\merge.py -r ./results/ -c 1721340615 #Does not yield merging of results as expected

Note here I rely on -ssh query randomization to give me some sense of mixing up what queries run concurrently with each other. Since the tool processes randomized query by query I rely on 10 processes to ensure that different workloads execute at the same time, collecting a large enough sample size of runs per query.

I also set a query cool down to help rate limit the queries executed so that I can get closer to 60 QPS. 60 QPS / 10 threads = 6 QPS per thread = 1 second / 6 QPS = 0.166 recurring cooldown per query. This logic might be flawed though, where I should only apply a cooldown if I am getting too many failures and not finding the QPH that your tool outputs.

Small note: I think that for this case, QPS is more appropriate than QPH but that's a very small note :)

(PS if you want me to start a discussion let me know and I can move it there)

Kind regards,

Korg95

[perdelt]

Hi @Korg95 ,

Thank you for sharing your experiences!

I see that you have become familiar with the tool well.

As short quick answers:

Yes, please feel free to open an issue if you want to discuss specific points.

Have you tried this?

python .\DBMS-Benchmarker\benchmark.py run -b -d -w connection -f ./config -r ./results -p 10 -pp -ssh 1 -e yes

So integrating the generation of the evaluation into the execution directly? This should also do the merging of results.
I will test why the single calls in three parts do not work.

PS: You might want to try the silent mode -vn. This suppresses most of the output and can be useful if you have a lot of fast queries (it saves a bit of overhead).

[Korg95]

I'll give it a spin and report back in a discussion thread moving the context there. Cheers @perdelt