[bruneliere] Scalable Views on Model Resources

[grammarware]

Submitted by @fmdkdd to https://github.com/modelsconf2018/artifact-evaluation/tree/master/bruneliere

Paper: http://fmdkdd.free.fr/scalable-views.pdf

[mherzberg]

I'm currently trying to reproduce the benchmarks using the docker image as described in https://github.com/atlanmod/scalable-views-heterogeneous-models#running-the-benchmarks-with-the-docker-image. I'm using a Ubuntu 18.04 VM with 8GB of RAM + 32GB of swap. When trying to reproduce the load-view benchmark (3.), I get the following errors:

Benching Load view with NeoEMF trace / NeoEMF weaving model of size 1000000... (0 warmups/1 measures)

== Measure 1
[INFO]  10:44:02,217 - DirectWriteBlueprintsStore created
[INFO]  10:44:02,218 - PersistentResource created
[INFO]  10:44:02,306 - Creating DirectWriteLongListSupport Store
[INFO]  10:44:02,306 - DirectWriteBlueprintsStoreLongListSupport created
[INFO]  10:44:02,307 - PersistentResource loaded: neo-blueprints:/root/workspace/models/neoemf-trace/1000000.graphdb
[INFO]  10:44:03,994 - DirectWriteBlueprintsStore created
[INFO]  10:44:03,994 - PersistentResource created
[INFO]  10:44:04,892 - Creating DirectWriteLongListSupport Store
[INFO]  10:44:04,892 - DirectWriteBlueprintsStoreLongListSupport created
[INFO]  10:44:04,892 - PersistentResource loaded: neo-blueprints:/root/workspace/views/neoemf-trace/weaving-1000000.graphdb
Load view [133147ms] [1027MB]
Get first element [0ms] [1028MB]

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "GC-Monitor"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "main"

When the crash occurs, only around 4GB of RAM are in use. Could I be missing an appropriate Java heap size parameter or something similar?

[Bitico]

Hello,
sorry for posting git problems but when I trying to clone the repository I get the following:

Cloning into 'scalable-views-heterogeneous-models'...
remote: Counting objects: 920, done.
remote: Total 920 (delta 0), reused 0 (delta 0), pack-reused 920
Receiving objects: 100% (920/920), 478.27 KiB | 539.00 KiB/s, done.
Resolving deltas: 100% (620/620), done.
Checking connectivity... done.
Downloading benchmarks/models/petstore-java.xmi (4.6 MB)
Error downloading object: benchmarks/models/petstore-java.xmi (d8da5ca): Smudge error: Error downloading benchmarks/models/petstore-java.xmi (d8da5caf2cee64b2f2c3456db5e1ee539af1b99a9ec3f2f21e0cbc476eaceec7): [d8da5caf2cee64b2f2c3456db5e1ee539af1b99a9ec3f2f21e0cbc476eaceec7] Object does not exist on the server: [404] Object does not exist on the server

Errors logged to /Users/francesco/git/TOOL_EVALUATION/scalable-views-heterogeneous-models/.git/lfs/logs/20180717T153318.506536329.log
Use `git lfs logs last` to view the log.
error: external filter git-lfs smudge -- %f failed 2
error: external filter git-lfs smudge -- %f failed
fatal: benchmarks/models/petstore-java.xmi: smudge filter lfs failed
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry the checkout with 'git checkout -f HEAD'

[fmdkdd]

@mherzberg Yes, you may want to up the maximum heap size. I've changed the default to 8G and updated the image and the README. (You shouldn't have to regenerate the models after updating the image).

@Bitico Looks like a mishandling of git-lfs on our part. I pushed the missing files, and this should now be fixed. Thanks for pointing that out!

[stsauer]

We are still having similar problems on our side like mherzberg when reproducing with docker. Even with the new image.

[fmdkdd]

@stsauer OOM exceptions you mean? I tested the new image and was able to run the load-view benchmark with the default settings. But I'm running Docker on my Linux box directly, not inside a VM.

Could you give more details on your setup?

You can try adjusting the heap size as described in the README, or you can also use the -s argument to not include the 10^6 model, which will require much less resources overall.

[mherzberg]

I now call the benchmark like this: docker run ... -j "-XX:+UseConcMarkSweepGC -Xmx16g" load-view
I don't get OOM exceptions anymore, but the NeoEMF 10^6 benchmark is running for over an hour now. I expected it to finish a while ago, given your values in the paper and the fact that the 10^5 one took 4 minutes. The benchmark still refuses to use more than ~4GB of RAM, so I suspect I'm just entertaining the GC now. Any more ideas? The last bit of output:

[INFO]  19:05:03,247 - PersistentResource created
[INFO]  19:05:05,901 - Creating DirectWriteLongListSupport Store
[INFO]  19:05:05,901 - DirectWriteBlueprintsStoreLongListSupport created
[INFO]  19:05:05,902 - AutocommitStoreDecorator created
[INFO]  19:05:05,902 - AutocommitStoreDecorator chunk = 50,000
[INFO]  19:05:05,953 - PersistentResource saved: neo-blueprints:/root/workspace/./views/neoemf-trace/weaving-1000000.graphdb

[fmdkdd]

The benchmark still refuses to use more than ~4GB of RAM

That's unexpected. Passing the argument to the JVM should remove that limit.

the NeoEMF 10^6 benchmark is running for over an hour now. I expected it to finish a while ago, given your values in the paper and the fact that the 10^5 one took 4 minutes.

Yes, that's clearly too long. My guess is that running the image inside a VM adds considerably to the overhead. There's a lot of I/O and memory loads in this benchmark.

Also, we got our numbers on an NVMe SSD, so if you are running on HDD, expect the numbers to be much higher.