[burgueno] Expressing Confidence

[grammarware]

Submitted by @lolybc88 to https://github.com/modelsconf2018/artifact-evaluation/tree/master/burgueno

Paper: http://www.lcc.uma.es/%7Eloli/downloads/MODELS18.pdf

[zolotas4]

Hi,

For the first two case studies (Surveillance and Social Media) I am able to run and get the results described in the paper, I am not able to do the same for the 3rd case study (Smart Home). For the latter, only one scenario is run (instead of 4 reported in the paper) for which the initial model elements reported from the code are 3997 (the 4 cases described in the paper are 2K, 4K, 6K and 8K). Am I doing something wrong?

Regards,
Thanos

[grammarware]

Ping @lolybc88

[lolybc88]

Ops, my mistake. You're doing it right. It's just that the main method is calling the run() method just once. I'll commit the new version in which the call is made four times with different parameters and you'll get the results we claim in the paper as for the other case studies.

[zolotas4]

Summary
The authors provide artefacts supporting their work on expressing confidence in models and model transformations. More specifically, they provide the code to re-run the experiments presented in the paper for the 3 case studies and the infrastructure for the transformations and the types containing uncertainty (for Integers, Booleans and Real).

Is the artefact consistent with the paper?
The artefacts are consistent with the paper. All the 3 examples described in the paper are provided along with the needed infrastructure. By re-running the experiments, the results (in terms of time performance) were slightly off but this is justified by the fact that my machine compared to the one used in the paper is slower. The overhead between the compared approaches was as described in the paper.
(In the 3rd experiment the initial model sizes presented in the paper are 2000, 4000, 6000 and 8000 while in the experiments their size is 2012, 4004, 5996 and 7988. I don't believe though that this has any impact at all at the presented results as the difference in size is small)

Is the artefact as complete as possible?
Yes, the artefacts are as complete as possible.

Is the artefact well-documented?
The artefacts are documented through a readme file in the Github repository. I was able to run everything on the first try (except the replication of the 3rd experiment where some iterations were missing but this was fixed by the authors immediately).

Is the artefact easy to (re)use?
To replicate the experiments one just simply need to run 6 java programs in total. Main methods used as runners were provided by the authors making the whole process quite easy. Regarding the approach itself, one should of course have some familiarity with the transformation tool used and read the paper for understanding the details and how to re-use. However, they provide 3 use cases which can also help in understanding how to use the proposed approach.

[manuelleduc]

Summary
The submitted artifact is part of a paper presenting an approach to the integration of the confidence on model elements.
To evaluate the approach, a performance measurement on three use cases is proposed.
For each use case, a transformation is applied twice on generated models of increasing size, once without the introduced concept of uncertainty, and once with the introduced concept.
This way, the overhead of the introduction of the concept of confidence during model transformation can be evaluated by comparison of the execution times.

Documented
The documentation is simple and straightforward.
The provided explanations are clear and allows the artifacts to be exercised easily.

Consistent
The submitted artifacts are relevant and exhaustive in regard to the evaluation discussed in the paper.
They contribute to the main results of the paper.
The result obtained from running the evaluations on my own computer yield results that lead to the same conclusions as those of the paper.

Complete
Everything relevant to the paper is included.

Exercisable
Following the instructions is easy and I did not encounter any issue during the execution of the use cases.
The artifacts follow eclipse and java standards.
The results of the executions are easily linked to the results displayed in the paper.

Availability
The artifacts of the experiment are available in the ReMoDD repository.

Minor comments
The documentation can be improved by integrating a explicit inventory of the provided artifacts, as advised here https://www.acm.org/publications/policies/artifact-review-badging.

[TheoLeCalvar]

Summary

This artefact is part of a paper aiming at integrating uncertainty in model transformation.
The approach is evaluated in three benchmarks.
Each is run with and without uncertainty so that overhead of the approach can be measured.

Is the artefact consistent with the paper?

The artifact provided are consistent with the paper.
Three of the provided projects directly correspond to examples used in the paper.
Running time on my machine are different but yield to similar overheads.

I noticed that on my machine overhead for the Smart Home example is significantly smaller than in the paper.
It is around 5 for all tested sizes.

Is the artefact as complete as possible?

Yes, the artefact comes with everything needed to evaluate results presented in the paper.

Is the artefact well-documented?

The artefact is very well documented.
The project comes with a README that clearly describes how to install and use every case study.

Is the artefact easy to (re)use?

Initial setup was really easy. Importing the project into Eclipse is enough as there is no need for any external.
Thanks to very precise instructions I had no issue replicating results presented in the paper.

Side note

In the drone case study, I see that the modified coordinate of the drone is added to the modified elements in the uncertain transformation but not in the plain one.

[grammarware]

Dear @lolybc88,

Based on all the comments and the reviews provided by the members of the Artifact Evaluation Committee of MoDELS 2018, we have reached the conclusion that this artifact conforms to the expectations and is hereby approved. Please use the badge instructions page to add the badge of approval to your article, and add the link to the ReMoDD entry with URI http://remodd.org/node/581 to the camera ready version of the paper.

Thank you very much for putting extra effort into the preparation and finalising of the artifact. If any of the comments above are still not addressed, please try to accommodate them before the conference.