How do you implement the JVMmanage() API and where do you invoke it?

[Zhang-Xiaoda]

Hello,

I'm very interested in this project, and have read you paper. I want to carefully read your implementation about Allocation Module. It seems that the JVM part is not in the current repository. Could you please specify how do you implement the JVMmanage() API inside OpenJDK, and where do you invoke the API in current Spark?

thanks,
Xiaoda

[LibinLiu0189]

Thanks for your interests. As we stated that we refer to the paper ElasticMem which focuses on elastic memory allocation, we follow its implementation and install it (https://github.com/jingjingwang/elastic-mem) in our system. JVMManager() is a socket-based API in Elasecutor/ElasticMem to enable interaction with JVM. It only reads profiles (also called instructions in ElasticMem) from Allocation Module. For its detailed implementation, you can refer to ElasticMem's paper and source codes. It had been open-sourced and thus we didn't invole it in our project codes. Allocation Module directly sends corresponding profiles to it when ElasticMem installed and started, which is unrelated to Spark implementation.

[Zhang-Xiaoda]

Thanks for your answer, Libin.

I still have a question about your paper. When dynamic reprovisioning resources for executors (JVMs), it seems the module only supports growing JVM's memory limit, but lacks of shrinking the limit. When multiple executors belonging to different jobs are running on the same machine, and they are all determined (by the mechanism) to grow their limits, it seems impossible to enforce the decision because of the machine memory capacity.

Did I misunderstand the mechanism? thanks.

[LibinLiu0189]

Let me make it clear. Firstly, we can reduce the memory assigned to a JVM by performing GC and recycling space. Secondly, even the memory usage grows over time, our algorithm has considered the executors' resource demands over time (how it grows). In section 3.2.1, we have analyzed that when Elasecutor assigns executors to machines, it has considered the whole resource demand time series (when it grows to peak usage) and the available resource time series on the machine. Our algorithm tries to pack executors together without exceeding the machine's memory capacity at any time.

[Zhang-Xiaoda]

Hi Libin. I recently read the code about this project. I still have questions.

Where is the Scheduling Module and Allocation Module is the code. I have checked spark/deploy/master/Master.scala and spark/deploy/worker/Worker.scala, but did not find the modules. In the ./Subsystem/ , I only found the base code of controlling cgroup. Where is the component of communicating with JVM(), or you just reuse the code of ElasticMem project and simply invoke the socket-based API?

I am a little confused with the current code, thanks for your replying.

[LibinLiu0189]

Thanks for your questions. For the scheduling part, please refer to the /scheduler folder. We already separated it from the spark source codes. For the allocation part, it includes controlling cgroups and borrows the socket API in ElasticMem to implement memory allocation inside /subsystem, Thanks.

[Zhang-Xiaoda]

Hi, Libin. I read the codes in ./Subsystem folder, but did not find the lines calling the APIs in ElasticMem. If you borrow the existing implementation of ElasticMem, you should invoke it, right? Did you miss some code in the current repository. Second, I check the code in /spark/scheduler, and realize the current implementation is exactly the origin of Spark. I did not find your scheduler implementation. Did I miss some key lines, or just because you forgot to upload it?

Any help in specifying the code details is appreciated. thanks for you patience, :)

[LibinLiu0189]

Thanks for your careful reading. Yes, I mean we only modified the socket API in ElasticMem which is not included in the current repo now as I said. Then, for scheduling, sorry for my misunderstanding. Now I understand your concern about the BFD scheduling logic implementation. In our implementation, we didn't modify the scheduling function (e.g., canlaunchexecutor) in spark executor scheduler source codes, As our paper written, it is a separate part to produce the executor launch decision to the executor launch part to force our scheduling scheme (we use our calculation decision as its inputs). I will update this separate part then and notify you, thanks again for your careful reading our codes.