GSoC 2025 - Making ZeroMQ a first-class feature of concore

[saksham-gera]

General Discussion For GSoC 2025

[saksham-gera]

Hi @pradeeban ,

I hope you’re doing well! I’ve been actively contributing to Concore for about two months now and have been continuously studying its code, ideas, and vision. I’m really interested in making meaningful contributions.

If there are any feature implementations or improvements you’ve been considering in concore or concore-editor but haven’t had the chance to work on yet, I’d love to discuss them and see how I can help. Please let me know if there’s anything on your mind!

Looking forward to your insights.

Best regards,
Saksham Gera

[saksham-gera]

Hi @pradeeban , I've Some Questions regarding the project idea, understanding the expectation would help me writing a good proposal.

Question 1: Edge Identification for ZMQ Connections
How are edges identified for establishing ZeroMQ (ZMQ) connections between nodes? Is this based on "null nodes" in the graph, or is it a generalized approach where any two nodes (even in separate graphs) are connected via ZMQ if their edge follows a naming convention like 0x[PORT]_[LOGICAL_NAME]? For example, if two nodes in a single graph are directly connected with such an edge name, does that automatically imply a ZMQ connection, regardless of their placement in the same graph?

Question 2: Automating funbody/funcall Configuration Files
The funbody and funcall files currently serve as manual configuration files to define communication setups. Since these files primarily specify edge ports and communication direction (e.g., IN1 -> funcall indicating a receiving port), could this process be automated? For instance, can we generate these files dynamically by parsing edge metadata (ports, directionality, ZMQ vs. file-based communication) instead of requiring manual creation?

Question 3: Scope of the Project
Is the primary goal of this project to optimize ZMQ-based communication efficiency, or does it also involve enhancing the Concore Editor’s visualization of ZMQ connections? If editor integration is required, what specific improvements are expected? For example:

• Should ZMQ connections be visually distinguished from file-based edges in the editor?
• Are new node/edge types (e.g., "ZMQ nodes") needed to clarify the communication layer?
• Is there a need to display live connection status or performance metrics (e.g., latency) in the editor?

[pradeeban]

1. Our current approach is to assign 0 as the edge label in the current concore-editor and then interpret it in concore as zeromq. Similarly, 1-9 are currently assigned for shared-memory-based communication. So, if an edge is labeled with a string that starts with one of those 9 numbers, it is a shared-memory communication. Anything else is classic concore-file-sharing-based communication. We also ideally will need to represent the communication between nodes that are connected through Mediator. But I fear we are putting too much into our plate. You could list that as "Optional/If time permits"), to avoid over-promising.

2. Sure.

3. Latter. We need to optimize the visualization as well. This is a 350-hour project after all.

   Should ZMQ connections be visually distinguished from file-based edges in the editor?

   That can be neat. The current approach is merely using the first character to identify the zeromq vs shared memory vs file-sharing.

   Are new node/edge types (e.g., "ZMQ nodes") needed to clarify the communication layer?

   Edge types, for sure. I am not sure about the node types. Up to implementation and design choices, I think?

   Is there a need to display live connection status or performance metrics (e.g., latency) in the editor?

   Perhaps, an "Optional/If time permits?" Another edge type to consider would be the asynchronous communications. "Send now and expect a response later" type communications.

[saksham-gera]

@pradeeban thank you so much for clarifying.

[saksham-gera]

Hi @pradeeban ,

I’ve sent my GSoC proposal to your email (CCing all other mentors too) from sakshamgera1456@gmail.com. Could you kindly review it and share your feedback when you get a chance?

Thanks!

[pradeeban]

On it... Nicely written, and you know the project well. I don't think I have additional feedback at this time. But keep an eye for comments from other mentors - specifically from Mayuresh. Thanks.

[saksham-gera]

Sure sir, thanks for your review.
Looking forward to a long and successful collaboration.

[saksham-gera]

Your polite and positive feedback always boosts my confidence and lead me one step further.

[saksham-gera]

@pradeeban sir, I've Submitted My Proposal On GSOC Portal.

Thanks for you support and guidance :D

[pradeeban]

Awesome! I see it.

[saksham-gera]

Hi @pradeeban ,

I believe it would be best for me to continue focusing on Concore and put L4SBOA on hold for now. I feel that dedicating my time to a single project will allow me to delve deeper, gain a better understanding, and contribute more meaningfully. My goal is to refine my knowledge and make a significant impact on this project.

Please let me know your thoughts. I truly appreciate your guidance.

[pradeeban]

I don't know about your experience with Linux (in the kernel level, not as a regular user), networking, and L4S. Unless you have a strong interest in these fronts, it makes sense to stick with CONTROL-CORE. You have made the strongest possible proposal to CONTROL-CORE.

I remember your message months ago showing interest in working on L4SBOA.

The difference between CONTROL-CORE and L4SBOA is, CONTROL-CORE is a mature project. So, your work is going to be purely implementation - which is actually the most common case with most GSoC organizations. L4SBOA is a young, relatively new project. It has a heavy research component as we go into a trial-and-error approach.

[saksham-gera]

Yes, that’s exactly why I don’t have much knowledge about the kernel level. However, I did set up the initial process, such as configuring two L4S nodes by patching the Linux kernel. But then I shifted my focus to Concore because I realized that working on it would allow me to make a significant impact on the entire CONTROL-CORE project.

I also see the potential for long-term contributions in Concore. I believe that by making it a completely stable application, it can reach a much wider audience. Right now, it requires a deep understanding just to set it up, but I can envision a future where it becomes as accessible and widely used as MATLAB. My goal is to refine and simplify it so that it can serve many more people efficiently.

[pradeeban]

That is perfect. I love the vision.

[saksham-gera]

@pradeeban Sir, I'm so Excited for the Results, I don't know if I'll get selected or not, But had an amazing experience working with you.

Thanks!

[pradeeban]

We are all excited and waiting for the announcement from Google as well, in a couple of hours. Thank you for all your contributions. Greatly appreciated.

[saksham-gera]

Hi @pradeeban , I've Quantified the improvements after adding ZeroMQ in concore along with the primary goal of distributed execution, this will help us in writing the paper.

[pradeeban]

@saksham-gera This is awesome. Try to put these in the Overleaf document I shared with you, in parallel. That way, we will have a paper by the end of GSoC. (I will work on polishing it up and submitting).

[pradeeban]

Maybe also a good idea to include any scripts you use in the measurements in a new "measurements" folder in concore.

[saksham-gera]

@saksham-gera This is awesome. Try to put these in the Overleaf document I shared with you, in parallel. That way, we will have a paper by the end of GSoC. (I will work on polishing it up and submitting).

For sure, will start it soon.

[saksham-gera]

Maybe also a good idea to include any scripts you use in the measurements in a new "measurements" folder in concore.

That's a great idea, I'll start preparing documentation on measurements.