Is this library able to load pods from babushka's pod-registry?

[judepayne]

I am looking for a bridge from python to Clojure (which I have written and could expose as a pod). Is this one of the intended uses of this library?

[Rohan2002]

Hello @judepayne

Thanks for your question.

Unfortunately no, this library acts as a bridge only between two python processes. As you can see this function send_data (src) does interprocess communication with another python process.

[judepayne]

Thanks @Rohan2002. Looking more closely, this feels like never a project goal? Certainly doable though! :)

[Rohan2002]

Actually this project was inspired from Babashka pods - a Clojure-to-Clojure bridge. Similarly, this project applies the same paradigm to Python—a Python-to-Python bridge using Babashka pod's pod-like architecture.

Eventually, I’d love to extend this idea to enable interoperability between Python and Clojure—allowing Clojure functions to be called from Python, and vice versa.

From a high-level perspective, here’s how we might approach this:

1. Create a Babashka PyPod with a bb.edn file to declare pod-specific dependencies.
2. Within the PyPod, implement a mechanism that loads Clojure modules into the Python runtime, exposing them as callable fuctions within the Python namespace.
3. To call a Clojure function like clojure_foo from Python, we send a serialized payload to the Babashka PyPod. The PyPod evaluates it and returns the result via inter-process communication (IPC). Since this IPC mechanism already exists in the PyPod library, we’d mainly focus on extending its capabilities.

What do you think about this approach @judepayne ? If you are interested, we could discuss on collaborating.

Thanks

[judepayne]

I like the idea! Not much of a python programmer - yet, so that you are warned :)

I did contribute to the pods library a couple of years ago, and so know it fairly well. Some of the pods in the pod registry are already not written in Clojure; pretty sure than there's at least one written in go.

I think the core of this idea is to port babashka.pods to python. Correct me if I'm wrong.
If so, do you think it is worth continuing with this library or starting a new one that is a faithful port of babashka.pods? The goal is pypods is a little different.

On your points:

1. Do you mean that someone uses PyPods would have a local bb.edn file? I like this idea (as long as it doesn't break a python idiom I don't know of..)
2. Like your create_a_function fn already does, but sending a message to the loaded pod. Absolutely.. should stick close to babashka.pods on this.
3. In order to be compatible with existing pods from the registry, we'd need to implement build the library around bencode and offer the same set of payload formats: json, edn and transit+json. There are existing bencode, edn and transmit libraries in the python ecosystem.
4. Some pods use metadata attached to args & return values. This would need to be accommodated.

[Rohan2002]

Thank you for the comment @judepayne

Oops — I misunderstood the requirement earlier. Please correct me if I'm wrong, but you're asking whether it's possible to support the Babashka pod protocol using Python, correct?

If so, then yes — the goal of PyPods is a bit different, and supporting your use case would likely require a separate project.

Initially, my idea with PyPods was to create a Clojure (or Babashka)-specific pod that communicates with the Python runtime using a custom protocol designed for PyPods.

If we went with PyPod's idea then to answer your question regarding point 1 and point 2:

Within the Python runtime, the developer would instantiate a PodLoader object that specifies which pod to load. Internally create_a_function is called to send a request execution of the function to the pod.

For the pod itself:

1. The developer creates a pod folder within their project.
2. Inside that folder, they add the pod’s source code. If the pod is written in Go, it would include a go.mod file; if in Clojure, a deps.edn file.
3. Finally, the developer adds a file that enables communication using the PyPods protocol. For example, see this pod template, which defines a PodListener class. This class is responsible for receiving and executing functions on behalf of the calling pod. This file was designed for python-specific pods only, so a new one needs to be created for go or clojure.

For some context, the idea behind PodListener is inspired from babashka pod too (src: /pod-lispyclouds-sqlite pod). Here borkdude creates a protocol to communicate with the client via bencode format. I simply used bson.

Regarding the new project - yes I am OK to build the new projects with incorporating the ideas from your point 3 and 4.

Let's say the current project's python runtime wants to add the following babashka pod: pod-lispyclouds-sqlite. These are the following steps maybe (please correct me if I am wrong):

The python runtime would need to send a message following the babashka pod protocol.

1. The first message that the python runtime will send is {"op" "describe"} to the pod via bencode format.
2. The babashka pod will reply via either json or edn or transit+json
3. Python runtime will parse that reply - so we need to add parsing support for son or edn or transit+json
4. The output of {"op" "describe"} contains the namespace info of the pod - which needs to be loaded within python. Maybe we could use something like create_a_function (src) to do that.
5. Similar steps would apply to the invoke and shutdown operation.
6. Regarding the load-ns operation - honestly I still don't get the idea about the Deferred namespace loading that they implemented.

Would you be down for a quick call? My email is rohandeshpande832@gmail.com and I am based in Tokyo. Anytime early morning (7am-9am) Japan time or evening after 7pm Japan time works for me.

[judepayne]

Hi, I will email you to set up a call.
On the Pypods idea I like it; if I understand you correctly it does not require that the pod run as a self contained binary, so should be simpler to implement for the pod developer?
My main interest is the port of babashka.pods to python. We can discuss if it should be a fairly faithful port, or more of a rewrite. I agree to your points. I understand the use of deferred namespace, but it maybe feels like a bit of a corner use case.
I would add a point about the python developer experience of using a pod (via this new library we are talking about); the thing I like about pods is that it feels very natural/ integrated into the developer experience; almost* like you just require a library and you're ready to use it, as opposed to calling a REST service or using an RPC library. I think a goal of this new project should be to similarly integrated into the normal python experience of importing (if possible).

*almost, but not quite: you still need to call load-pod. I did have a conversation with borkdude about ideas for getting rid of that step (since he controls babashka.deps), but nothing transpired.

[Rohan2002]

Hello @judepayne

Yes you are right. A pod is not a self-contained binary.
However, a pod does have its own virtual environment that contains a python interpreter. So when the pod gets a command from the client and executes a function - it calls the python interpreter located within the pod to execute the function. The client runs using the project's python interpreter.

As you can see, we are essentially make a separation at the process-layer (client and pod). Both processes talk to each other via IPC.

"I would add a point about the python developer experience of using a pod (via this new library we are talking about); the thing I like about pods is that it feels very natural/ integrated into the developer experience" - agreed.