Integrate a streaming gRPC client

[edeandrea]

This originated by an idea from @krickert in quarkiverse/quarkus-docling#95

@ThomasVitale what do you think? Basically create a protobuf schema around what we have and expose a gRPC interface that talks to the docling-serve api?

If we do it I think we'd introduce a new docling-serve-api-grpc module or something like that.

[krickert]

I already wrote this but I have my own interface for my project but I would adjust it to work with this interface. I could get it done in a day or two

So no harm in trying to get out, right?

[ThomasVitale]

Just to make sure I understood the scope :) Are we talking about building a gRPC Server that wraps the Docling HTTP API, as well as defining a gRPC Client that can sends calls to that wrapper?

[krickert]

yes! 100%. And the interface will be a 1:1 mapping of your own. I'll have one additional call that is streaming too. I already wrote it, just need to define the client.

[edeandrea]

Why would we write a grpc server that just acts as a proxy to a rest server?

I could potentially see the benefit of a grpc wrapper around the client api, but I'm not sure of the value of a full blown grpc server? Unless I'm missing something?

At the end of the day it wouldnt be "full streaming" because docling-serve itself doesn't have full streaming. Just submit-and-wait, which is what we're already doing with the async api.

[krickert]

I'd love to make it fully streaming - that's not a problem either (I'd have to do this in python - that's not a big deal). There's still a big advantage to introducing a grpc interface, even if it's just a wrapper. Having a grpc protobuf record that's storage friendly and strongly typed OOTB that can interact with other grpc tools can be a huge win when it comes to records management and S3 storage.

But the best arch would be to do this in the python project itself - which would probably be easier than doing this in java.

Most important - a python version can be true streaming.

Should I open up an issue there instead?

[krickert]

Opened docling-project/docling#2953 LMK if either of these would work

[edeandrea]

I 100% agree that it would be best if docling itself (or docling-serve, or some other sub-project of docling) did this on the server side.

For the client side, having a gRPC implementation in Java could be interesting. For now, it could just be a wrapper around the REST api.

I would just want to try and figure out a way for it to not be a maintenance headache. Let me put my thinking cap on for a bit. @ThomasVitale thoughts?

[krickert]

I'd recommend we make a branch, I'll dump out some code, we test it and see. If we don't like it - throw it out and go back to the drawing board. Code is cattle :)

Personally, IMHO this is best to do on the docling server - then the java client doesn't even need to be coded, just a quick near codeless extension in quarkus to have mutiny stubs and some sample integration tests so people can watch their AWS bills go up to parse PDFs.

That being said, if we get any pushback from the docling folks then we can try a wrapper here. I already have the server "done-ish", just would need to convert it somewhere. It's really just a massive mapping exercise

[edeandrea]

We can't go directly to Quarkus. This library exists because we need a framework-agnostic Java client. That is the whole point of Docling-Java.

The Quarkus extension integrates the API in the Quarkus way. Apache Camel integrates it the "camel" way. Spring boot integrates it the "spring" way.

The best approach in my opinion would be a grpc version of docling-serve, where docling itself "owns" the schema, like it does today with the OpenAPI schema. Then we can build a grpc client for it here.

But rewriting the docling server itself to expose a grpc interface honestly is outside the scope of this project.

I'm not opposed to building a grpc wrapper around the rest client (for now) and potentially refactoring it when/if a "real" grpc server implementation arises.

[krickert]

We'll need to just do a minor tweak to have a streaming call in docling main though - that part is key. But baby steps is fine here - people aren't that open to streaming calls (or call things streaming that aren't) yet.

I have the grpc streaming issue open with the main docling line - I can evangelize that for a bit on another thread while we do the wrapper.

but wherever you think it's good, I'll be glad to shift the code and have us scrutinize it for whatever changes we want to make in the contract. It's a pretty straightforward conversion -

[edeandrea]

If you'd like to propose/do a PR for a new module inside the docling-serve folder at the root of this project I would certainly consider it.

For now it would have to just be a wrapper around the docling-serve-api module. Honestly I don't have any say in what they do in the main docling project. Its a completely different team with different sets of people.

We'd also probably want to keep the model/API similar to the rest api for consistency, which includes the docling-core module.

[krickert]

Submitted PR #329 -

• full implementation - no stone left unturned
• defensive to prevent deep nesting NPEs
• following grpc standards
• maintained a 1:1 mapping of all REST standards
• Following non-breaking naming conventions as well, but they do not affect the underlying functionality.