Message max size

[tondacrha]

Hi. I wanted to make sure I understand the "Design Overview" correctly. Is it true I cannot use Bruce for a use case where messages of size over ~426KB needs to be send to Kafka?

If I want to overcome this limitation I need to cut the big messages into smaller once prior to the to_bruce call, is that correct?

[dspeterson]

Yes that's correct, although in the near future I intend to add support for stream socket input (UNIX domain stream sockets and local TCP). This will allow larger messages, and make Bruce available to clients that cannot easily write to a UNIX domain datagram socket. I'd say chances are pretty good that the feature will be available in the next couple of months, although I can't make any promises.

[dspeterson]

One thing to keep in mind is that Kafka imposes a limit on the size of a single message (message.max.bytes as documented here: https://kafka.apache.org/08/configuration.html ). Although this limit is configurable, increasing it beyond a certain point may have performance implications. If you want to send messages much larger than 1 Mb, you may want to check the Kafka mailing lists to see if increasing message.max.bytes may have unintended effects.

[tondacrha]

Thanks for reply. Looking forward to the stream sockets. It's not that I want to constantly send large messages to Kafka yet I need to plan for relatively small amount of big messages alongside the majority of small messages.

[dspeterson]

The stream socket input feature is now available in Dory, Bruce's successor. See https://github.com/dspeterson/dory for details.