Implement BufferedTokenizer to return an iterable that can verify size limit for every token emitted - #17229

Merged

andsel merged 30 commits into

elastic:mainfrom

andsel:fix/bufftok_to_return_itereator

Jul 2, 2025

Commits on Jul 2, 2025

First rough idea of reimplementation to return an iterator
andsel
committed
9eab0c8
Exposed Java BufferedTokenizer under FileWatch module and adapted tests to use it
andsel
committed
7213acb
Moved string encoding logic to outer Ruby extension BufferedTokenizerExt class
andsel
committed
ef5ead0
Fixed flush behavior
andsel
committed
51d8edd
Minor, removed commented code
andsel
committed
950b8d0
Moved creation of iterable upfront in the constructor to be executed just one time
andsel
committed
32b05eb
Fixed tests to grab the failure on exceeded size limit on itereation once reached the next separator
andsel
committed
f4a9d9d
Fixed license headers
andsel
committed
3f5b82c
[Test] added test to verify buffer full error is notified not only on the first token
andsel
committed
e40da7f
Updated benchmark to consume effectively the iterator
andsel
committed
49e4d9c
[Benchmark] JMH report in milliseconds instead of nanoseconds
andsel
committed
84342a1
Added an isEmpty method to the iterable returned by BufferedTokenizerExt because it's expected in some use cases, like: https://github.com/logstash-plugins/logstash-input-file/blob/55a4a7099f05f293…
andsel
committed
d5bc543
Aligned with main
andsel
committed
57b6c0c
Removes a test that's not anymore valid. Not int math happens in code, but an OOM error is thrown from JDK libraries if an int overflow happens.
andsel
committed
c32578d
- inverted BufferedTokenizerExt§IterableAdapterWithEmptyCheck#isEmpty
andsel
committed
7111559
Avoid to executed double token scan, in hasNext and next methods
andsel
committed
8e9d8e9
Emptied StringBuilder accumulator in flush and protected the state of DataSplitter with synchornized so that can be used in multithreaded contexts
andsel
committed
74e8993
[Test] Added test to verify that tokenizer condition empty is true when also unterminated token parts are present in the buffer
andsel
committed
81fb197
Switched tokenizer implementation's empty to consider also the unterminated token in the buffer
andsel
committed
dc3d7b2
Minor, removed commented code
andsel
committed
e2cd80f
[Test] Added test to verify the avoidance of infnite accumulation
andsel
committed
92c3437
Inloined PR #17293 to check for potential OOM condition during the append to the DataSplitter
andsel
committed
602bf94
Fixed matchNextSeparatorIdx to raise the flag to start dropping on write side also when there isn't yet done any read
andsel
committed
0051a13
[Test] fixed test, given the flush call the accumulator is emptied so no token to throw the exception
andsel
committed
b2c44b8
Updated append method to avoid OOM accumulation without that token listener needs to be engaged
andsel
committed
1f4193a
Removed unused code and expanded * imports in tests
andsel
committed
983a5ea
[Test] remove commented code in test
andsel
committed
1c67ff6
Lifted a else branch into the first if statement
andsel
committed
6976451
Removed not necessary condition, because if currentIdx is unset (min int) then the indexOf clamp it at least at 0. This test is useful only when the first delimiter is not yet found
andsel
committed
b3d4089
Removed redundant if statement
andsel
committed
1162882

Navigation Menu

Uh oh!

Implement BufferedTokenizer to return an iterable that can verify size limit for every token emitted - #17229

Implement BufferedTokenizer to return an iterable that can verify size limit for every token emitted#17229
andsel merged 30 commits into
elastic:mainfrom
andsel:fix/bufftok_to_return_itereator

Commits on Jul 2, 2025

First rough idea of reimplementation to return an iterator

Exposed Java BufferedTokenizer under FileWatch module and adapted tests to use it

Moved string encoding logic to outer Ruby extension BufferedTokenizerExt class

Fixed flush behavior

Minor, removed commented code

Moved creation of iterable upfront in the constructor to be executed just one time

Fixed tests to grab the failure on exceeded size limit on itereation once reached the next separator

Fixed license headers

[Test] added test to verify buffer full error is notified not only on the first token

Updated benchmark to consume effectively the iterator

[Benchmark] JMH report in milliseconds instead of nanoseconds

Added an isEmpty method to the iterable returned by BufferedTokenizerExt because it's expected in some use cases, like: https://github.com/logstash-plugins/logstash-input-file/blob/55a4a7099f05f293…

Aligned with main

Removes a test that's not anymore valid. Not int math happens in code, but an OOM error is thrown from JDK libraries if an int overflow happens.

- inverted BufferedTokenizerExt§IterableAdapterWithEmptyCheck#isEmpty

Avoid to executed double token scan, in hasNext and next methods

Emptied StringBuilder accumulator in flush and protected the state of DataSplitter with synchornized so that can be used in multithreaded contexts

[Test] Added test to verify that tokenizer condition empty is true when also unterminated token parts are present in the buffer

Switched tokenizer implementation's empty to consider also the unterminated token in the buffer

Minor, removed commented code

[Test] Added test to verify the avoidance of infnite accumulation

Inloined PR #17293 to check for potential OOM condition during the append to the DataSplitter

Fixed matchNextSeparatorIdx to raise the flag to start dropping on write side also when there isn't yet done any read

[Test] fixed test, given the flush call the accumulator is emptied so no token to throw the exception

Updated append method to avoid OOM accumulation without that token listener needs to be engaged

Removed unused code and expanded * imports in tests

[Test] remove commented code in test

Lifted a else branch into the first if statement

Removed not necessary condition, because if currentIdx is unset (min int) then the indexOf clamp it at least at 0. This test is useful only when the first delimiter is not yet found

Removed redundant if statement