Repository navigation
support for long lines in filter files - #123
Open
teruteru128 wants to merge 2 commits into
Open
teruteru128 wants to merge 2 commits into
teruteru128 wants to merge 2 commits into
Conversation
Read filter file lines into a growable buffer instead of a fixed 128 byte one, so that long lines (PCRE2 patterns in particular) are no longer silently split into several filters.
A filter file written on Windows kept a trailing CR when read on other platforms, which became part of the filter itself. A lone CR is still left alone so that it cannot split a filter in two.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When many patterns are joined into a single alternation, for example
(?:pat1)|(?:pat2)|..., so thatDOFILTERperforms onepcre2_match()call per key instead of one per filter, the resulting line easily exceeds
the 128 byte buffer
loadfilterfile()read into. My own 522 byte filterfile was cut into five fragments, none of which compiled, leaving zero
filters registered while reporting five confusing "unmatched closing
parenthesis" errors. A plain filter file is worse off still: the
fragments are valid base32, so they are registered as filters that were
never asked for.
The first commit reads each line into a buffer that grows as needed, so a
line of any length ends up as a single filter. The second one strips the
CR of CRLF line endings, which otherwise became part of the filter when a
filter file written on Windows was read elsewhere; a lone CR is left
alone so that it cannot split a filter in two.
Behaviour is otherwise unchanged: empty lines,
#and//comments and alast line without a terminating newline are handled as before.
Tested with lines of 125-3000 characters, lines ending exactly at a buffer
boundary, files with and without a trailing newline, CRLF files, and files
containing NUL bytes (no unbounded buffer growth); built with
-Wall -Wextra -pedanticwith no new warnings.