Assertion failure in `parse_perl_extension()` when `newline_alt` is combined with `no_except`

[CrJyyy]

Bug Report

A 6-byte pattern makes basic_regex::assign() abort on
BOOST_REGEX_ASSERT in basic_regex_parser<>::parse_perl_extension()
(basic_regex_parser.hpp:2613),
when the syntax options combine newline_alt with no_except.

no_except is documented as the way to ask Boost.Regex not to fail hard on an invalid
expression, so aborting the process on one is the opposite of what the flag is for.

Scope, up front: with NDEBUG defined the assertion is compiled out and nothing bad
happens — assign() returns and status() reports an error, which is the correct
behaviour. So this affects assertion-enabled builds only (debug builds, sanitizer builds,
OSS-Fuzz, anything using BOOST_ENABLE_ASSERT_HANDLER). I did not find any
memory-safety consequence in a release build; see "Release builds" below.

Version

Reproduced against boostorg/regex master, commit a640597 (2026-05-22), Boost.Regex v5.
I have not tested tagged releases, but the code at the assert is unchanged on master today.

Reproducer

Self-contained — needs only this repository, in standalone mode, no other Boost libraries:

#include <boost/regex.hpp>
#include <cstdio>
#include <exception>

int main()
{
    // 6 bytes:  ( ? : \ R 0x0c
    const char pat[] = { '(', '?', ':', '\\', 'R', '\x0c' };

    const boost::regex_constants::syntax_option_type flags =
        static_cast<boost::regex_constants::syntax_option_type>(
            boost::regbase::newline_alt | boost::regex_constants::no_except);

    boost::basic_regex<char> re;
    try {
        re.assign(pat, pat + sizeof(pat), flags);
    } catch (const std::exception& e) {
        std::printf("threw: %s\n", e.what());
        return 0;
    }
    std::printf("returned, status = %d\n", static_cast<int>(re.status()));
    return 0;
}

$ g++ -std=c++17 -DBOOST_REGEX_STANDALONE -I regex/include -o repro repro.cpp
$ ./repro
repro: regex/include/boost/regex/v5/basic_regex_parser.hpp:2613:
  bool boost::re_detail_600::basic_regex_parser<charT, traits>::parse_perl_extension()
  [with charT = char; traits = boost::regex_traits<char>]:
  Assertion `this->m_traits.syntax_type(*m_position) == regex_constants::syntax_close_mark' failed.
Aborted (core dumped)

The basic_regex constructor taking the same flags aborts identically; it is not specific
to assign().

What is required to trigger it

All of (?:, \R, a following byte that is not ), and both flags are needed. Same flags
throughout, only the pattern varies:

pattern	result
(?:\R\x0c	assertion failure
(?:\R	returns, status() == 13
(?:\R)	returns, status() == 13
\R\x0c	returns, status() == 13
(?:\N\x0c	returns, status() == 5
(?:x\x0c	returns, status() == 8

And with the pattern fixed at (?:\R\x0c, both flags are required — newline_alt alone
throws a regex_error, no_except alone returns cleanly. Neither aborts.

Root cause

Boost.Regex expands \R internally into an atomic group. Its own diagnostic shows the
expansion, if you run the same pattern with newline_alt but without no_except
(non-printables escaped by me):

Invalid preceding regular expression prior to repetition operator.
  The error occurred while parsing the regular expression fragment:
  '?-x:(?>\x0d\x0a?>>>HERE>>>|[\x0a\x0b\x0c\x85]))'.

That is \R → (?>\r\n?|[\n\v\f\x85]). The >>>HERE>>> marker sits immediately after the
\x0a, i.e. newline_alt is being applied to the parser's own synthesized expansion:
the literal \n inside (?>\r\n?|...) is structural, but with newline_alt set it is
reinterpreted as an alternation |, leaving the ? that follows with nothing to repeat.
This looks like the underlying defect — a user-supplied option is leaking into text the
parser generated itself.

From there the two flags interact:

• Without no_except, fail()
  (basic_regex_parser.hpp:244)
  raises regex_error and the parse unwinds.
• With no_except, fail() only records m_status and returns. Control returns to the
  parse_perl_extension() frame handling the outer (?:, which checks for the two failure
  shapes it knows about — unwind_alts() failing (line 2594) and running out of input
  (line 2605) — and then asserts that what remains must be a ):

   if(m_position == m_end)
   {
      // Rewind to start of (? sequence:
      ...
      return false;
   }
   BOOST_REGEX_ASSERT(this->m_traits.syntax_type(*m_position) == regex_constants::syntax_close_mark);

The assert encodes "if we did not run out of input and alternatives unwound, we are at
)", which no longer holds once an error can be recorded without unwinding.

Release builds

With -DNDEBUG the assert disappears and every case above returns normally with an error
status (13), including the aborting one — so I could not produce a wrong-memory or
wrong-result outcome in a release build.

One separate oddity, present in both build configurations: grep | no_except on the same
pattern returns status() == 0 with size() == 6, i.e. the expression is accepted as
valid, while egrep | no_except reports status() == 13. That may be worth a look
independently of the assert.

Suggested direction

Two things look separable:

1. Stop newline_alt from applying to the internally generated \R expansion. That
   removes the bogus parse error and, with it, this abort. It would also fix (?:\R) being
   rejected under newline_alt.
2. Independently, make parse_perl_extension() tolerate the "an error was recorded but no
   exception was thrown" state rather than asserting — e.g. check
   this->m_pdata->m_status alongside the m_position == m_end test at line 2605 before
   reaching line 2613. Otherwise the same shape is likely reachable from other error paths
   whenever no_except is set.

How this was found

Found by an automated fuzz-harness generation experiment built on OSS-Fuzz, then reduced
by hand to the 6-byte pattern and the two-flag combination above and re-verified with the
standalone program in this report — no fuzzing infrastructure, sanitizers, or OSS-Fuzz
build are needed to reproduce it.

I searched this repository's open and closed issues for parse_perl_extension,
syntax_close_mark, and no_except, and reviewed the recent issue list, and did not find
a match. Apologies if I missed one.