[ariwired]

Adds compaction for C, C++, Java, and C#. The public compact() function checks the input and calls the compactor for the selected language.

Files added

File	Changes
llm_compact/core/common.py	Compares token signatures and checks when a space is needed between adjacent tokens.
llm_compact/core/cfamily.py	Reads C-family source into tokens and rebuilds it with reduced whitespace. Handles comments, literals, operators, and supported directives. Compares the token sequence before and after compaction and returns the original when a check fails or an unsupported construct is detected.
llm_compact/core/__init__.py	Exports compact(), CompactOptions, CoreResult, and SUPPORTED_LANGUAGES. Checks the input type, language, and UTF-8 byte-size limit, then calls the Python or C-family compactor.
tests/test_cfamily.py	Tests spacing, literals, comments, directives, token comparisons, and cases that must return the original source.
tests/test_compilers.py	Compiles C/C++ examples with GCC/G++ and compares their output before and after processing. Also checks that the multiline macro-comment case remains compilable when returned unchanged.
tests/test_core.py	Tests input validation, unsupported languages, size limits, option validation, and language dispatch through compact().
examples/sample.c	C input for lexer and compiler tests.
examples/sample.cpp	C++ input for lexer and compiler tests.
examples/Sample.java	Java input for lexer tests.
examples/Sample.cs	C# input that includes syntax requiring the original source to be kept.

Usage

from llm_compact.core import CompactOptions, compact

result = compact(
    source="int value = 1;",
    language="c",
    options=CompactOptions(level="full"),
)

print(result.code)
print(result.validation)
print(result.fallback)

compact() returns a CoreResult with the output code, validation category, warnings, and fallback status.

Behavior

• Supports full and light modes.
• Keeps comments by default; removal is optional.
• Checks that compaction preserves the token sequence recognized by the internal lexer.
• Returns the original source when a transformation is rejected.
• Rejects C/C++ directives containing /* and C# Unicode escapes outside literals, comments, and preserved directives.
• Uses the existing Python compactor for Python input.

The C/C++ directive check also rejects inline block comments and /* inside directive strings. This is a known restriction of the current lexer.

Tests

Local results:

• python -m ruff check .: passed.
• python -m pytest -q: 192 passed, with no warnings or skipped tests reported.

The count includes the Python tests from PR #1 .

Limitations

C-family validation uses the internal lexer, not a language parser. It does not prove that the code is syntactically valid or behaves identically. Invalid or incomplete input may pass this check.

GCC/G++ tests cover the supplied examples and regression cases. Java and C# compilation are not tested.

Token counting, the analysis service, CLI, and Streamlit interface are planned for later PRs.

[luizcodedthat]

Sorry for the wait. Ran the PR in a container (gcc/g++ 12, javac 17) and added tests: Hypothesis property tests plus gcc/g++ assembly and javac bytecode comparisons before and after compaction. Ruff passes, the suite gives 1170 passed and extended fuzzing gives 245 passed. One bug blocks the merge and one performance issue should be fixed; the rest can go in a follow-up PR.

Blocking

A lone \r ends a // comment, but the lexer only looks for \n (strip_comments=True). The comment swallows the next statement, with no fallback:

compact("int a; // note\rint b;\n", "c", CompactOptions("full", True)).code
# 'int a;'  (fallback=False, validation="lexical")

This changes the program: gcc, g++ and javac compile int b;, but the compacted output no longer defines b. The cause is cfamily.py:99, where s.find("\n", p) makes the token // note\rint b;, so the token comparison sees nothing wrong. In C#, U+0085, U+2028 and U+2029 also end a // comment.

_EOL = re.compile(r"[\r\n]")
_CSHARP_EOL = re.compile("[\r\n\x85\u2028\u2029]")

if s.startswith("//", p):
    m = (_CSHARP_EOL if lang == "csharp" else _EOL).search(s, p)
    p = len(s) if m is None else m.start()
    kd = "comment"

Important

The C# lexer is O(n²). cfamily.py:129 runs re.match(r'\${2,}"', s[p:]), copying the rest of the file for every token. At ~990 KB (limit: 1,000,000 bytes), C# took ~18 s versus ~4 s for Java and C. Pre-compile the pattern and pass the position (~18 s drops to ~5 s):

_INTERP = re.compile(r'\${2,}"')
...
or _INTERP.match(s, p)

Minor (follow-up PR)

• Trigraph created by compaction. a ?? /* c */ b becomes a??/* c */b, and ??/ is \ in ISO C (an edge case, since ?? is not a C/C++ token). Check the output too, right after new = "".join(buf):

  if lang in ("c", "cpp") and re.search(r"\?\?[/=()'!<>-]", new):
      raise UnsafeSourceError("Compaction would create a trigraph.")

• Wrong options type. A dict raises AttributeError; raise TypeError instead:

  if options is not None and not isinstance(options, CompactOptions):
      raise TypeError("options must be a CompactOptions instance.")

Validation

With all snippets applied to a throwaway copy, the suite with --runxfail goes from 15 failed / 1169 passed (original PR) to 1184 passed. The new tests mark each bug xfail(strict=True): when a fix lands the test XPASSes, which is the cue to remove the marker. I can open the tests as a follow-up PR, or push them here if you prefer.

The rest of the PR looks good, as always.

[ariwired]

Thanks, my camarada, for the thorough review and for running it against real compilers. Everything marked blocking or important is fixed in this branch, along with the minor items.

Blocking

lone \r ends a // comment

The lexer now looks for any line terminator instead of only \n:

_EOL = re.compile(r"[\r\n]")
_CSHARP_EOL = re.compile("[\r\n\x85\u2028\u2029]")

m = eol.search(s, p)
p = len(s) if m is None else m.start()

The same terminator set is used for preprocessor directives and for the start-of-line check, so #define X 1\rint b; also works. Before and after, with strip_comments=True:

compact("int a; // note\rint b;\n", "c", CompactOptions("full", True)).code
# before: 'int a;'        (fallback=False)
# after:  'int a;int b;'

C# with \u2028 as the terminator behaves the same way.

Important

C# lexer was O(n²)

re.match(r'\${2,}"', s[p:]) is now a precompiled _INTERP.match(s, p). I also replaced the repeated string slicing in the directive check with a single join. On a ~962 KB C# file, compaction went from about 9.9 s to 2.1 s on my machine. Java stayed at about 1.5 to 1.7 s.

Validation

• The suite goes from 192 to 238 tests, all passing, and ruff check is clean.
• The new line terminator and trigraph tests fail on the previous cfamily.py (13 failures) and pass now.
• I did not add xfail(strict=True) markers, since the fixes are already in. I added plain regression tests, covering lone \r, \r\n, the C# Unicode terminators, directives ending at a lone \r, the trigraph fallback and the options type.
• There is no timing test for the C# fix, since it would be flaky. The numbers above are manual measurements.