[ariwired]

feat: add project setup and Python compaction

This PR establishes the initial Python implementation of LLM Compact. It adds the package configuration, shared data structures, a Python source compactor, automated tests, and a GitHub Actions workflow.

The compactor removes unnecessary formatting while checking that accepted transformations preserve the original Python AST and compile successfully. This provides the foundation for the token-counting service, additional language support, CLI, and interface planned for subsequent PRs.

Files added

File	Purpose and implementation
pyproject.toml	Defines the package metadata, Python 3.12+ requirement, setuptools build backend, and dependency groups. Declares tiktoken as a required dependency and provides optional groups for Streamlit and development tools. Configures pytest discovery and Ruff checks. The CLI entry point is deferred until the CLI implementation is introduced.
requirements.txt	Installs the local package with the UI dependencies through .[ui]. This prepares the dependency configuration for the later interface implementation.
requirements-dev.txt	Installs the package in editable mode with UI and development dependencies through -e .[ui,dev], allowing local source changes to be tested without reinstalling the package.
requirements-lock.txt	Records the exact dependency versions from the reference environment. The default development installation uses the version constraints in pyproject.toml; it does not automatically enforce this snapshot.
llm_compact/core/types.py	Defines supported languages and compaction levels, the source-size limit constant, immutable configuration and result dataclasses, the token representation, and the exception used for unsupported transformations. CompactOptions validates the selected level and the comment-removal option. The shared size limit will be enforced by the public dispatcher introduced in a later PR.
llm_compact/core/python.py	Implements Python compaction using the standard tokenize and ast modules. Reconstructs source text with reduced whitespace, preserves indentation and literal contents, optionally removes comments, and validates the candidate against the original AST. Returns the original source with a warning when parsing or transformation validation fails.
tests/test_python.py	Tests formatting reduction, AST preservation, indentation, multiline expressions, explicit line continuation, comments, docstrings, f-strings, CRLF input, invalid or incomplete code, and repeated compaction. Includes a runtime comparison for a controlled debug f-string example.
examples/sample.py	Provides a Python fixture containing type annotations, collections, conditionals, f-strings, raw strings, line continuation, classes, properties, a lambda, and an assignment expression. The tests use this file to exercise several syntax constructs together.
.github/workflows/ci.yml	Runs on pushes, pull requests, and manual dispatch. Configures Python 3.12, installs development dependencies, loads both tokenizer tables directly through tiktoken, checks for GCC/G++, and runs Ruff and pytest. Loading the tables directly avoids depending on the future CLI.
.gitignore	Excludes virtual environments, Python caches, test and lint caches, build outputs, package metadata, local secrets, and editor-specific files from version control.

Compaction behavior

• full removes unnecessary spacing and eligible line breaks, including within bracketed expressions.
• light retains line breaks and checks that the newline count remains unchanged.
• Comments are preserved by default. When comment removal is enabled, the implementation retains shebangs and recognized encoding-header comments.
• String contents, f-strings, and docstrings are preserved.
• Accepted candidates must have the same AST as the original, excluding source-position attributes, and must pass compilation checks.
• Invalid, incomplete, or incompatible Python is returned unchanged with a warning.

The compaction implementation parses and compiles source for validation; it does not execute the submitted program.

Validation

The local checks reported:

• python -m ruff check .: passed.
• python -m pytest -q: 36 tests passed.

The GitHub Actions result should be checked before merging.

Scope and limitations

This PR introduces the Python compaction foundation. Token counting, measured savings, the public language dispatcher, CLI commands, and the Streamlit interface are delivered in subsequent PRs.

AST comparison does not guarantee identical behavior for source introspection, traceback positions, or tools that depend on formatting. Syntax support also depends on the Python interpreter running the compactor.

This PR does not measure the effect of compaction on LLM response quality.

[luizcodedthat]

Approving. Some minor adjustments, none blocking:

• src.splitlines() splits on more characters than the tokenizer sees (e.g. \x0c), so valid source can get a false "Transformation rejected" warning; split rows the same way the tokenizer reads lines (e.g. io.StringIO(src).readlines()).
• Coding header detection uses "coding" in tk.string, so any comment in the first two lines containing the word (e.g. # coding design patterns is my passion) is kept; use the PEP 263 regex.
• strip_comments=True also removes # type: ignore, # noqa and # fmt:; if the output goes back into the codebase, linters and formatters will be affected. Best to keep them, no?

I also ran my own fuzzing over ~1,000 CPython stdlib files (plus form feed injection): no invalid output, AST changes or unhandled exceptions, and no wrong output was accepted.

[ariwired]

Thanks, my camarada! All three points addressed:

• Line splitting: replaced src.splitlines() with io.StringIO(src).readlines(), so rows match what the tokenizer reads. Valid source containing \x0c, \x1c, \x85, \u2028 or \u2029 no longer gets a false "Transformation rejected" warning.
• Coding header: now uses the PEP 263 regex instead of "coding" in tk.string. The line 2 cookie is only kept when line 1 is blank or a comment, like Python does. # coding design patterns is my passion is now removed.
• Tool comments: strip_comments=True now keeps # type:, # noqa, # fmt: and other tool pragmas (isort, pylint, mypy, pyright, ruff, flake8, nosec, pragma:). A false positive only keeps one extra comment, so I went with a broad match.

Added tests for each point (95 passing now, 36 before). Ruff passes. The new tests fail on the previous code.

Also ran a quick fuzz over 400 stdlib files with random form feed injection, in both levels with and without strip_comments: no fallbacks and no AST changes.