Problem
The supported set is .pdf .pptx .docx .txt .md .py .json. Python and JSON are in, but not JavaScript, TypeScript, Go, Rust, Java, C, C++, HTML, CSS, YAML, TOML or XML. There is no technical reason for the split: _read_text already handles all of them, so this is adding entries to a set.
Counting tokens in a codebase before feeding it to a model is a mainstream use case this currently refuses.
Where
parsing.py:11 -> SUPPORTED_EXTENSIONS = {...}
What to add
.js .ts .tsx .jsx .go .rs .java .c .cpp .h .hpp .html .css .yaml .yml .toml .xml .sql .sh .bash
Scope
- Extend
SUPPORTED_EXTENSIONS
- Add tests to
test_parsing.py
- Update the file-picker filter in
api.py:pick_files
- Update the website format list
Perfect first PR.
Problem
The supported set is
.pdf .pptx .docx .txt .md .py .json. Python and JSON are in, but not JavaScript, TypeScript, Go, Rust, Java, C, C++, HTML, CSS, YAML, TOML or XML. There is no technical reason for the split:_read_textalready handles all of them, so this is adding entries to a set.Counting tokens in a codebase before feeding it to a model is a mainstream use case this currently refuses.
Where
parsing.py:11->SUPPORTED_EXTENSIONS = {...}What to add
.js .ts .tsx .jsx .go .rs .java .c .cpp .h .hpp .html .css .yaml .yml .toml .xml .sql .sh .bashScope
SUPPORTED_EXTENSIONStest_parsing.pyapi.py:pick_filesPerfect first PR.