Problem
errors="ignore" drops every byte that fails to decode, with no signal. A Windows-1252 or UTF-16 document loses characters, so the token count comes out low and the cost estimate under-reports. Silent undercounting is a critical failure mode.
Where
parsing.py:56 : path.read_text(encoding="utf-8", errors="ignore")
Fix
Detect the encoding (charset-normalizer ships with requests already), or count the dropped bytes and warn when they exceed a threshold.
Severity
This affects any user analyzing a legacy document or file from Windows systems.
Problem
errors="ignore"drops every byte that fails to decode, with no signal. A Windows-1252 or UTF-16 document loses characters, so the token count comes out low and the cost estimate under-reports. Silent undercounting is a critical failure mode.Where
parsing.py:56:path.read_text(encoding="utf-8", errors="ignore")Fix
Detect the encoding (
charset-normalizerships withrequestsalready), or count the dropped bytes and warn when they exceed a threshold.Severity
This affects any user analyzing a legacy document or file from Windows systems.