Vulnerable Library - tokenizers-0.20.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
No project description provided
Library home page: https://files.pythonhosted.org/packages/50/f6/2841de926bc4118af996eaf0bdf0ea5b012245044766ffc0347e6c968e63/tokenizers-0.20.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Sample Path to Dependency File: /modules/programming/module_programming_themisml/.ws-temp-JAAXML-requirements.txt
Path to vulnerable library: /home/wss-scanner/.cache/pypoetry/virtualenvs/module-programming-themisml-6JtHFAOJ-py3.11/lib/python3.11/site-packages/tokenizers-0.20.3.dist-info
Vulnerabilities
| Vulnerability |
Severity |
 CVSS |
Dependency |
Type |
Fixed in (tokenizers version) |
Remediation Possible** |
| CVE-2026-85670 |
 Medium |
6.5 |
tokenizers-0.20.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
Direct |
N/A |
❌ |
**In some cases, Remediation PR cannot be created automatically for a vulnerability despite the availability of remediation
Details
CVE-2026-85670
Vulnerable Library - tokenizers-0.20.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
No project description provided
Library home page: https://files.pythonhosted.org/packages/50/f6/2841de926bc4118af996eaf0bdf0ea5b012245044766ffc0347e6c968e63/tokenizers-0.20.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Sample Path to Dependency File: /modules/programming/module_programming_themisml/.ws-temp-JAAXML-requirements.txt
Path to vulnerable library: /home/wss-scanner/.cache/pypoetry/virtualenvs/module-programming-themisml-6JtHFAOJ-py3.11/lib/python3.11/site-packages/tokenizers-0.20.3.dist-info
Dependency Hierarchy:
- ❌ tokenizers-0.20.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (Vulnerable Library)
Found in base branch: develop
Vulnerability Details
tokenizers (Hugging Face) is affected by an out-of-bounds buffer access in BpeBuilder::build (tokenizers/src/models/bpe/model.rs). When loading a tokenizer.json via Tokenizer::from_file/from_str, the builder sizes a scratch buffer to the longest vocabulary key, then writes each concatenated merge rule into it. A merge whose concatenated token exceeds the longest vocabulary key overruns the buffer, which Rust turns into a panic that aborts the process in Rust and FFI embeddings. This occurs at load time with no encoding required, so an attacker who supplies a crafted tokenizer.json can cause a denial of service. A secondary defect at the same location can cause a usize underflow (panic in debug, potential memory corruption in release) when continuing_subword_prefix is set and a merge token is shorter than the prefix. Observed in version 0.23.1.
Publish Date: 2026-09-04
URL: CVE-2026-85670
CVSS 3 Score Details (6.5)
Base Score Metrics:
- Exploitability Metrics:
- Attack Vector: Network
- Attack Complexity: Low
- Privileges Required: None
- User Interaction: Required
- Scope: Unchanged
- Impact Metrics:
- Confidentiality Impact: None
- Integrity Impact: None
- Availability Impact: High
For more information on CVSS3 Scores, click here.
Step up your Open Source Security Game with Mend here
No project description provided
Library home page: https://files.pythonhosted.org/packages/50/f6/2841de926bc4118af996eaf0bdf0ea5b012245044766ffc0347e6c968e63/tokenizers-0.20.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Sample Path to Dependency File: /modules/programming/module_programming_themisml/.ws-temp-JAAXML-requirements.txt
Path to vulnerable library: /home/wss-scanner/.cache/pypoetry/virtualenvs/module-programming-themisml-6JtHFAOJ-py3.11/lib/python3.11/site-packages/tokenizers-0.20.3.dist-info
Vulnerabilities
**In some cases, Remediation PR cannot be created automatically for a vulnerability despite the availability of remediation
Details
Vulnerable Library - tokenizers-0.20.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
No project description provided
Library home page: https://files.pythonhosted.org/packages/50/f6/2841de926bc4118af996eaf0bdf0ea5b012245044766ffc0347e6c968e63/tokenizers-0.20.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Sample Path to Dependency File: /modules/programming/module_programming_themisml/.ws-temp-JAAXML-requirements.txt
Path to vulnerable library: /home/wss-scanner/.cache/pypoetry/virtualenvs/module-programming-themisml-6JtHFAOJ-py3.11/lib/python3.11/site-packages/tokenizers-0.20.3.dist-info
Dependency Hierarchy:
Found in base branch: develop
Vulnerability Details
tokenizers (Hugging Face) is affected by an out-of-bounds buffer access in BpeBuilder::build (tokenizers/src/models/bpe/model.rs). When loading a tokenizer.json via Tokenizer::from_file/from_str, the builder sizes a scratch buffer to the longest vocabulary key, then writes each concatenated merge rule into it. A merge whose concatenated token exceeds the longest vocabulary key overruns the buffer, which Rust turns into a panic that aborts the process in Rust and FFI embeddings. This occurs at load time with no encoding required, so an attacker who supplies a crafted tokenizer.json can cause a denial of service. A secondary defect at the same location can cause a usize underflow (panic in debug, potential memory corruption in release) when continuing_subword_prefix is set and a merge token is shorter than the prefix. Observed in version 0.23.1.
Publish Date: 2026-09-04
URL: CVE-2026-85670
CVSS 3 Score Details (6.5)
Base Score Metrics:
- Exploitability Metrics:
- Attack Vector: Network
- Attack Complexity: Low
- Privileges Required: None
- User Interaction: Required
- Scope: Unchanged
- Impact Metrics:
- Confidentiality Impact: None
- Integrity Impact: None
- Availability Impact: High
For more information on CVSS3 Scores, click here.Step up your Open Source Security Game with Mend here