Summary
lm-eval[api]==0.4.11 pulls in three transitive dependencies that have outstanding CVEs without an available upstream fix on a stable line we can adopt. Tracking them here so they don't keep showing up as recurring CI noise without context.
Current vulnerable transitive dependencies
Why we are not patching now
- We can't safely silence these in CI per the repo's "no-bypasses" policy. They're real findings; we acknowledge them.
- Bumping
transformers to a 5.0 RC would break lm-eval[api]==0.4.11's pin range and is unstable for a production release. We pin lm-eval explicitly for benchmark reproducibility.
sqlitedict and diskcache have no upstream fix to adopt at all.
Risk assessment
These dependencies are only loaded when running lm_eval locally for benchmarks (a deliberate, isolated workload running our own configs against our own models). They are not exposed to the network or to user input. Risk in our usage: low.
Action items
CI behavior
CI Gate (_python-security.yml and _osv-scan.yml) currently surfaces these on every run. Until upstream fixes ship, this is expected and acceptable — the alerts are documented, not ignored.
Summary
lm-eval[api]==0.4.11pulls in three transitive dependencies that have outstanding CVEs without an available upstream fix on a stable line we can adopt. Tracking them here so they don't keep showing up as recurring CI noise without context.Current vulnerable transitive dependencies
sqlitedictdiskcachetransformersWhy we are not patching now
transformersto a 5.0 RC would breaklm-eval[api]==0.4.11's pin range and is unstable for a production release. We pin lm-eval explicitly for benchmark reproducibility.sqlitedictanddiskcachehave no upstream fix to adopt at all.Risk assessment
These dependencies are only loaded when running
lm_evallocally for benchmarks (a deliberate, isolated workload running our own configs against our own models). They are not exposed to the network or to user input. Risk in our usage: low.Action items
lm-evalfor a 0.4.x bump that dropssqlitedict/diskcache(issue: EleutherAI/lm-evaluation-harness)transformers5.0.0 ships GA, evaluate replacing the current lm-eval pin with one compatible with transformers 5pip-auditandosv-scannerquarterly until clearedCI behavior
CI Gate (
_python-security.ymland_osv-scan.yml) currently surfaces these on every run. Until upstream fixes ship, this is expected and acceptable — the alerts are documented, not ignored.