fix(verify): don't penalise devices for placeholder SoC dates - #51
Merged
Conversation
`soc_not_after_device` compared the device's release year against the SoC's exactly. 1,954 of the 2,104 SoC records (92.9%) carry a placeholder "YYYY-01-01" date whose year itself runs up to ~2 years late, so for almost every phone that check measured the placeholder rather than the device: it fails 5,488 otherwise-sound smartphones, costing each ~5.8 consistency points and holding a large near-miss pool just under the green threshold. Where the SoC date is a placeholder we now allow 2 years of slack and only flag a gross mismatch. A real, day-precise SoC date keeps the strict comparison, so the genuine signal is preserved. Refs #1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The verification layer's
soc_not_after_deviceconsistency signal compares the device's release year against its SoC's release year exactly.Measured against the current dataset:
YYYY-01-01A placeholder year is imprecise by up to ~2 years (Snapdragon 888, shipped Nov 2020, is stored as
2022-01-01). So for ~93% of comparisons the signal measures the placeholder, not the device.Cost per affected record: consistency is
35 × passed/evaluated, so one failing signal out of ~6 removes ~5.8 points. The near-miss pool (yellow, T1/T2 host, score 65–74) has a consistency mean of 29.3/35 — exactly one failure — and sits just under the green threshold of 75.Change
When the SoC's date is a placeholder, allow 2 years of slack and flag only a gross mismatch. A day-precise SoC date keeps the strict comparison, so genuine ordering errors are still caught.
Two regression tests cover both halves; the existing softness test now uses a gross mismatch so it still exercises what it was written for.
Refs #1