Skip to content

fix(t4metric_v2): filter GT by annotation num_lidar_pts - #224

Open
vividf wants to merge 1 commit into
tier4:mainfrom
vividf:fix/t4metric-v2-gt-min-points-filter
Open

fix(t4metric_v2): filter GT by annotation num_lidar_pts#224
vividf wants to merge 1 commit into
tier4:mainfrom
vividf:fix/t4metric-v2-gt-min-points-filter

Conversation

@vividf

@vividf vividf commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

IMPORTANT

Don't merge this PR since we are moving to autoware-ml.

What

T4MetricV2's min_num_points GT filter recomputed per-box point counts with
points_in_rbbox on the model input point cloud (multi-sweep concatenation,
after remove_close and range filtering). This PR changes it to use the
annotation num_lidar_pts stored in the info pkl (keyframe count, the
nuScenes/Waymo convention).

Why

Recomputing on the input cloud couples the evaluation GT set to the model input
pipeline:

  • Changing the sweep count or any point-cloud preprocessing changes which GT
    boxes are evaluated
    , so runs with different input configs are not comparable
    (an experiment that improves detection can show a lower mAP simply because its
    input config resurrects harder GT boxes).
  • Sweep accumulation is ego-motion compensated but not object-motion
    compensated
    , so the recomputed counts are physically wrong for moving
    objects (smeared points fall outside the box).
  • Preprocessing artifacts (remove_close, range filter) remove points from
    boxes that the sensor actually observed, wrongly excluding them from GT.

The benchmark GT should be a fixed property of the dataset, not of the model
config. Note that the training-side filters (ObjectMinPointsFilter)
intentionally keep recomputing on the input cloud — that is a training
hyperparameter and is not changed here.

Verification

On the BEVFusion-L j6gen2_base val split (3,645 frames, kokseang_2_8_1 infos):

  • The num_lidar_pts field is fully populated (209,759 instances, 0 missing;
    17.5 % zero-point boxes from interpolated annotations are correctly dropped
    by min_num_points=2).
  • The filtered GT set now matches an independent implementation
    (autoware-ml detection metric suite) class by class
    (car 92,351 / bus 3,755 / bicycle 2,456 / pedestrian 35,186 /
    traffic_cone 3,583 / barrier 524).
  • 0–121 m mAP moves 0.6518 → 0.6499 and now agrees with the independent
    implementation to ±0.0004 (was −0.006 before this fix).

Side benefit: removes one points_in_rbbox call per frame from evaluation.

…computing from input points

The min_num_points GT filter recomputed per-box point counts with
points_in_rbbox on the model input point cloud (multi-sweep, after
remove_close and range filtering). This couples the evaluation GT set
to the input pipeline configuration: changing the sweep count or any
point-cloud preprocessing changes which GT boxes are evaluated, so runs
with different input configs are not comparable. Sweep accumulation is
also not object-motion compensated, so the recomputed counts are
physically wrong for moving objects.

Use the annotation num_lidar_pts stored in the info pkl instead
(keyframe count, the nuScenes/Waymo convention), which keeps the GT set
a fixed property of the dataset. Verified on the j6gen2_base val split
(3645 frames): the field is fully populated, and the filtered GT set
now matches an independent implementation class by class.

Signed-off-by: vividf <yihsiang.fang@tier4.jp>
@vividf
vividf requested a review from KSeangTan as a code owner August 5, 2026 04:17
@vividf vividf self-assigned this Aug 5, 2026
@vividf vividf changed the title fix(t4metric_v2): filter GT by annotation num_lidar_pts instead of re… fix(t4metric_v2): filter GT by annotation num_lidar_pts Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant