Can Boxer be used for small-object tracking in a fixed desktop RGB-D scene?

[yaozhu0102]

Hi, I tried running Boxer on my own RGB-D dataset captured from a fixed desktop camera.

The dataset is organized in the static_rgbd format:

  root/
  cam_K.txt
  rgb/
  depth/

I used the prompt apple. The 2D detection looks correct and detects the apple, but the predicted 3D box does not stay aligned with the apple. It appears to remain near the initial position or drift away from the object.

My command was:
bash

python run_boxer.py \
  --input /path/to/apple \
  --labels=apple \
  --track \
  --track_mode dynamic \
  --thresh2d 0.15 \
  --thresh3d 0.3

I would like to ask:
Is Boxer expected to work well on fixed-camera desktop RGB-D scenes with small objects such as apples?
Is the model mainly intended for room-scale indoor scenes / Aria-style data rather than tabletop object tracking?
Are there any recommended settings or data requirements for this type of use case?

[ddetone]

can you share a visual with the per-frame 2DBB and 3DBB detections?

the tracker is not designed to support dynamics, but the per-frame 3DBBs should work in an RGB-D setting. it may be an issue with your custom dataloader

[yaozhu0102]

Here are two output images from my Boxer run. The 2D bounding boxes are correct, but the 3D bounding boxes are completely wrong.

Currently, the method assumes that the gravity direction is exactly the vertical direction in the image, and that the real camera pose remains fixed. The camera is a RealSense D435. I have aligned RGB-D data and the camera intrinsics.

The results appear as shown in the figures. Does this mean that Boxer is only suitable for static, large-scale indoor scenes, where it detects relatively large objects, and is not suitable for tabletop scenarios?

[ddetone]

Boxer should work in your case.

Ah, this looks like an issue with rotation. Can you confirm that you are setting the input boolean to BoxerNet as rotate0=False? (e.g. aria images they are rotated by 90 degrees coming out of the device so we set rotate0=True)

See how its used here: https://github.com/facebookresearch/boxer/blob/main/boxernet/boxernet.py#L664

[yaozhu0102]

Thanks for the suggestion. I checked the input passed into BoxerNet for this custom static RGB-D dataset, and rotated0 is already set to False.

Specifically, this dataset uses StaticRgbdLoader, not the Aria loader, and the datum passed to BoxerNet contains:

"rotated0": torch.tensor(False).reshape(1)

So the current results are already produced with rotate0/rotated0=False. The output still looks like this, so the issue does not seem to be caused by the BoxerNet input rotation flag for this dataset.

[ddetone]

Interesting. Could you by chance save your input data into a file and attach it so I can debug it myself? Something like a .pkl or .pt or .tar file I can load and reproduce, along with a commit hash? There may be a bug in my code

[yaozhu0102]

Thank you very much for your patience and help. This is my dataset. It was originally prepared for use with FoundationPose, and I would like to directly use it to test Boxer as well. Since the compressed file is too large to be uploaded or attached through the GitHub issue, I am sending it to you directly by email instead. Thank you again for your time and support. I really appreciate your help. Original From:Daniel DeTone ***@***.***> Sent Time:Apr 27, 2026 00:18 To:facebookresearch/boxer ***@***.***> Cc:yaozhu0102 ***@***.***>, Author ***@***.***> Subject:Re: [facebookresearch/boxer] Can Boxer be used for small-object tracking in a fixed desktop RGB-D scene? (Issue #9) ddetone left a comment (facebookresearch/boxer#9) Interesting. Could you by chance save your input data into a file and attach it so I can debug it myself? Something like a .pkl or .pt or .tar file I can load and reproduce, along with a commit hash? There may be a bug in my code — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today! You are receiving this because you authored the thread. 从QQ邮箱发来的超大附件 apple.zip (235.5MB, 2026年5月28日 14:6) 进入下载页面 ：https://wx.mail.qq.com/ftn/download?func=3&k=9e9e5266ccf4d888aced196636336136f0568f6834336136494e5a001b03075302590507001e510152591a520c52581b58520251190a0005565407530d5659535268370744430d534f1b5e1620637bdfdb1f0f98797de8cbdbabe3cfdf9ed077c06037663433613661&key=9e9e5266ccf4d888aced196636336136f0568f6834336136494e5a001b03075302590507001e510152591a520c52581b58520251190a0005565407530d5659535268370744430d534f1b5e1620637bdfdb1f0f98797de8cbdbabe3cfdf9ed077c06037663433613661&code=aa7f43a6&from=

[ddetone]

can you send it to ddetone@meta.com?

[ddetone]

or you could commit the data and push it to a github branch on whatever fork you are using? you can always delete that branch later since it will be big

[yaozhu0102]

I have sent it to ddetone@meta.com

[ddetone]

got the data. did you modify the code at all to load this?

[ddetone]

Update -- I am able to reproduce the issue. My hypothesis is that this is actually a pretty fundamental issue in the training dataloader learning the wrong biases. I'm re-training the model now and will report back as soon as possible

[ddetone]

@yaozhu0102 I trained a new model and I think it fixed the issue. Its an early snapshot (still needs 2-3 days to finish converging), but looks promising:

[yaozhu0102]

Thank you for the update! Glad to hear that the new model looks promising.

I will wait for the fully converged model results. Please let me know if you need any additional data or examples from my side.

Thanks again for your help!

[ddetone]

Hi, can you try this checkpoint referenced in the code? If you re-run the download checkpoint script it should work: fd6bf73. You rebase on top of the latest main and it should work.

Note this has only trained about 7 days and takes 21 days to fully train, so I may update it later to share the best possible model.

Let me know how it works.

[yaozhu0102]

Hi, thanks for the update!

I rebased on the latest main and re-ran the checkpoint download script as suggested. I tested the new checkpoint on my own data, and it indeed works very well. The results look much better than before, especially for the 3D detections.

Thanks again for sharing this checkpoint. I’ll continue testing it and let you know if I find any issues.

boxer_test_result_small.mp4