Poor Reconstruction and Black Render Output When Using vanilla_gs Configuration in LE3D

[luyuxuan98]

First of all, I would like to express my appreciation for your amazing work on this project. It has been really helpful in my work! However,I'm encountering an issue while using the LE3D project, following the LE3D configuration file located in LE3D/options/vanilla_gs/base.yaml to reproduce the original 3DGS model. The results are poor, with distant objects poorly reconstructed, significant artifacts, and a low PSNR of 22.4262, compared to the original 3DGS model's PSNR of 23.845. Additionally, setting convert_SHs_python=false in the base.yaml file leads to a black rendered output.

Steps to Reproduce

1. Use the configuration file below:

base: LE3D/options/vanilla_gs/base.yaml
name: vanilla_gs/nerf_llff_data/fern
datasets:
  train:
    name: nerf_llff_data_fern_train
    scene_root: path/to/my/datasets/nerf_llff_data/nerf_llff_data/fern
  val:
    name: nerf_llff_data_fern_val
    scene_root: path/to/my/datasets/nerf_llff_data/nerf_llff_data/fern
network_g:
  init_ply_path: path/to/my/datasets/nerf_llff_data/nerf_llff_data/fern/sparse/0/points3D.ply

2. Run the experiment and observe poor results (low PSNR and artifacts).

3. Modify the base.yaml file by setting convert_SHs_python=false and notice the black rendering output.

Expected Results:

• Proper reconstruction with minimal artifacts and a PSNR closer to the original model's value (~23.845).
• Correct rendered images without the black output when using convert_SHs_python=false.

Observed Results:

• PSNR of 22.4262.

• Distant parts of the scene are poorly reconstructed, with visible artifacts.
  
  
  

• Setting convert_SHs_python=false leads to a black render.
  

What I've Tried:

• Replaced gcnt-rasterization with base-rasterization from the repository, but the issue persists.
• Attempted to use the diff-gaussian-rasterization module from https://github.com/graphdeco-inria/gaussian-splatting with no improvement.

I apologize for the disturbance and sincerely hope you can help me resolve this issue. Thank you for your time and assistance!

[prinasi]

Have you solved the problems now? Actually I trained LE3D with self-captured RAW datasets, organized similar to the RAW-NeRF dataset, where the results were poor as well. I will appreciate if the authors can give some advice, if any!

[luyuxuan98]

Have you solved the problems now? Actually I trained LE3D with self-captured RAW datasets, organized similar to the RAW-NeRF dataset, where the results were poor as well. I will appreciate if the authors can give some advice, if any!

Hi,

Yes, I have managed to resolve the issue. The root cause was a bug in the #SSIMLoss function interface. I have submitted a Pull Request (PR) to fix this: #12. If you are using the built-in dssim during training, this fix should be helpful.

Regarding the RAW datasets: I haven't tested with self-captured RAW data yet; I'm currently using the official RAW-NeRF dataset. Although my results didn't quite reach the peak metrics reported in the paper (and didn't outperform RAW-NeRF), the reconstruction quality is now acceptable for a 3DGS-based approach.

Hope this helps!

[prinasi]

Thanks for your help. I have additional questions. First, when you trained LE3D on the RAW-NeRF dataset, did you down-sample the RAW images as the authors did? In other words, were the metrics of LE3D and RAW-NeRF compared with  the same resolution of RAW images? Second, I concern that PSNR (or so-called RAW-PSNR) is somewhat tricky, especially when the pixel values are far smaller than 1. Hope to hear from your opinion and response, Thanks! 张凯旋 ***@***.***

[luyuxuan98]

Thanks for your help. I have additional questions. First, when you trained LE3D on the RAW-NeRF dataset, did you down-sample the RAW images as the authors did? In other words, were the metrics of LE3D and RAW-NeRF compared with  the same resolution of RAW images? Second, I concern that PSNR (or so-called RAW-PSNR) is somewhat tricky, especially when the pixel values are far smaller than 1. Hope to hear from your opinion and response, Thanks!

张凯旋
@.***

I conducted all my training at 2K resolution (as specified in Table 1 of the original paper), and RAW-NeRF was also evaluated at the same 2K resolution. The results show that LE3D's PSNR and SSIM are both slightly lower than those of RAW-NeRF.

[prinasi]

I see. Thanks for your patience and response! 张凯旋 ***@***.***