diff --git a/.gitignore b/.gitignore index 215effa..9f4392f 100644 --- a/.gitignore +++ b/.gitignore @@ -6,3 +6,4 @@ build/ *.tar *.pyd +*.egg-info \ No newline at end of file diff --git a/DCVC-RT/README.md b/DCVC-RT/README.md deleted file mode 100644 index 8a0018e..0000000 --- a/DCVC-RT/README.md +++ /dev/null @@ -1,113 +0,0 @@ -# Introduction - -Official Pytorch implementation for DCVC-RT: [Towards Practical **R**eal-**T**ime Neural Video Compression](https://arxiv.org/abs/2502.20762), in CVPR 2025. - -# Prerequisites -* Python 3.12 and conda, get [Conda](https://www.anaconda.com/) -* CUDA 12.6 (other versions may also work. Make sure the CUDA version matches with pytorch.) -* pytorch (We have tested that pytorch-2.6 works. Other versions may also work.) -* Environment - ``` - conda create -n $YOUR_PY_ENV_NAME python=3.12 - conda activate $YOUR_PY_ENV_NAME - - pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126 - pip install -r requirements.txt - ``` - -# Test dataset - -We support arbitrary original resolution. The input video resolution will be padded automatically. The reconstructed video will be cropped back to the original size. The distortion (PSNR) is calculated at original resolution. - -## YUV 420 content - -Put *.yuv in the folder structure similar to the following structure. - - /media/data/HEVC_B/ - - BQTerrace_1920x1080_60.yuv - - BasketballDrive_1920x1080_50.yuv - - ... - /media/data/HEVC_D/ - /media/data/HEVC_C/ - ... - -The dataset structure can be seen in dataset_config_example_yuv420.json. - -## RGB content - -We highly suggest testing YUV420 content. To test RGB content, please refer to the [DCVC-FM](../DCVC-FM) folder. - -# Build the project -Please build the C++ code to support bitstream writing and customized CUDA kernels to fuse operations. - -```bash -sudo apt-get install cmake g++ ninja-build -conda activate $YOUR_PY_ENV_NAME -cd ./src/cpp/ -pip install . -cd ../layers/extensions/inference/ -pip install . -``` - -# CPU performance scaling - -Note that the arithmetic coding runs on the CPU, please make sure your CPU runs at high performance while writing the actual bitstream. Otherwise, the arithmetic coding may take a long time. - -Check the CPU frequency by -``` -grep -E '^model name|^cpu MHz' /proc/cpuinfo -``` - -Run the following command to maximum CPU frequency -``` -echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor -``` - -Run the following command to recover the default frequency -``` -echo ondemand | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor -``` - -# Pretrained models - -* Download [our pretrained models](https://1drv.ms/f/c/2866592d5c55df8c/Esu0KJ-I2kxCjEP565ARx_YB88i0UnR6XnODqFcvZs4LcA?e=by8CO8) and put them into ./checkpoints folder. -* There are 2 models, one for image coding and the other for video coding. - -# Test the models - -Example to test pretrained model with four rate points: -```bash - python test_video.py --model_path_i ./checkpoints/cvpr2025_image.pth.tar --model_path_p ./checkpoints/cvpr2025_video.pth.tar --rate_num 4 --test_config ./dataset_config_example_yuv420.json --cuda 1 -w 1 --write_stream 1 --force_zero_thres 0.12 --output_path output.json --force_intra_period -1 --reset_interval 64 --force_frame_num -1 --check_existing 0 -``` - -It is recommended that the ```-w``` number is equal to your GPU number. - -You can also specify different ```--rate_num``` values (2~64) to test finer bitrate adjustment. - -# Comparing with other method -Bit saving over VTM-17.0 (UVG all frames with single intra-frame setting (i.e. intra-period = –1) and YUV420 colorspace.) - - - -The BD-Rate and encoding/decoding speed on Nvidia A100 GPU - - - -# Acknowledgement -The implementation is based on [CompressAI](https://github.com/InterDigitalInc/CompressAI). - -# Citation -If you find this work useful for your research, please cite: - -``` -@inproceedings{jia2025towards, - title={Towards Practical Real-Time Neural Video Compression}, - author={Jia, Zhaoyang and Li, Bin and Li, Jiahao and Xie, Wenxuan and Qi, Linfeng and Li, Houqiang and Lu, Yan}, - booktitle={{IEEE/CVF} Conference on Computer Vision and Pattern Recognition, - {CVPR} 2025, Nashville, TN, USA, June 11-25, 2024}, - year={2025} -} -``` - -# Trademarks -This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow [Microsoft’s Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general). Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party’s policies. \ No newline at end of file diff --git a/DCVC-RT/assets/bd_rate_speed.png b/DCVC-RT/assets/bd_rate_speed.png deleted file mode 100644 index 6da70d1..0000000 Binary files a/DCVC-RT/assets/bd_rate_speed.png and /dev/null differ diff --git a/DCVC-DC/README.md b/DCVC-family/DCVC-DC/README.md similarity index 90% rename from DCVC-DC/README.md rename to DCVC-family/DCVC-DC/README.md index 95dad3c..be9bbd6 100644 --- a/DCVC-DC/README.md +++ b/DCVC-family/DCVC-DC/README.md @@ -1,6 +1,9 @@ # Introduction Official Pytorch implementation for DCVC-DC: [Neural Video Compression with **D**iverse **C**ontexts](https://arxiv.org/abs/2302.14402), in CVPR 2023. +- The first end-to-end neural video codec to exceed [ECM](https://jvet-experts.org/doc_end_user/documents/27_Teleconference/wg11/JVET-AA0006-v1.zip) using the highest compression ratio low delay configuration with a intra refresh period roughly to one second (32 frames), in terms of PSNR and MS-SSIM for RGB content. +- The first end-to-end neural video codec to exceed ECM using the highest compression ratio low delay configuration with a intra refresh period roughly to one second (32 frames), in terms of PSNR for YUV420 content. + # Prerequisites * Python 3.8 and conda, get [Conda](https://www.anaconda.com/) diff --git a/DCVC-DC/assets/bitsaving.png b/DCVC-family/DCVC-DC/assets/bitsaving.png similarity index 100% rename from DCVC-DC/assets/bitsaving.png rename to DCVC-family/DCVC-DC/assets/bitsaving.png diff --git a/DCVC-DC/assets/rd_rgb_psnr.png b/DCVC-family/DCVC-DC/assets/rd_rgb_psnr.png similarity index 100% rename from DCVC-DC/assets/rd_rgb_psnr.png rename to DCVC-family/DCVC-DC/assets/rd_rgb_psnr.png diff --git a/DCVC-DC/assets/rd_yuv420_psnr.png b/DCVC-family/DCVC-DC/assets/rd_yuv420_psnr.png similarity index 100% rename from DCVC-DC/assets/rd_yuv420_psnr.png rename to DCVC-family/DCVC-DC/assets/rd_yuv420_psnr.png diff --git a/DCVC-DC/checkpoints/download.py b/DCVC-family/DCVC-DC/checkpoints/download.py similarity index 100% rename from DCVC-DC/checkpoints/download.py rename to DCVC-family/DCVC-DC/checkpoints/download.py diff --git a/DCVC-DC/dataset_config_example_rgb.json b/DCVC-family/DCVC-DC/dataset_config_example_rgb.json similarity index 100% rename from DCVC-DC/dataset_config_example_rgb.json rename to DCVC-family/DCVC-DC/dataset_config_example_rgb.json diff --git a/DCVC-DC/dataset_config_example_yuv420.json b/DCVC-family/DCVC-DC/dataset_config_example_yuv420.json similarity index 100% rename from DCVC-DC/dataset_config_example_yuv420.json rename to DCVC-family/DCVC-DC/dataset_config_example_yuv420.json diff --git a/DCVC-DC/requirements.txt b/DCVC-family/DCVC-DC/requirements.txt similarity index 100% rename from DCVC-DC/requirements.txt rename to DCVC-family/DCVC-DC/requirements.txt diff --git a/DCVC-DC/src/cpp/3rdparty/CMakeLists.txt b/DCVC-family/DCVC-DC/src/cpp/3rdparty/CMakeLists.txt similarity index 100% rename from DCVC-DC/src/cpp/3rdparty/CMakeLists.txt rename to DCVC-family/DCVC-DC/src/cpp/3rdparty/CMakeLists.txt diff --git a/DCVC-DC/src/cpp/3rdparty/pybind11/CMakeLists.txt b/DCVC-family/DCVC-DC/src/cpp/3rdparty/pybind11/CMakeLists.txt similarity index 100% rename from DCVC-DC/src/cpp/3rdparty/pybind11/CMakeLists.txt rename to DCVC-family/DCVC-DC/src/cpp/3rdparty/pybind11/CMakeLists.txt diff --git a/DCVC-DC/src/cpp/3rdparty/pybind11/CMakeLists.txt.in b/DCVC-family/DCVC-DC/src/cpp/3rdparty/pybind11/CMakeLists.txt.in similarity index 100% rename from DCVC-DC/src/cpp/3rdparty/pybind11/CMakeLists.txt.in rename to DCVC-family/DCVC-DC/src/cpp/3rdparty/pybind11/CMakeLists.txt.in diff --git a/DCVC-DC/src/cpp/3rdparty/ryg_rans/CMakeLists.txt b/DCVC-family/DCVC-DC/src/cpp/3rdparty/ryg_rans/CMakeLists.txt similarity index 100% rename from DCVC-DC/src/cpp/3rdparty/ryg_rans/CMakeLists.txt rename to DCVC-family/DCVC-DC/src/cpp/3rdparty/ryg_rans/CMakeLists.txt diff --git a/DCVC-DC/src/cpp/3rdparty/ryg_rans/CMakeLists.txt.in b/DCVC-family/DCVC-DC/src/cpp/3rdparty/ryg_rans/CMakeLists.txt.in similarity index 100% rename from DCVC-DC/src/cpp/3rdparty/ryg_rans/CMakeLists.txt.in rename to DCVC-family/DCVC-DC/src/cpp/3rdparty/ryg_rans/CMakeLists.txt.in diff --git a/DCVC-DC/src/cpp/CMakeLists.txt b/DCVC-family/DCVC-DC/src/cpp/CMakeLists.txt similarity index 100% rename from DCVC-DC/src/cpp/CMakeLists.txt rename to DCVC-family/DCVC-DC/src/cpp/CMakeLists.txt diff --git a/DCVC-DC/src/cpp/ops/CMakeLists.txt b/DCVC-family/DCVC-DC/src/cpp/ops/CMakeLists.txt similarity index 100% rename from DCVC-DC/src/cpp/ops/CMakeLists.txt rename to DCVC-family/DCVC-DC/src/cpp/ops/CMakeLists.txt diff --git a/DCVC-DC/src/cpp/ops/ops.cpp b/DCVC-family/DCVC-DC/src/cpp/ops/ops.cpp similarity index 100% rename from DCVC-DC/src/cpp/ops/ops.cpp rename to DCVC-family/DCVC-DC/src/cpp/ops/ops.cpp diff --git a/DCVC-DC/src/cpp/py_rans/CMakeLists.txt b/DCVC-family/DCVC-DC/src/cpp/py_rans/CMakeLists.txt similarity index 100% rename from DCVC-DC/src/cpp/py_rans/CMakeLists.txt rename to DCVC-family/DCVC-DC/src/cpp/py_rans/CMakeLists.txt diff --git a/DCVC-DC/src/cpp/py_rans/py_rans.cpp b/DCVC-family/DCVC-DC/src/cpp/py_rans/py_rans.cpp similarity index 100% rename from DCVC-DC/src/cpp/py_rans/py_rans.cpp rename to DCVC-family/DCVC-DC/src/cpp/py_rans/py_rans.cpp diff --git a/DCVC-DC/src/cpp/py_rans/py_rans.h b/DCVC-family/DCVC-DC/src/cpp/py_rans/py_rans.h similarity index 100% rename from DCVC-DC/src/cpp/py_rans/py_rans.h rename to DCVC-family/DCVC-DC/src/cpp/py_rans/py_rans.h diff --git a/DCVC-DC/src/cpp/rans/CMakeLists.txt b/DCVC-family/DCVC-DC/src/cpp/rans/CMakeLists.txt similarity index 100% rename from DCVC-DC/src/cpp/rans/CMakeLists.txt rename to DCVC-family/DCVC-DC/src/cpp/rans/CMakeLists.txt diff --git a/DCVC-DC/src/cpp/rans/rans.cpp b/DCVC-family/DCVC-DC/src/cpp/rans/rans.cpp similarity index 100% rename from DCVC-DC/src/cpp/rans/rans.cpp rename to DCVC-family/DCVC-DC/src/cpp/rans/rans.cpp diff --git a/DCVC-DC/src/cpp/rans/rans.h b/DCVC-family/DCVC-DC/src/cpp/rans/rans.h similarity index 100% rename from DCVC-DC/src/cpp/rans/rans.h rename to DCVC-family/DCVC-DC/src/cpp/rans/rans.h diff --git a/DCVC-DC/src/models/common_model.py b/DCVC-family/DCVC-DC/src/models/common_model.py similarity index 100% rename from DCVC-DC/src/models/common_model.py rename to DCVC-family/DCVC-DC/src/models/common_model.py diff --git a/DCVC-DC/src/models/entropy_models.py b/DCVC-family/DCVC-DC/src/models/entropy_models.py similarity index 100% rename from DCVC-DC/src/models/entropy_models.py rename to DCVC-family/DCVC-DC/src/models/entropy_models.py diff --git a/DCVC-DC/src/models/image_model.py b/DCVC-family/DCVC-DC/src/models/image_model.py similarity index 100% rename from DCVC-DC/src/models/image_model.py rename to DCVC-family/DCVC-DC/src/models/image_model.py diff --git a/DCVC-DC/src/models/layers.py b/DCVC-family/DCVC-DC/src/models/layers.py similarity index 100% rename from DCVC-DC/src/models/layers.py rename to DCVC-family/DCVC-DC/src/models/layers.py diff --git a/DCVC-DC/src/models/video_model.py b/DCVC-family/DCVC-DC/src/models/video_model.py similarity index 100% rename from DCVC-DC/src/models/video_model.py rename to DCVC-family/DCVC-DC/src/models/video_model.py diff --git a/DCVC-DC/src/models/video_net.py b/DCVC-family/DCVC-DC/src/models/video_net.py similarity index 100% rename from DCVC-DC/src/models/video_net.py rename to DCVC-family/DCVC-DC/src/models/video_net.py diff --git a/DCVC-DC/src/transforms/functional.py b/DCVC-family/DCVC-DC/src/transforms/functional.py similarity index 100% rename from DCVC-DC/src/transforms/functional.py rename to DCVC-family/DCVC-DC/src/transforms/functional.py diff --git a/DCVC-DC/src/utils/common.py b/DCVC-family/DCVC-DC/src/utils/common.py similarity index 100% rename from DCVC-DC/src/utils/common.py rename to DCVC-family/DCVC-DC/src/utils/common.py diff --git a/DCVC-DC/src/utils/metrics.py b/DCVC-family/DCVC-DC/src/utils/metrics.py similarity index 100% rename from DCVC-DC/src/utils/metrics.py rename to DCVC-family/DCVC-DC/src/utils/metrics.py diff --git a/DCVC-DC/src/utils/stream_helper.py b/DCVC-family/DCVC-DC/src/utils/stream_helper.py similarity index 100% rename from DCVC-DC/src/utils/stream_helper.py rename to DCVC-family/DCVC-DC/src/utils/stream_helper.py diff --git a/DCVC-DC/src/utils/video_reader.py b/DCVC-family/DCVC-DC/src/utils/video_reader.py similarity index 100% rename from DCVC-DC/src/utils/video_reader.py rename to DCVC-family/DCVC-DC/src/utils/video_reader.py diff --git a/DCVC-DC/src/utils/video_writer.py b/DCVC-family/DCVC-DC/src/utils/video_writer.py similarity index 100% rename from DCVC-DC/src/utils/video_writer.py rename to DCVC-family/DCVC-DC/src/utils/video_writer.py diff --git a/DCVC-DC/test_data_to_png.py b/DCVC-family/DCVC-DC/test_data_to_png.py similarity index 100% rename from DCVC-DC/test_data_to_png.py rename to DCVC-family/DCVC-DC/test_data_to_png.py diff --git a/DCVC-DC/test_video.py b/DCVC-family/DCVC-DC/test_video.py similarity index 100% rename from DCVC-DC/test_video.py rename to DCVC-family/DCVC-DC/test_video.py diff --git a/DCVC-FM/README.md b/DCVC-family/DCVC-FM/README.md similarity index 93% rename from DCVC-FM/README.md rename to DCVC-family/DCVC-FM/README.md index 5ac751e..c38fb2f 100644 --- a/DCVC-FM/README.md +++ b/DCVC-family/DCVC-FM/README.md @@ -1,6 +1,8 @@ # Introduction Official Pytorch implementation for DCVC-FM: [Neural Video Compression with **F**eature **M**odulation](https://arxiv.org/abs/2402.17414), in CVPR 2024. +- The first end-to-end neural video codec to exceed ECM using the highest compression ratio low delay configuration with only one intra frame, in terms of PSNR for both YUV420 content and RGB content in a single model. +- The first end-to-end neural video codec that support a large quality and bitrate range in a single model. # Prerequisites * Python 3.10 and conda, get [Conda](https://www.anaconda.com/) diff --git a/DCVC-FM/assets/bitsaving.png b/DCVC-family/DCVC-FM/assets/bitsaving.png similarity index 100% rename from DCVC-FM/assets/bitsaving.png rename to DCVC-family/DCVC-FM/assets/bitsaving.png diff --git a/DCVC-FM/assets/rd_yuv420_psnr.png b/DCVC-family/DCVC-FM/assets/rd_yuv420_psnr.png similarity index 100% rename from DCVC-FM/assets/rd_yuv420_psnr.png rename to DCVC-family/DCVC-FM/assets/rd_yuv420_psnr.png diff --git a/DCVC-FM/checkpoints/download.py b/DCVC-family/DCVC-FM/checkpoints/download.py similarity index 100% rename from DCVC-FM/checkpoints/download.py rename to DCVC-family/DCVC-FM/checkpoints/download.py diff --git a/DCVC-FM/dataset_config_example_rgb.json b/DCVC-family/DCVC-FM/dataset_config_example_rgb.json similarity index 100% rename from DCVC-FM/dataset_config_example_rgb.json rename to DCVC-family/DCVC-FM/dataset_config_example_rgb.json diff --git a/DCVC-FM/dataset_config_example_yuv420.json b/DCVC-family/DCVC-FM/dataset_config_example_yuv420.json similarity index 100% rename from DCVC-FM/dataset_config_example_yuv420.json rename to DCVC-family/DCVC-FM/dataset_config_example_yuv420.json diff --git a/DCVC-FM/requirements.txt b/DCVC-family/DCVC-FM/requirements.txt similarity index 100% rename from DCVC-FM/requirements.txt rename to DCVC-family/DCVC-FM/requirements.txt diff --git a/DCVC-FM/src/cpp/3rdparty/CMakeLists.txt b/DCVC-family/DCVC-FM/src/cpp/3rdparty/CMakeLists.txt similarity index 100% rename from DCVC-FM/src/cpp/3rdparty/CMakeLists.txt rename to DCVC-family/DCVC-FM/src/cpp/3rdparty/CMakeLists.txt diff --git a/DCVC-FM/src/cpp/3rdparty/pybind11/CMakeLists.txt b/DCVC-family/DCVC-FM/src/cpp/3rdparty/pybind11/CMakeLists.txt similarity index 100% rename from DCVC-FM/src/cpp/3rdparty/pybind11/CMakeLists.txt rename to DCVC-family/DCVC-FM/src/cpp/3rdparty/pybind11/CMakeLists.txt diff --git a/DCVC-FM/src/cpp/3rdparty/pybind11/CMakeLists.txt.in b/DCVC-family/DCVC-FM/src/cpp/3rdparty/pybind11/CMakeLists.txt.in similarity index 100% rename from DCVC-FM/src/cpp/3rdparty/pybind11/CMakeLists.txt.in rename to DCVC-family/DCVC-FM/src/cpp/3rdparty/pybind11/CMakeLists.txt.in diff --git a/DCVC-FM/src/cpp/CMakeLists.txt b/DCVC-family/DCVC-FM/src/cpp/CMakeLists.txt similarity index 100% rename from DCVC-FM/src/cpp/CMakeLists.txt rename to DCVC-family/DCVC-FM/src/cpp/CMakeLists.txt diff --git a/DCVC-FM/src/cpp/ops/CMakeLists.txt b/DCVC-family/DCVC-FM/src/cpp/ops/CMakeLists.txt similarity index 100% rename from DCVC-FM/src/cpp/ops/CMakeLists.txt rename to DCVC-family/DCVC-FM/src/cpp/ops/CMakeLists.txt diff --git a/DCVC-FM/src/cpp/ops/ops.cpp b/DCVC-family/DCVC-FM/src/cpp/ops/ops.cpp similarity index 100% rename from DCVC-FM/src/cpp/ops/ops.cpp rename to DCVC-family/DCVC-FM/src/cpp/ops/ops.cpp diff --git a/DCVC-FM/src/cpp/py_rans/CMakeLists.txt b/DCVC-family/DCVC-FM/src/cpp/py_rans/CMakeLists.txt similarity index 100% rename from DCVC-FM/src/cpp/py_rans/CMakeLists.txt rename to DCVC-family/DCVC-FM/src/cpp/py_rans/CMakeLists.txt diff --git a/DCVC-FM/src/cpp/py_rans/py_rans.cpp b/DCVC-family/DCVC-FM/src/cpp/py_rans/py_rans.cpp similarity index 100% rename from DCVC-FM/src/cpp/py_rans/py_rans.cpp rename to DCVC-family/DCVC-FM/src/cpp/py_rans/py_rans.cpp diff --git a/DCVC-FM/src/cpp/py_rans/py_rans.h b/DCVC-family/DCVC-FM/src/cpp/py_rans/py_rans.h similarity index 100% rename from DCVC-FM/src/cpp/py_rans/py_rans.h rename to DCVC-family/DCVC-FM/src/cpp/py_rans/py_rans.h diff --git a/DCVC-FM/src/cpp/rans/CMakeLists.txt b/DCVC-family/DCVC-FM/src/cpp/rans/CMakeLists.txt similarity index 100% rename from DCVC-FM/src/cpp/rans/CMakeLists.txt rename to DCVC-family/DCVC-FM/src/cpp/rans/CMakeLists.txt diff --git a/DCVC-FM/src/cpp/rans/rans.cpp b/DCVC-family/DCVC-FM/src/cpp/rans/rans.cpp similarity index 100% rename from DCVC-FM/src/cpp/rans/rans.cpp rename to DCVC-family/DCVC-FM/src/cpp/rans/rans.cpp diff --git a/DCVC-FM/src/cpp/rans/rans.h b/DCVC-family/DCVC-FM/src/cpp/rans/rans.h similarity index 100% rename from DCVC-FM/src/cpp/rans/rans.h rename to DCVC-family/DCVC-FM/src/cpp/rans/rans.h diff --git a/DCVC-FM/src/cpp/rans/rans_byte.h b/DCVC-family/DCVC-FM/src/cpp/rans/rans_byte.h similarity index 100% rename from DCVC-FM/src/cpp/rans/rans_byte.h rename to DCVC-family/DCVC-FM/src/cpp/rans/rans_byte.h diff --git a/DCVC-FM/src/models/block_mc.py b/DCVC-family/DCVC-FM/src/models/block_mc.py similarity index 100% rename from DCVC-FM/src/models/block_mc.py rename to DCVC-family/DCVC-FM/src/models/block_mc.py diff --git a/DCVC-FM/src/models/common_model.py b/DCVC-family/DCVC-FM/src/models/common_model.py similarity index 100% rename from DCVC-FM/src/models/common_model.py rename to DCVC-family/DCVC-FM/src/models/common_model.py diff --git a/DCVC-FM/src/models/entropy_models.py b/DCVC-family/DCVC-FM/src/models/entropy_models.py similarity index 100% rename from DCVC-FM/src/models/entropy_models.py rename to DCVC-family/DCVC-FM/src/models/entropy_models.py diff --git a/DCVC-FM/src/models/extensions/block_mc.cpp b/DCVC-family/DCVC-FM/src/models/extensions/block_mc.cpp similarity index 100% rename from DCVC-FM/src/models/extensions/block_mc.cpp rename to DCVC-family/DCVC-FM/src/models/extensions/block_mc.cpp diff --git a/DCVC-FM/src/models/extensions/block_mc.h b/DCVC-family/DCVC-FM/src/models/extensions/block_mc.h similarity index 100% rename from DCVC-FM/src/models/extensions/block_mc.h rename to DCVC-family/DCVC-FM/src/models/extensions/block_mc.h diff --git a/DCVC-FM/src/models/extensions/block_mc_kernel.cu b/DCVC-family/DCVC-FM/src/models/extensions/block_mc_kernel.cu similarity index 100% rename from DCVC-FM/src/models/extensions/block_mc_kernel.cu rename to DCVC-family/DCVC-FM/src/models/extensions/block_mc_kernel.cu diff --git a/DCVC-FM/src/models/extensions/common.h b/DCVC-family/DCVC-FM/src/models/extensions/common.h similarity index 100% rename from DCVC-FM/src/models/extensions/common.h rename to DCVC-family/DCVC-FM/src/models/extensions/common.h diff --git a/DCVC-FM/src/models/extensions/setup.py b/DCVC-family/DCVC-FM/src/models/extensions/setup.py similarity index 100% rename from DCVC-FM/src/models/extensions/setup.py rename to DCVC-family/DCVC-FM/src/models/extensions/setup.py diff --git a/DCVC-FM/src/models/image_model.py b/DCVC-family/DCVC-FM/src/models/image_model.py similarity index 100% rename from DCVC-FM/src/models/image_model.py rename to DCVC-family/DCVC-FM/src/models/image_model.py diff --git a/DCVC-FM/src/models/layers.py b/DCVC-family/DCVC-FM/src/models/layers.py similarity index 100% rename from DCVC-FM/src/models/layers.py rename to DCVC-family/DCVC-FM/src/models/layers.py diff --git a/DCVC-FM/src/models/video_model.py b/DCVC-family/DCVC-FM/src/models/video_model.py similarity index 100% rename from DCVC-FM/src/models/video_model.py rename to DCVC-family/DCVC-FM/src/models/video_model.py diff --git a/DCVC-FM/src/models/video_net.py b/DCVC-family/DCVC-FM/src/models/video_net.py similarity index 100% rename from DCVC-FM/src/models/video_net.py rename to DCVC-family/DCVC-FM/src/models/video_net.py diff --git a/DCVC-FM/src/transforms/functional.py b/DCVC-family/DCVC-FM/src/transforms/functional.py similarity index 100% rename from DCVC-FM/src/transforms/functional.py rename to DCVC-family/DCVC-FM/src/transforms/functional.py diff --git a/DCVC-FM/src/transforms/transforms.py b/DCVC-family/DCVC-FM/src/transforms/transforms.py similarity index 100% rename from DCVC-FM/src/transforms/transforms.py rename to DCVC-family/DCVC-FM/src/transforms/transforms.py diff --git a/DCVC-FM/src/utils/common.py b/DCVC-family/DCVC-FM/src/utils/common.py similarity index 100% rename from DCVC-FM/src/utils/common.py rename to DCVC-family/DCVC-FM/src/utils/common.py diff --git a/DCVC-FM/src/utils/metrics.py b/DCVC-family/DCVC-FM/src/utils/metrics.py similarity index 100% rename from DCVC-FM/src/utils/metrics.py rename to DCVC-family/DCVC-FM/src/utils/metrics.py diff --git a/DCVC-FM/src/utils/stream_helper.py b/DCVC-family/DCVC-FM/src/utils/stream_helper.py similarity index 100% rename from DCVC-FM/src/utils/stream_helper.py rename to DCVC-family/DCVC-FM/src/utils/stream_helper.py diff --git a/DCVC-FM/src/utils/test_helper.py b/DCVC-family/DCVC-FM/src/utils/test_helper.py similarity index 100% rename from DCVC-FM/src/utils/test_helper.py rename to DCVC-family/DCVC-FM/src/utils/test_helper.py diff --git a/DCVC-FM/src/utils/video_reader.py b/DCVC-family/DCVC-FM/src/utils/video_reader.py similarity index 100% rename from DCVC-FM/src/utils/video_reader.py rename to DCVC-family/DCVC-FM/src/utils/video_reader.py diff --git a/DCVC-FM/src/utils/video_writer.py b/DCVC-family/DCVC-FM/src/utils/video_writer.py similarity index 100% rename from DCVC-FM/src/utils/video_writer.py rename to DCVC-family/DCVC-FM/src/utils/video_writer.py diff --git a/DCVC-FM/test_data_to_png.py b/DCVC-family/DCVC-FM/test_data_to_png.py similarity index 100% rename from DCVC-FM/test_data_to_png.py rename to DCVC-family/DCVC-FM/test_data_to_png.py diff --git a/DCVC-FM/test_video.py b/DCVC-family/DCVC-FM/test_video.py similarity index 100% rename from DCVC-FM/test_video.py rename to DCVC-family/DCVC-FM/test_video.py diff --git a/DCVC-HEM/README.md b/DCVC-family/DCVC-HEM/README.md similarity index 95% rename from DCVC-HEM/README.md rename to DCVC-family/DCVC-HEM/README.md index be7c40b..a3ceba9 100644 --- a/DCVC-HEM/README.md +++ b/DCVC-family/DCVC-HEM/README.md @@ -1,6 +1,8 @@ # Introduction Official Pytorch implementation for DCVC-HEM: [**H**ybrid Spatial-Temporal **E**ntropy **M**odelling for Neural Video Compression](https://arxiv.org/abs/2207.05894), in ACM MM 2022. +- The first end-to-end neural video codec to exceed H.266 (VTM) using the highest compression ratio configuration, in terms of both PSNR and MS-SSIM. +- The first end-to-end neural video codec to achieve rate adjustment in single model. # Prerequisites * Python 3.8 and conda, get [Conda](https://www.anaconda.com/) diff --git a/DCVC-HEM/assets/rd_curve_psnr.png b/DCVC-family/DCVC-HEM/assets/rd_curve_psnr.png similarity index 100% rename from DCVC-HEM/assets/rd_curve_psnr.png rename to DCVC-family/DCVC-HEM/assets/rd_curve_psnr.png diff --git a/DCVC-HEM/checkpoints/download.py b/DCVC-family/DCVC-HEM/checkpoints/download.py similarity index 100% rename from DCVC-HEM/checkpoints/download.py rename to DCVC-family/DCVC-HEM/checkpoints/download.py diff --git a/DCVC-HEM/dataset_config_example.json b/DCVC-family/DCVC-HEM/dataset_config_example.json similarity index 100% rename from DCVC-HEM/dataset_config_example.json rename to DCVC-family/DCVC-HEM/dataset_config_example.json diff --git a/DCVC-HEM/requirements.txt b/DCVC-family/DCVC-HEM/requirements.txt similarity index 100% rename from DCVC-HEM/requirements.txt rename to DCVC-family/DCVC-HEM/requirements.txt diff --git a/DCVC-HEM/src/cpp/3rdparty/CMakeLists.txt b/DCVC-family/DCVC-HEM/src/cpp/3rdparty/CMakeLists.txt similarity index 100% rename from DCVC-HEM/src/cpp/3rdparty/CMakeLists.txt rename to DCVC-family/DCVC-HEM/src/cpp/3rdparty/CMakeLists.txt diff --git a/DCVC-HEM/src/cpp/3rdparty/pybind11/CMakeLists.txt b/DCVC-family/DCVC-HEM/src/cpp/3rdparty/pybind11/CMakeLists.txt similarity index 100% rename from DCVC-HEM/src/cpp/3rdparty/pybind11/CMakeLists.txt rename to DCVC-family/DCVC-HEM/src/cpp/3rdparty/pybind11/CMakeLists.txt diff --git a/DCVC-HEM/src/cpp/3rdparty/pybind11/CMakeLists.txt.in b/DCVC-family/DCVC-HEM/src/cpp/3rdparty/pybind11/CMakeLists.txt.in similarity index 100% rename from DCVC-HEM/src/cpp/3rdparty/pybind11/CMakeLists.txt.in rename to DCVC-family/DCVC-HEM/src/cpp/3rdparty/pybind11/CMakeLists.txt.in diff --git a/DCVC-HEM/src/cpp/3rdparty/ryg_rans/CMakeLists.txt b/DCVC-family/DCVC-HEM/src/cpp/3rdparty/ryg_rans/CMakeLists.txt similarity index 100% rename from DCVC-HEM/src/cpp/3rdparty/ryg_rans/CMakeLists.txt rename to DCVC-family/DCVC-HEM/src/cpp/3rdparty/ryg_rans/CMakeLists.txt diff --git a/DCVC-HEM/src/cpp/3rdparty/ryg_rans/CMakeLists.txt.in b/DCVC-family/DCVC-HEM/src/cpp/3rdparty/ryg_rans/CMakeLists.txt.in similarity index 100% rename from DCVC-HEM/src/cpp/3rdparty/ryg_rans/CMakeLists.txt.in rename to DCVC-family/DCVC-HEM/src/cpp/3rdparty/ryg_rans/CMakeLists.txt.in diff --git a/DCVC-HEM/src/cpp/CMakeLists.txt b/DCVC-family/DCVC-HEM/src/cpp/CMakeLists.txt similarity index 100% rename from DCVC-HEM/src/cpp/CMakeLists.txt rename to DCVC-family/DCVC-HEM/src/cpp/CMakeLists.txt diff --git a/DCVC-HEM/src/cpp/ops/CMakeLists.txt b/DCVC-family/DCVC-HEM/src/cpp/ops/CMakeLists.txt similarity index 100% rename from DCVC-HEM/src/cpp/ops/CMakeLists.txt rename to DCVC-family/DCVC-HEM/src/cpp/ops/CMakeLists.txt diff --git a/DCVC-HEM/src/cpp/ops/ops.cpp b/DCVC-family/DCVC-HEM/src/cpp/ops/ops.cpp similarity index 100% rename from DCVC-HEM/src/cpp/ops/ops.cpp rename to DCVC-family/DCVC-HEM/src/cpp/ops/ops.cpp diff --git a/DCVC-HEM/src/cpp/rans/CMakeLists.txt b/DCVC-family/DCVC-HEM/src/cpp/rans/CMakeLists.txt similarity index 100% rename from DCVC-HEM/src/cpp/rans/CMakeLists.txt rename to DCVC-family/DCVC-HEM/src/cpp/rans/CMakeLists.txt diff --git a/DCVC-HEM/src/cpp/rans/rans_interface.cpp b/DCVC-family/DCVC-HEM/src/cpp/rans/rans_interface.cpp similarity index 100% rename from DCVC-HEM/src/cpp/rans/rans_interface.cpp rename to DCVC-family/DCVC-HEM/src/cpp/rans/rans_interface.cpp diff --git a/DCVC-HEM/src/cpp/rans/rans_interface.hpp b/DCVC-family/DCVC-HEM/src/cpp/rans/rans_interface.hpp similarity index 100% rename from DCVC-HEM/src/cpp/rans/rans_interface.hpp rename to DCVC-family/DCVC-HEM/src/cpp/rans/rans_interface.hpp diff --git a/DCVC-HEM/src/entropy_models/entropy_models.py b/DCVC-family/DCVC-HEM/src/entropy_models/entropy_models.py similarity index 100% rename from DCVC-HEM/src/entropy_models/entropy_models.py rename to DCVC-family/DCVC-HEM/src/entropy_models/entropy_models.py diff --git a/DCVC-HEM/src/layers/layers.py b/DCVC-family/DCVC-HEM/src/layers/layers.py similarity index 100% rename from DCVC-HEM/src/layers/layers.py rename to DCVC-family/DCVC-HEM/src/layers/layers.py diff --git a/DCVC-HEM/src/models/common_model.py b/DCVC-family/DCVC-HEM/src/models/common_model.py similarity index 100% rename from DCVC-HEM/src/models/common_model.py rename to DCVC-family/DCVC-HEM/src/models/common_model.py diff --git a/DCVC-HEM/src/models/image_model.py b/DCVC-family/DCVC-HEM/src/models/image_model.py similarity index 100% rename from DCVC-HEM/src/models/image_model.py rename to DCVC-family/DCVC-HEM/src/models/image_model.py diff --git a/DCVC-HEM/src/models/video_model.py b/DCVC-family/DCVC-HEM/src/models/video_model.py similarity index 100% rename from DCVC-HEM/src/models/video_model.py rename to DCVC-family/DCVC-HEM/src/models/video_model.py diff --git a/DCVC-HEM/src/models/video_net.py b/DCVC-family/DCVC-HEM/src/models/video_net.py similarity index 100% rename from DCVC-HEM/src/models/video_net.py rename to DCVC-family/DCVC-HEM/src/models/video_net.py diff --git a/DCVC-HEM/src/utils/common.py b/DCVC-family/DCVC-HEM/src/utils/common.py similarity index 100% rename from DCVC-HEM/src/utils/common.py rename to DCVC-family/DCVC-HEM/src/utils/common.py diff --git a/DCVC-HEM/src/utils/png_reader.py b/DCVC-family/DCVC-HEM/src/utils/png_reader.py similarity index 100% rename from DCVC-HEM/src/utils/png_reader.py rename to DCVC-family/DCVC-HEM/src/utils/png_reader.py diff --git a/DCVC-HEM/src/utils/stream_helper.py b/DCVC-family/DCVC-HEM/src/utils/stream_helper.py similarity index 100% rename from DCVC-HEM/src/utils/stream_helper.py rename to DCVC-family/DCVC-HEM/src/utils/stream_helper.py diff --git a/DCVC-HEM/test_video.py b/DCVC-family/DCVC-HEM/test_video.py similarity index 100% rename from DCVC-HEM/test_video.py rename to DCVC-family/DCVC-HEM/test_video.py diff --git a/DCVC-TCM/README.md b/DCVC-family/DCVC-TCM/README.md similarity index 100% rename from DCVC-TCM/README.md rename to DCVC-family/DCVC-TCM/README.md diff --git a/DCVC-TCM/checkpoints/.gitkeep b/DCVC-family/DCVC-TCM/checkpoints/.gitkeep similarity index 100% rename from DCVC-TCM/checkpoints/.gitkeep rename to DCVC-family/DCVC-TCM/checkpoints/.gitkeep diff --git a/DCVC-TCM/recommended_test_full_results_IP12.json b/DCVC-family/DCVC-TCM/recommended_test_full_results_IP12.json similarity index 100% rename from DCVC-TCM/recommended_test_full_results_IP12.json rename to DCVC-family/DCVC-TCM/recommended_test_full_results_IP12.json diff --git a/DCVC-TCM/recommended_test_full_results_IP32.json b/DCVC-family/DCVC-TCM/recommended_test_full_results_IP32.json similarity index 100% rename from DCVC-TCM/recommended_test_full_results_IP32.json rename to DCVC-family/DCVC-TCM/recommended_test_full_results_IP32.json diff --git a/DCVC-TCM/requirements.txt b/DCVC-family/DCVC-TCM/requirements.txt similarity index 100% rename from DCVC-TCM/requirements.txt rename to DCVC-family/DCVC-TCM/requirements.txt diff --git a/DCVC-TCM/src/cpp/3rdparty/CMakeLists.txt b/DCVC-family/DCVC-TCM/src/cpp/3rdparty/CMakeLists.txt similarity index 100% rename from DCVC-TCM/src/cpp/3rdparty/CMakeLists.txt rename to DCVC-family/DCVC-TCM/src/cpp/3rdparty/CMakeLists.txt diff --git a/DCVC-TCM/src/cpp/3rdparty/pybind11/CMakeLists.txt b/DCVC-family/DCVC-TCM/src/cpp/3rdparty/pybind11/CMakeLists.txt similarity index 100% rename from DCVC-TCM/src/cpp/3rdparty/pybind11/CMakeLists.txt rename to DCVC-family/DCVC-TCM/src/cpp/3rdparty/pybind11/CMakeLists.txt diff --git a/DCVC-TCM/src/cpp/3rdparty/pybind11/CMakeLists.txt.in b/DCVC-family/DCVC-TCM/src/cpp/3rdparty/pybind11/CMakeLists.txt.in similarity index 100% rename from DCVC-TCM/src/cpp/3rdparty/pybind11/CMakeLists.txt.in rename to DCVC-family/DCVC-TCM/src/cpp/3rdparty/pybind11/CMakeLists.txt.in diff --git a/DCVC-TCM/src/cpp/3rdparty/ryg_rans/CMakeLists.txt b/DCVC-family/DCVC-TCM/src/cpp/3rdparty/ryg_rans/CMakeLists.txt similarity index 100% rename from DCVC-TCM/src/cpp/3rdparty/ryg_rans/CMakeLists.txt rename to DCVC-family/DCVC-TCM/src/cpp/3rdparty/ryg_rans/CMakeLists.txt diff --git a/DCVC-TCM/src/cpp/3rdparty/ryg_rans/CMakeLists.txt.in b/DCVC-family/DCVC-TCM/src/cpp/3rdparty/ryg_rans/CMakeLists.txt.in similarity index 100% rename from DCVC-TCM/src/cpp/3rdparty/ryg_rans/CMakeLists.txt.in rename to DCVC-family/DCVC-TCM/src/cpp/3rdparty/ryg_rans/CMakeLists.txt.in diff --git a/DCVC-TCM/src/cpp/CMakeLists.txt b/DCVC-family/DCVC-TCM/src/cpp/CMakeLists.txt similarity index 100% rename from DCVC-TCM/src/cpp/CMakeLists.txt rename to DCVC-family/DCVC-TCM/src/cpp/CMakeLists.txt diff --git a/DCVC-TCM/src/cpp/ops/CMakeLists.txt b/DCVC-family/DCVC-TCM/src/cpp/ops/CMakeLists.txt similarity index 100% rename from DCVC-TCM/src/cpp/ops/CMakeLists.txt rename to DCVC-family/DCVC-TCM/src/cpp/ops/CMakeLists.txt diff --git a/DCVC-TCM/src/cpp/ops/ops.cpp b/DCVC-family/DCVC-TCM/src/cpp/ops/ops.cpp similarity index 100% rename from DCVC-TCM/src/cpp/ops/ops.cpp rename to DCVC-family/DCVC-TCM/src/cpp/ops/ops.cpp diff --git a/DCVC-TCM/src/cpp/rans/CMakeLists.txt b/DCVC-family/DCVC-TCM/src/cpp/rans/CMakeLists.txt similarity index 100% rename from DCVC-TCM/src/cpp/rans/CMakeLists.txt rename to DCVC-family/DCVC-TCM/src/cpp/rans/CMakeLists.txt diff --git a/DCVC-TCM/src/cpp/rans/rans_interface.cpp b/DCVC-family/DCVC-TCM/src/cpp/rans/rans_interface.cpp similarity index 100% rename from DCVC-TCM/src/cpp/rans/rans_interface.cpp rename to DCVC-family/DCVC-TCM/src/cpp/rans/rans_interface.cpp diff --git a/DCVC-TCM/src/cpp/rans/rans_interface.hpp b/DCVC-family/DCVC-TCM/src/cpp/rans/rans_interface.hpp similarity index 100% rename from DCVC-TCM/src/cpp/rans/rans_interface.hpp rename to DCVC-family/DCVC-TCM/src/cpp/rans/rans_interface.hpp diff --git a/DCVC-TCM/src/entropy_models/entropy_models.py b/DCVC-family/DCVC-TCM/src/entropy_models/entropy_models.py similarity index 100% rename from DCVC-TCM/src/entropy_models/entropy_models.py rename to DCVC-family/DCVC-TCM/src/entropy_models/entropy_models.py diff --git a/DCVC-TCM/src/entropy_models/video_entropy_models.py b/DCVC-family/DCVC-TCM/src/entropy_models/video_entropy_models.py similarity index 100% rename from DCVC-TCM/src/entropy_models/video_entropy_models.py rename to DCVC-family/DCVC-TCM/src/entropy_models/video_entropy_models.py diff --git a/DCVC-TCM/src/layers/gdn.py b/DCVC-family/DCVC-TCM/src/layers/gdn.py similarity index 100% rename from DCVC-TCM/src/layers/gdn.py rename to DCVC-family/DCVC-TCM/src/layers/gdn.py diff --git a/DCVC-TCM/src/layers/layers.py b/DCVC-family/DCVC-TCM/src/layers/layers.py similarity index 100% rename from DCVC-TCM/src/layers/layers.py rename to DCVC-family/DCVC-TCM/src/layers/layers.py diff --git a/DCVC-TCM/src/models/priors.py b/DCVC-family/DCVC-TCM/src/models/priors.py similarity index 100% rename from DCVC-TCM/src/models/priors.py rename to DCVC-family/DCVC-TCM/src/models/priors.py diff --git a/DCVC-TCM/src/models/utils.py b/DCVC-family/DCVC-TCM/src/models/utils.py similarity index 100% rename from DCVC-TCM/src/models/utils.py rename to DCVC-family/DCVC-TCM/src/models/utils.py diff --git a/DCVC-TCM/src/models/video_net.py b/DCVC-family/DCVC-TCM/src/models/video_net.py similarity index 100% rename from DCVC-TCM/src/models/video_net.py rename to DCVC-family/DCVC-TCM/src/models/video_net.py diff --git a/DCVC-TCM/src/models/video_net_dmc.py b/DCVC-family/DCVC-TCM/src/models/video_net_dmc.py similarity index 100% rename from DCVC-TCM/src/models/video_net_dmc.py rename to DCVC-family/DCVC-TCM/src/models/video_net_dmc.py diff --git a/DCVC-TCM/src/ops/bound_ops.py b/DCVC-family/DCVC-TCM/src/ops/bound_ops.py similarity index 100% rename from DCVC-TCM/src/ops/bound_ops.py rename to DCVC-family/DCVC-TCM/src/ops/bound_ops.py diff --git a/DCVC-TCM/src/ops/parametrizers.py b/DCVC-family/DCVC-TCM/src/ops/parametrizers.py similarity index 100% rename from DCVC-TCM/src/ops/parametrizers.py rename to DCVC-family/DCVC-TCM/src/ops/parametrizers.py diff --git a/DCVC-TCM/src/utils/stream_helper.py b/DCVC-family/DCVC-TCM/src/utils/stream_helper.py similarity index 100% rename from DCVC-TCM/src/utils/stream_helper.py rename to DCVC-family/DCVC-TCM/src/utils/stream_helper.py diff --git a/DCVC-TCM/test_video.py b/DCVC-family/DCVC-TCM/test_video.py similarity index 100% rename from DCVC-TCM/test_video.py rename to DCVC-family/DCVC-TCM/test_video.py diff --git a/DCVC/README.md b/DCVC-family/DCVC/README.md similarity index 100% rename from DCVC/README.md rename to DCVC-family/DCVC/README.md diff --git a/DCVC/assets/rd_curve_psnr.png b/DCVC-family/DCVC/assets/rd_curve_psnr.png similarity index 100% rename from DCVC/assets/rd_curve_psnr.png rename to DCVC-family/DCVC/assets/rd_curve_psnr.png diff --git a/DCVC/checkpoints/download_compressai_models.py b/DCVC-family/DCVC/checkpoints/download_compressai_models.py similarity index 100% rename from DCVC/checkpoints/download_compressai_models.py rename to DCVC-family/DCVC/checkpoints/download_compressai_models.py diff --git a/DCVC/dataset_config_example.json b/DCVC-family/DCVC/dataset_config_example.json similarity index 100% rename from DCVC/dataset_config_example.json rename to DCVC-family/DCVC/dataset_config_example.json diff --git a/DCVC/requirements.txt b/DCVC-family/DCVC/requirements.txt similarity index 100% rename from DCVC/requirements.txt rename to DCVC-family/DCVC/requirements.txt diff --git a/DCVC/src/cpp/3rdparty/CMakeLists.txt b/DCVC-family/DCVC/src/cpp/3rdparty/CMakeLists.txt similarity index 100% rename from DCVC/src/cpp/3rdparty/CMakeLists.txt rename to DCVC-family/DCVC/src/cpp/3rdparty/CMakeLists.txt diff --git a/DCVC/src/cpp/3rdparty/pybind11/CMakeLists.txt b/DCVC-family/DCVC/src/cpp/3rdparty/pybind11/CMakeLists.txt similarity index 100% rename from DCVC/src/cpp/3rdparty/pybind11/CMakeLists.txt rename to DCVC-family/DCVC/src/cpp/3rdparty/pybind11/CMakeLists.txt diff --git a/DCVC/src/cpp/3rdparty/pybind11/CMakeLists.txt.in b/DCVC-family/DCVC/src/cpp/3rdparty/pybind11/CMakeLists.txt.in similarity index 100% rename from DCVC/src/cpp/3rdparty/pybind11/CMakeLists.txt.in rename to DCVC-family/DCVC/src/cpp/3rdparty/pybind11/CMakeLists.txt.in diff --git a/DCVC/src/cpp/3rdparty/ryg_rans/CMakeLists.txt b/DCVC-family/DCVC/src/cpp/3rdparty/ryg_rans/CMakeLists.txt similarity index 100% rename from DCVC/src/cpp/3rdparty/ryg_rans/CMakeLists.txt rename to DCVC-family/DCVC/src/cpp/3rdparty/ryg_rans/CMakeLists.txt diff --git a/DCVC/src/cpp/3rdparty/ryg_rans/CMakeLists.txt.in b/DCVC-family/DCVC/src/cpp/3rdparty/ryg_rans/CMakeLists.txt.in similarity index 100% rename from DCVC/src/cpp/3rdparty/ryg_rans/CMakeLists.txt.in rename to DCVC-family/DCVC/src/cpp/3rdparty/ryg_rans/CMakeLists.txt.in diff --git a/DCVC/src/cpp/CMakeLists.txt b/DCVC-family/DCVC/src/cpp/CMakeLists.txt similarity index 100% rename from DCVC/src/cpp/CMakeLists.txt rename to DCVC-family/DCVC/src/cpp/CMakeLists.txt diff --git a/DCVC/src/cpp/ops/CMakeLists.txt b/DCVC-family/DCVC/src/cpp/ops/CMakeLists.txt similarity index 100% rename from DCVC/src/cpp/ops/CMakeLists.txt rename to DCVC-family/DCVC/src/cpp/ops/CMakeLists.txt diff --git a/DCVC/src/cpp/ops/ops.cpp b/DCVC-family/DCVC/src/cpp/ops/ops.cpp similarity index 100% rename from DCVC/src/cpp/ops/ops.cpp rename to DCVC-family/DCVC/src/cpp/ops/ops.cpp diff --git a/DCVC/src/cpp/rans/CMakeLists.txt b/DCVC-family/DCVC/src/cpp/rans/CMakeLists.txt similarity index 100% rename from DCVC/src/cpp/rans/CMakeLists.txt rename to DCVC-family/DCVC/src/cpp/rans/CMakeLists.txt diff --git a/DCVC/src/cpp/rans/rans_interface.cpp b/DCVC-family/DCVC/src/cpp/rans/rans_interface.cpp similarity index 100% rename from DCVC/src/cpp/rans/rans_interface.cpp rename to DCVC-family/DCVC/src/cpp/rans/rans_interface.cpp diff --git a/DCVC/src/cpp/rans/rans_interface.hpp b/DCVC-family/DCVC/src/cpp/rans/rans_interface.hpp similarity index 100% rename from DCVC/src/cpp/rans/rans_interface.hpp rename to DCVC-family/DCVC/src/cpp/rans/rans_interface.hpp diff --git a/DCVC/src/entropy_models/entropy_models.py b/DCVC-family/DCVC/src/entropy_models/entropy_models.py similarity index 100% rename from DCVC/src/entropy_models/entropy_models.py rename to DCVC-family/DCVC/src/entropy_models/entropy_models.py diff --git a/DCVC/src/entropy_models/video_entropy_models.py b/DCVC-family/DCVC/src/entropy_models/video_entropy_models.py similarity index 100% rename from DCVC/src/entropy_models/video_entropy_models.py rename to DCVC-family/DCVC/src/entropy_models/video_entropy_models.py diff --git a/DCVC/src/layers/gdn.py b/DCVC-family/DCVC/src/layers/gdn.py similarity index 100% rename from DCVC/src/layers/gdn.py rename to DCVC-family/DCVC/src/layers/gdn.py diff --git a/DCVC/src/layers/layers.py b/DCVC-family/DCVC/src/layers/layers.py similarity index 100% rename from DCVC/src/layers/layers.py rename to DCVC-family/DCVC/src/layers/layers.py diff --git a/DCVC/src/models/DCVC_net.py b/DCVC-family/DCVC/src/models/DCVC_net.py similarity index 100% rename from DCVC/src/models/DCVC_net.py rename to DCVC-family/DCVC/src/models/DCVC_net.py diff --git a/DCVC/src/models/priors.py b/DCVC-family/DCVC/src/models/priors.py similarity index 100% rename from DCVC/src/models/priors.py rename to DCVC-family/DCVC/src/models/priors.py diff --git a/DCVC/src/models/utils.py b/DCVC-family/DCVC/src/models/utils.py similarity index 100% rename from DCVC/src/models/utils.py rename to DCVC-family/DCVC/src/models/utils.py diff --git a/DCVC/src/models/video_net.py b/DCVC-family/DCVC/src/models/video_net.py similarity index 100% rename from DCVC/src/models/video_net.py rename to DCVC-family/DCVC/src/models/video_net.py diff --git a/DCVC/src/models/waseda.py b/DCVC-family/DCVC/src/models/waseda.py similarity index 100% rename from DCVC/src/models/waseda.py rename to DCVC-family/DCVC/src/models/waseda.py diff --git a/DCVC/src/ops/bound_ops.py b/DCVC-family/DCVC/src/ops/bound_ops.py similarity index 100% rename from DCVC/src/ops/bound_ops.py rename to DCVC-family/DCVC/src/ops/bound_ops.py diff --git a/DCVC/src/ops/parametrizers.py b/DCVC-family/DCVC/src/ops/parametrizers.py similarity index 100% rename from DCVC/src/ops/parametrizers.py rename to DCVC-family/DCVC/src/ops/parametrizers.py diff --git a/DCVC/src/utils/stream_helper.py b/DCVC-family/DCVC/src/utils/stream_helper.py similarity index 100% rename from DCVC/src/utils/stream_helper.py rename to DCVC-family/DCVC/src/utils/stream_helper.py diff --git a/DCVC/src/zoo/image.py b/DCVC-family/DCVC/src/zoo/image.py similarity index 100% rename from DCVC/src/zoo/image.py rename to DCVC-family/DCVC/src/zoo/image.py diff --git a/DCVC/test_video.py b/DCVC-family/DCVC/test_video.py similarity index 100% rename from DCVC/test_video.py rename to DCVC-family/DCVC/test_video.py diff --git a/DCVC/write_stream_readme.md b/DCVC-family/DCVC/write_stream_readme.md similarity index 100% rename from DCVC/write_stream_readme.md rename to DCVC-family/DCVC/write_stream_readme.md diff --git a/EVC/README.md b/DCVC-family/EVC/README.md similarity index 100% rename from EVC/README.md rename to DCVC-family/EVC/README.md diff --git a/EVC/assets/RD.png b/DCVC-family/EVC/assets/RD.png similarity index 100% rename from EVC/assets/RD.png rename to DCVC-family/EVC/assets/RD.png diff --git a/EVC/assets/dualprior.png b/DCVC-family/EVC/assets/dualprior.png similarity index 100% rename from EVC/assets/dualprior.png rename to DCVC-family/EVC/assets/dualprior.png diff --git a/EVC/assets/enc_dec.png b/DCVC-family/EVC/assets/enc_dec.png similarity index 100% rename from EVC/assets/enc_dec.png rename to DCVC-family/EVC/assets/enc_dec.png diff --git a/EVC/assets/framework.png b/DCVC-family/EVC/assets/framework.png similarity index 100% rename from EVC/assets/framework.png rename to DCVC-family/EVC/assets/framework.png diff --git a/EVC/assets/hyperprior.png b/DCVC-family/EVC/assets/hyperprior.png similarity index 100% rename from EVC/assets/hyperprior.png rename to DCVC-family/EVC/assets/hyperprior.png diff --git a/EVC/checkpoints/download.py b/DCVC-family/EVC/checkpoints/download.py similarity index 100% rename from EVC/checkpoints/download.py rename to DCVC-family/EVC/checkpoints/download.py diff --git a/EVC/model_complexity.py b/DCVC-family/EVC/model_complexity.py similarity index 100% rename from EVC/model_complexity.py rename to DCVC-family/EVC/model_complexity.py diff --git a/EVC/requirements.txt b/DCVC-family/EVC/requirements.txt similarity index 100% rename from EVC/requirements.txt rename to DCVC-family/EVC/requirements.txt diff --git a/EVC/results/RD_numbers.py b/DCVC-family/EVC/results/RD_numbers.py similarity index 100% rename from EVC/results/RD_numbers.py rename to DCVC-family/EVC/results/RD_numbers.py diff --git a/EVC/run_test_image.py b/DCVC-family/EVC/run_test_image.py similarity index 100% rename from EVC/run_test_image.py rename to DCVC-family/EVC/run_test_image.py diff --git a/EVC/src/cpp/3rdparty/CMakeLists.txt b/DCVC-family/EVC/src/cpp/3rdparty/CMakeLists.txt similarity index 100% rename from EVC/src/cpp/3rdparty/CMakeLists.txt rename to DCVC-family/EVC/src/cpp/3rdparty/CMakeLists.txt diff --git a/EVC/src/cpp/3rdparty/pybind11/CMakeLists.txt b/DCVC-family/EVC/src/cpp/3rdparty/pybind11/CMakeLists.txt similarity index 100% rename from EVC/src/cpp/3rdparty/pybind11/CMakeLists.txt rename to DCVC-family/EVC/src/cpp/3rdparty/pybind11/CMakeLists.txt diff --git a/EVC/src/cpp/3rdparty/pybind11/CMakeLists.txt.in b/DCVC-family/EVC/src/cpp/3rdparty/pybind11/CMakeLists.txt.in similarity index 100% rename from EVC/src/cpp/3rdparty/pybind11/CMakeLists.txt.in rename to DCVC-family/EVC/src/cpp/3rdparty/pybind11/CMakeLists.txt.in diff --git a/EVC/src/cpp/3rdparty/ryg_rans/CMakeLists.txt b/DCVC-family/EVC/src/cpp/3rdparty/ryg_rans/CMakeLists.txt similarity index 100% rename from EVC/src/cpp/3rdparty/ryg_rans/CMakeLists.txt rename to DCVC-family/EVC/src/cpp/3rdparty/ryg_rans/CMakeLists.txt diff --git a/EVC/src/cpp/3rdparty/ryg_rans/CMakeLists.txt.in b/DCVC-family/EVC/src/cpp/3rdparty/ryg_rans/CMakeLists.txt.in similarity index 100% rename from EVC/src/cpp/3rdparty/ryg_rans/CMakeLists.txt.in rename to DCVC-family/EVC/src/cpp/3rdparty/ryg_rans/CMakeLists.txt.in diff --git a/EVC/src/cpp/CMakeLists.txt b/DCVC-family/EVC/src/cpp/CMakeLists.txt similarity index 100% rename from EVC/src/cpp/CMakeLists.txt rename to DCVC-family/EVC/src/cpp/CMakeLists.txt diff --git a/EVC/src/cpp/ops/CMakeLists.txt b/DCVC-family/EVC/src/cpp/ops/CMakeLists.txt similarity index 100% rename from EVC/src/cpp/ops/CMakeLists.txt rename to DCVC-family/EVC/src/cpp/ops/CMakeLists.txt diff --git a/EVC/src/cpp/ops/ops.cpp b/DCVC-family/EVC/src/cpp/ops/ops.cpp similarity index 100% rename from EVC/src/cpp/ops/ops.cpp rename to DCVC-family/EVC/src/cpp/ops/ops.cpp diff --git a/EVC/src/cpp/py_rans/CMakeLists.txt b/DCVC-family/EVC/src/cpp/py_rans/CMakeLists.txt similarity index 100% rename from EVC/src/cpp/py_rans/CMakeLists.txt rename to DCVC-family/EVC/src/cpp/py_rans/CMakeLists.txt diff --git a/EVC/src/cpp/py_rans/py_rans.cpp b/DCVC-family/EVC/src/cpp/py_rans/py_rans.cpp similarity index 100% rename from EVC/src/cpp/py_rans/py_rans.cpp rename to DCVC-family/EVC/src/cpp/py_rans/py_rans.cpp diff --git a/EVC/src/cpp/py_rans/py_rans.h b/DCVC-family/EVC/src/cpp/py_rans/py_rans.h similarity index 100% rename from EVC/src/cpp/py_rans/py_rans.h rename to DCVC-family/EVC/src/cpp/py_rans/py_rans.h diff --git a/EVC/src/cpp/rans/CMakeLists.txt b/DCVC-family/EVC/src/cpp/rans/CMakeLists.txt similarity index 100% rename from EVC/src/cpp/rans/CMakeLists.txt rename to DCVC-family/EVC/src/cpp/rans/CMakeLists.txt diff --git a/EVC/src/cpp/rans/rans.cpp b/DCVC-family/EVC/src/cpp/rans/rans.cpp similarity index 100% rename from EVC/src/cpp/rans/rans.cpp rename to DCVC-family/EVC/src/cpp/rans/rans.cpp diff --git a/EVC/src/cpp/rans/rans.h b/DCVC-family/EVC/src/cpp/rans/rans.h similarity index 100% rename from EVC/src/cpp/rans/rans.h rename to DCVC-family/EVC/src/cpp/rans/rans.h diff --git a/EVC/src/models/__init__.py b/DCVC-family/EVC/src/models/__init__.py similarity index 100% rename from EVC/src/models/__init__.py rename to DCVC-family/EVC/src/models/__init__.py diff --git a/EVC/src/models/common_model.py b/DCVC-family/EVC/src/models/common_model.py similarity index 100% rename from EVC/src/models/common_model.py rename to DCVC-family/EVC/src/models/common_model.py diff --git a/EVC/src/models/entropy_models.py b/DCVC-family/EVC/src/models/entropy_models.py similarity index 100% rename from EVC/src/models/entropy_models.py rename to DCVC-family/EVC/src/models/entropy_models.py diff --git a/EVC/src/models/hyperprior.py b/DCVC-family/EVC/src/models/hyperprior.py similarity index 100% rename from EVC/src/models/hyperprior.py rename to DCVC-family/EVC/src/models/hyperprior.py diff --git a/EVC/src/models/image_model.py b/DCVC-family/EVC/src/models/image_model.py similarity index 100% rename from EVC/src/models/image_model.py rename to DCVC-family/EVC/src/models/image_model.py diff --git a/EVC/src/models/layers.py b/DCVC-family/EVC/src/models/layers.py similarity index 100% rename from EVC/src/models/layers.py rename to DCVC-family/EVC/src/models/layers.py diff --git a/EVC/src/models/scalable_encoder_model.py b/DCVC-family/EVC/src/models/scalable_encoder_model.py similarity index 100% rename from EVC/src/models/scalable_encoder_model.py rename to DCVC-family/EVC/src/models/scalable_encoder_model.py diff --git a/EVC/src/utils/common.py b/DCVC-family/EVC/src/utils/common.py similarity index 100% rename from EVC/src/utils/common.py rename to DCVC-family/EVC/src/utils/common.py diff --git a/EVC/src/utils/png_reader.py b/DCVC-family/EVC/src/utils/png_reader.py similarity index 100% rename from EVC/src/utils/png_reader.py rename to DCVC-family/EVC/src/utils/png_reader.py diff --git a/EVC/src/utils/stream_helper.py b/DCVC-family/EVC/src/utils/stream_helper.py similarity index 100% rename from EVC/src/utils/stream_helper.py rename to DCVC-family/EVC/src/utils/stream_helper.py diff --git a/EVC/test_cfg/local_kodak.json b/DCVC-family/EVC/test_cfg/local_kodak.json similarity index 100% rename from EVC/test_cfg/local_kodak.json rename to DCVC-family/EVC/test_cfg/local_kodak.json diff --git a/EVC/test_image.py b/DCVC-family/EVC/test_image.py similarity index 100% rename from EVC/test_image.py rename to DCVC-family/EVC/test_image.py diff --git a/DCVC-family/README.md b/DCVC-family/README.md new file mode 100644 index 0000000..54169df --- /dev/null +++ b/DCVC-family/README.md @@ -0,0 +1,68 @@ + +# DCVC-family + + +:rocket: The DCVC-family offers a series of neural image and video codecs. This document provides links to the code, papers, and checkpoints for each codec. + +| Model | Link | Checkpoint | +| ----- | ---- | ---------- | +| DCVC | [Code](DCVC) & [Paper (NeurIPS 2021)](https://proceedings.neurips.cc/paper/2021/file/96b250a90d3cf0868c83f8c965142d2a-Paper.pdf) & [Paper (arXiv)](https://arxiv.org/abs/2109.15047)| [Checkpoints](https://1drv.ms/u/s!AozfVVwtWWYoiS5mcGX320bFXI0k?e=iMeykH) | +| DCVC-TCM | [Code](DCVC-TCM) & [Paper (IEEE TMM)](https://ieeexplore.ieee.org/document/9941493) & [Paper (arXiv)](https://arxiv.org/abs/2111.13850)| [Checkpoints](https://onedrive.live.com/?authkey=%21ADwwaonwTGR%5FNR8&id=2866592D5C55DF8C%211234&cid=2866592D5C55DF8C) | +| DCVC-HEM | [Code](DCVC-HEM) & [Paper (ACM MM 2022)](https://dl.acm.org/doi/abs/10.1145/3503161.3547845) & [Paper (arXiv)](https://arxiv.org/abs/2207.05894)| [Checkpoints](https://1drv.ms/u/s!AozfVVwtWWYoiUAGk6xr-oELbodn?e=kry2Nk) | +| DCVC-DC | [Code](DCVC-DC) & [Paper (CVPR 2023)](https://openaccess.thecvf.com/content/CVPR2023/papers/Li_Neural_Video_Compression_With_Diverse_Contexts_CVPR_2023_paper.pdf) & [Paper (arXiv)](https://arxiv.org/abs/2302.14402)| [Checkpoints](https://1drv.ms/u/s!AozfVVwtWWYoiWdwDhEkZMIfpon5?e=JcGri5) | +| DCVC-FM | [Code](DCVC-FM) & [Paper (CVPR 2024)](https://openaccess.thecvf.com/content/CVPR2024/papers/Li_Neural_Video_Compression_with_Feature_Modulation_CVPR_2024_paper.pdf) & [Paper (arXiv)](https://arxiv.org/abs/2402.17414)| [Checkpoints](https://1drv.ms/f/s!AozfVVwtWWYoi1QkAhlIE-7aAaKV?e=OoemTr) | +| EVC | [Code](EVC) & [Paper (ICLR 2023)](https://openreview.net/forum?id=XUxad2Gj40n) & [Paper (arXiv)](https://arxiv.org/abs/2302.05071)| [Checkpoints](https://1drv.ms/u/s!AozfVVwtWWYoiUhZLZDx7vJjHK1C?e=qETpA1) | + +* As a backup, all the pretrained models could be found [here](https://1drv.ms/f/c/2866592d5c55df8c/EozfVVwtWWYggCitBAAAAAABbT4z2Z10fMXISnan72UtSA?e=BID7DA). + +:bouquet: The implementation of DCVC-family is based on [CompressAI](https://github.com/InterDigitalInc/CompressAI) and [PyTorchVideoCompression](https://github.com/ZhihaoHu/PyTorchVideoCompression). + +:page_facing_up: If you find this work useful for your research, please cite: + +``` +@article{li2021deep, + title={Deep Contextual Video Compression}, + author={Li, Jiahao and Li, Bin and Lu, Yan}, + journal={Advances in Neural Information Processing Systems}, + volume={34}, + year={2021} +} + +@article{sheng2022temporal, + title={Temporal context mining for learned video compression}, + author={Sheng, Xihua and Li, Jiahao and Li, Bin and Li, Li and Liu, Dong and Lu, Yan}, + journal={IEEE Transactions on Multimedia}, + year={2022}, + publisher={IEEE} +} + +@inproceedings{li2022hybrid, + title={Hybrid Spatial-Temporal Entropy Modelling for Neural Video Compression}, + author={Li, Jiahao and Li, Bin and Lu, Yan}, + booktitle={Proceedings of the 30th ACM International Conference on Multimedia}, + year={2022} +} + +@inproceedings{li2023neural, + title={Neural Video Compression with Diverse Contexts}, + author={Li, Jiahao and Li, Bin and Lu, Yan}, + booktitle={{IEEE/CVF} Conference on Computer Vision and Pattern Recognition, + {CVPR} 2023, Vancouver, Canada, June 18-22, 2023}, + year={2023} +} + +@inproceedings{li2024neural, + title={Neural Video Compression with Feature Modulation}, + author={Li, Jiahao and Li, Bin and Lu, Yan}, + booktitle={{IEEE/CVF} Conference on Computer Vision and Pattern Recognition, + {CVPR} 2024, Seattle, WA, USA, June 17-21, 2024}, + year={2024} +} + +@inproceedings{wang2023EVC, + title={EVC: Towards Real-Time Neural Image Compression with Mask Decay}, + author={Wang, Guo-Hua and Li, Jiahao and Li, Bin and Lu, Yan}, + booktitle={International Conference on Learning Representations}, + year={2023} +} +``` \ No newline at end of file diff --git a/README.md b/README.md index 72a927c..9c4e0e6 100644 --- a/README.md +++ b/README.md @@ -1,28 +1,169 @@ -# Introduction - -Official Pytorch implementation for Neural Video and Image Compression including: -* Neural Video Codec - * DCVC: [Deep Contextual Video Compression](https://proceedings.neurips.cc/paper/2021/file/96b250a90d3cf0868c83f8c965142d2a-Paper.pdf), NeurIPS 2021, in [this folder](./DCVC/). - * DCVC-TCM: [**T**emporal **C**ontext **M**ining for Learned Video Compression](https://ieeexplore.ieee.org/document/9941493), in IEEE Transactions on Multimedia, and [arxiv](https://arxiv.org/abs/2111.13850), in [this folder](./DCVC-TCM/). - * DCVC-HEM: [**H**ybrid Spatial-Temporal **E**ntropy **M**odelling for Neural Video Compression](https://arxiv.org/abs/2207.05894), ACM MM 2022, in [this folder](./DCVC-HEM/). - - The first end-to-end neural video codec to exceed H.266 (VTM) using the highest compression ratio configuration, in terms of both PSNR and MS-SSIM. - - The first end-to-end neural video codec to achieve rate adjustment in single model. - * DCVC-DC: [Neural Video Compression with **D**iverse **C**ontexts](https://arxiv.org/abs/2302.14402), CVPR 2023, in [this folder](./DCVC-DC/). - - The first end-to-end neural video codec to exceed [ECM](https://jvet-experts.org/doc_end_user/documents/27_Teleconference/wg11/JVET-AA0006-v1.zip) using the highest compression ratio low delay configuration with a intra refresh period roughly to one second (32 frames), in terms of PSNR and MS-SSIM for RGB content. - - The first end-to-end neural video codec to exceed ECM using the highest compression ratio low delay configuration with a intra refresh period roughly to one second (32 frames), in terms of PSNR for YUV420 content. - * DCVC-FM: [Neural Video Compression with **F**eature **M**odulation](https://arxiv.org/abs/2402.17414), CVPR 2024, in [this folder](./DCVC-FM/). - - The first end-to-end neural video codec to exceed ECM using the highest compression ratio low delay configuration with only one intra frame, in terms of PSNR for both YUV420 content and RGB content in a single model. - - The first end-to-end neural video codec that support a large quality and bitrate range in a single model. - * DCVC-RT: [Towards Practical **R**eal-**T**ime Neural Video Compression](https://arxiv.org/abs/2502.20762), CVPR 2025, in [this folder](./DCVC-RT/). - - The first end-to-end neural video codec achieving 100+ fps encoding/decoding for 1080p on Nvidia A100 GPU. The overall compression ratio is comparable with DCVC-FM. -* Neural Image Codec - * [EVC: Towards Real-Time Neural Image Compression with Mask Decay](https://openreview.net/forum?id=XUxad2Gj40n), ICLR 2023, in [this folder](./EVC/). - -# Pretrained models - -As a backup, all the pretrained models could be found [here](https://1drv.ms/f/c/2866592d5c55df8c/EozfVVwtWWYggCitBAAAAAABbT4z2Z10fMXISnan72UtSA?e=BID7DA). - -# On the comparison +
+ +# DCVC-RT: Towards Practical Real-Time Neural Video Compression + +**CVPR2025** + +
+ +[![arXiv](https://img.shields.io/badge/arXiv-2502.20762-b31b1b.svg)](https://arxiv.org/abs/2502.20762) + + + + +** +The first neural video codec achieving 100+ FPS 1080p coding and 4K real-time coding with a comparable compression ratio with ECM. +** + + +## :book: Overview + +Welcome to the official implementation of DCVC-RT and the broader [DCVC-family](#clipboard-dcvc-family) models. The DCVC family is designed to push the boundaries of high-performance practical neural video codecs, delivering cutting-edge compression efficiency, real-time capabilities, and versatile functionalities. + +:rocket: In this section, we provide a brief overview of DCVC-RT. For an in-depth understanding, we encourage you to read our [paper](https://arxiv.org/abs/2502.20762). + +:hammer: Ready to get started? Head over to the [Usage](#hammer-usage) to start using this repo. + +:page_facing_up: If you find our work helpful, feel free to [cite](#page_facing_up-citation) us. We truly appreciate your support. + +:bouquet: The implementation of DCVC-RT is based on [CompressAI](https://github.com/InterDigitalInc/CompressAI). + +### Abstract + +We introduce a practical real-time neural video codec (NVC) designed to deliver high compression ratio, low latency and broad versatility. In practice, the coding speed of NVCs depends on 1) computational costs, and 2) non-computational operational costs, such as memory I/O and the number of function calls. While most efficient NVCs prioritize reducing computational cost, we identify operational cost as the primary bottleneck to achieving higher coding speed. Leveraging this insight, we introduce a set of efficiency-driven design improvements focused on minimizing operational costs. Specifically, we employ implicit temporal modeling to eliminate complex explicit motion modules, and use single low-resolution latent representations rather than progressive downsampling. These innovations significantly accelerate NVC without sacrificing compression quality. Additionally, we implement model integerization for consistent cross-device coding and a module-bank-based rate control scheme to improve practical adaptability. Experiments show our proposed DCVC-RT achieves an impressive average encoding/decoding speed at 125.2/112.8 fps (frames per second) for 1080p video, while saving an average of 21\% in bitrate compared to H.266/VTM. + + +### Performance +Bit saving over VTM-17.0 (UVG all frames with single intra-frame setting (i.e. intra-period = –1) and YUV420 colorspace.) + + + +The BD-Rate and 1080p encoding/decoding speed on NVIDIA A100 GPU + + + + +The complexity analysis and encoding/decoding speed evaluation across various resolutions and devices. + + + + +## :hammer: Usage + +For each step, click it to expand and view details. + +
+ Prerequisites
+ +* Python 3.12 and conda, get [Conda](https://www.anaconda.com/) +* CUDA 12.6 (other versions may also work. Make sure the CUDA version matches with pytorch.) +* pytorch (We have tested that pytorch-2.6 works. Other versions may also work.) +* Environment + ``` + conda create -n $YOUR_PY_ENV_NAME python=3.12 + conda activate $YOUR_PY_ENV_NAME + + pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126 + pip install -r requirements.txt + ``` + +
+ +
+ + Build the project
+ +Please build the C++ code to support bitstream writing and customized CUDA kernels to fuse operations. + +```bash +sudo apt-get install cmake g++ ninja-build +conda activate $YOUR_PY_ENV_NAME +cd ./src/cpp/ +pip install . +cd ../layers/extensions/inference/ +pip install . +``` + +
+ +
+ + CPU performance scaling
+ +Note that the arithmetic coding runs on the CPU, please make sure your CPU runs at high performance while writing the actual bitstream. Otherwise, the arithmetic coding may take a long time. + +Check the CPU frequency by +``` +grep -E '^model name|^cpu MHz' /proc/cpuinfo +``` + +Run the following command to maximum CPU frequency +``` +echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +``` + +Run the following command to recover the default frequency +``` +echo ondemand | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +``` + +
+ +
+ + Pretrained models
+ +* Download [our pretrained models](https://1drv.ms/f/c/2866592d5c55df8c/Esu0KJ-I2kxCjEP565ARx_YB88i0UnR6XnODqFcvZs4LcA?e=by8CO8) and put them into ./checkpoints folder. +* There are 2 models, one for image coding and the other for video coding. +* As a backup, all the pretrained models could be found [here](https://1drv.ms/f/c/2866592d5c55df8c/EozfVVwtWWYggCitBAAAAAABbT4z2Z10fMXISnan72UtSA?e=BID7DA). + + +
+ +
+ + Test dataset
+ +We support arbitrary original resolution. The input video resolution will be padded automatically. The reconstructed video will be cropped back to the original size. The distortion (PSNR) is calculated at original resolution. + +#### YUV 420 content + +Put *.yuv in the folder structure similar to the following structure. + + /media/data/HEVC_B/ + - BQTerrace_1920x1080_60.yuv + - BasketballDrive_1920x1080_50.yuv + - ... + /media/data/HEVC_D/ + /media/data/HEVC_C/ + ... + +The dataset structure can be seen in dataset_config_example_yuv420.json. + +#### RGB content + +We highly suggest testing YUV420 content. To test RGB content, please refer to the [DCVC-FM](DCVC-family/DCVC-FM) folder. + +
+ +
+ + Test the models
+ +Example to test pretrained model with four rate points: +```bash + python test_video.py --model_path_i ./checkpoints/cvpr2025_image.pth.tar --model_path_p ./checkpoints/cvpr2025_video.pth.tar --rate_num 4 --test_config ./dataset_config_example_yuv420.json --cuda 1 -w 1 --write_stream 1 --force_zero_thres 0.12 --output_path output.json --force_intra_period -1 --reset_interval 64 --force_frame_num -1 --check_existing 0 +``` + +It is recommended that the ```-w``` number is equal to your GPU number. + +You can also specify different ```--rate_num``` values (2~64) to test finer bitrate adjustment. + +
+ +
+ + On the comparison
Please note that different methods may use different configurations to test different models, such as * Source video may be different, e.g., cropped or padded to the desired resolution. @@ -33,52 +174,18 @@ So, it does not make sense to compare the numbers in different methods directly, Please find more details on the [test conditions](./test_conditions.md). -# Acknowledgement -The implementation is based on [CompressAI](https://github.com/InterDigitalInc/CompressAI) and [PyTorchVideoCompression](https://github.com/ZhihaoHu/PyTorchVideoCompression). +
-# Citation -If you find this work useful for your research, please cite: - -``` -@article{li2021deep, - title={Deep Contextual Video Compression}, - author={Li, Jiahao and Li, Bin and Lu, Yan}, - journal={Advances in Neural Information Processing Systems}, - volume={34}, - year={2021} -} -@article{sheng2022temporal, - title={Temporal context mining for learned video compression}, - author={Sheng, Xihua and Li, Jiahao and Li, Bin and Li, Li and Liu, Dong and Lu, Yan}, - journal={IEEE Transactions on Multimedia}, - year={2022}, - publisher={IEEE} -} +## :clipboard: DCVC-family -@inproceedings{li2022hybrid, - title={Hybrid Spatial-Temporal Entropy Modelling for Neural Video Compression}, - author={Li, Jiahao and Li, Bin and Lu, Yan}, - booktitle={Proceedings of the 30th ACM International Conference on Multimedia}, - year={2022} -} +The details of DCVC family models can be found in [DCVC-family](DCVC-family/README.md). -@inproceedings{li2023neural, - title={Neural Video Compression with Diverse Contexts}, - author={Li, Jiahao and Li, Bin and Lu, Yan}, - booktitle={{IEEE/CVF} Conference on Computer Vision and Pattern Recognition, - {CVPR} 2023, Vancouver, Canada, June 18-22, 2023}, - year={2023} -} -@inproceedings{li2024neural, - title={Neural Video Compression with Feature Modulation}, - author={Li, Jiahao and Li, Bin and Lu, Yan}, - booktitle={{IEEE/CVF} Conference on Computer Vision and Pattern Recognition, - {CVPR} 2024, Seattle, WA, USA, June 17-21, 2024}, - year={2024} -} +## :page_facing_up: Citation +If you find this work useful for your research, please cite: +``` @inproceedings{jia2025towards, title={Towards Practical Real-Time Neural Video Compression}, author={Jia, Zhaoyang and Li, Bin and Li, Jiahao and Xie, Wenxuan and Qi, Linfeng and Li, Houqiang and Lu, Yan}, @@ -86,14 +193,7 @@ If you find this work useful for your research, please cite: {CVPR} 2025, Nashville, TN, USA, June 11-25, 2024}, year={2025} } - -@inproceedings{wang2023EVC, - title={EVC: Towards Real-Time Neural Image Compression with Mask Decay}, - author={Wang, Guo-Hua and Li, Jiahao and Li, Bin and Lu, Yan}, - booktitle={International Conference on Learning Representations}, - year={2023} -} ``` -# Trademarks +## Trademarks This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow [Microsoft’s Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general). Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party’s policies. diff --git a/DCVC-RT/assets/RD-Curve.png b/assets/RD-Curve.png similarity index 100% rename from DCVC-RT/assets/RD-Curve.png rename to assets/RD-Curve.png diff --git a/assets/bd_rate_speed.png b/assets/bd_rate_speed.png new file mode 100644 index 0000000..3773a1d Binary files /dev/null and b/assets/bd_rate_speed.png differ diff --git a/assets/complexity.png b/assets/complexity.png new file mode 100644 index 0000000..8bee325 Binary files /dev/null and b/assets/complexity.png differ diff --git a/assets/practical_performance.png b/assets/practical_performance.png new file mode 100644 index 0000000..7d77b9b Binary files /dev/null and b/assets/practical_performance.png differ diff --git a/DCVC-RT/dataset_config_example_yuv420.json b/dataset_config_example_yuv420.json similarity index 100% rename from DCVC-RT/dataset_config_example_yuv420.json rename to dataset_config_example_yuv420.json diff --git a/DCVC-RT/requirements.txt b/requirements.txt similarity index 100% rename from DCVC-RT/requirements.txt rename to requirements.txt diff --git a/DCVC-RT/src/cpp/py_rans/py_rans.cpp b/src/cpp/py_rans/py_rans.cpp similarity index 100% rename from DCVC-RT/src/cpp/py_rans/py_rans.cpp rename to src/cpp/py_rans/py_rans.cpp diff --git a/DCVC-RT/src/cpp/py_rans/py_rans.h b/src/cpp/py_rans/py_rans.h similarity index 100% rename from DCVC-RT/src/cpp/py_rans/py_rans.h rename to src/cpp/py_rans/py_rans.h diff --git a/DCVC-RT/src/cpp/py_rans/rans.cpp b/src/cpp/py_rans/rans.cpp similarity index 100% rename from DCVC-RT/src/cpp/py_rans/rans.cpp rename to src/cpp/py_rans/rans.cpp diff --git a/DCVC-RT/src/cpp/py_rans/rans.h b/src/cpp/py_rans/rans.h similarity index 100% rename from DCVC-RT/src/cpp/py_rans/rans.h rename to src/cpp/py_rans/rans.h diff --git a/DCVC-RT/src/cpp/py_rans/rans_byte.h b/src/cpp/py_rans/rans_byte.h similarity index 100% rename from DCVC-RT/src/cpp/py_rans/rans_byte.h rename to src/cpp/py_rans/rans_byte.h diff --git a/DCVC-RT/src/cpp/setup.py b/src/cpp/setup.py similarity index 100% rename from DCVC-RT/src/cpp/setup.py rename to src/cpp/setup.py diff --git a/DCVC-RT/src/layers/cuda_inference.py b/src/layers/cuda_inference.py similarity index 100% rename from DCVC-RT/src/layers/cuda_inference.py rename to src/layers/cuda_inference.py diff --git a/DCVC-RT/src/layers/extensions/inference/bind.cpp b/src/layers/extensions/inference/bind.cpp similarity index 100% rename from DCVC-RT/src/layers/extensions/inference/bind.cpp rename to src/layers/extensions/inference/bind.cpp diff --git a/DCVC-RT/src/layers/extensions/inference/common.h b/src/layers/extensions/inference/common.h similarity index 100% rename from DCVC-RT/src/layers/extensions/inference/common.h rename to src/layers/extensions/inference/common.h diff --git a/DCVC-RT/src/layers/extensions/inference/def.h b/src/layers/extensions/inference/def.h similarity index 100% rename from DCVC-RT/src/layers/extensions/inference/def.h rename to src/layers/extensions/inference/def.h diff --git a/DCVC-RT/src/layers/extensions/inference/impl.cpp b/src/layers/extensions/inference/impl.cpp similarity index 100% rename from DCVC-RT/src/layers/extensions/inference/impl.cpp rename to src/layers/extensions/inference/impl.cpp diff --git a/DCVC-RT/src/layers/extensions/inference/kernel.cu b/src/layers/extensions/inference/kernel.cu similarity index 100% rename from DCVC-RT/src/layers/extensions/inference/kernel.cu rename to src/layers/extensions/inference/kernel.cu diff --git a/DCVC-RT/src/layers/extensions/inference/setup.py b/src/layers/extensions/inference/setup.py similarity index 100% rename from DCVC-RT/src/layers/extensions/inference/setup.py rename to src/layers/extensions/inference/setup.py diff --git a/DCVC-RT/src/layers/layers.py b/src/layers/layers.py similarity index 100% rename from DCVC-RT/src/layers/layers.py rename to src/layers/layers.py diff --git a/DCVC-RT/src/models/common_model.py b/src/models/common_model.py similarity index 100% rename from DCVC-RT/src/models/common_model.py rename to src/models/common_model.py diff --git a/DCVC-RT/src/models/entropy_models.py b/src/models/entropy_models.py similarity index 100% rename from DCVC-RT/src/models/entropy_models.py rename to src/models/entropy_models.py diff --git a/DCVC-RT/src/models/image_model.py b/src/models/image_model.py similarity index 100% rename from DCVC-RT/src/models/image_model.py rename to src/models/image_model.py diff --git a/DCVC-RT/src/models/video_model.py b/src/models/video_model.py similarity index 100% rename from DCVC-RT/src/models/video_model.py rename to src/models/video_model.py diff --git a/DCVC-RT/src/utils/common.py b/src/utils/common.py similarity index 100% rename from DCVC-RT/src/utils/common.py rename to src/utils/common.py diff --git a/DCVC-RT/src/utils/metrics.py b/src/utils/metrics.py similarity index 100% rename from DCVC-RT/src/utils/metrics.py rename to src/utils/metrics.py diff --git a/DCVC-RT/src/utils/stream_helper.py b/src/utils/stream_helper.py similarity index 100% rename from DCVC-RT/src/utils/stream_helper.py rename to src/utils/stream_helper.py diff --git a/DCVC-RT/src/utils/transforms.py b/src/utils/transforms.py similarity index 100% rename from DCVC-RT/src/utils/transforms.py rename to src/utils/transforms.py diff --git a/DCVC-RT/src/utils/video_reader.py b/src/utils/video_reader.py similarity index 100% rename from DCVC-RT/src/utils/video_reader.py rename to src/utils/video_reader.py diff --git a/DCVC-RT/src/utils/video_writer.py b/src/utils/video_writer.py similarity index 100% rename from DCVC-RT/src/utils/video_writer.py rename to src/utils/video_writer.py diff --git a/DCVC-RT/test_video.py b/test_video.py similarity index 100% rename from DCVC-RT/test_video.py rename to test_video.py