SwinCS-VFIT

Video Frame Interpolation Transformer based on Shifted-window and Cross-Scale window.

This repo is the implementation of SwinCS-VFIT. Paper

Packages

The following pakages are required to run the code:

• python==3.7.6
• pytorch==1.5.1
• cudatoolkit==10.1
• torchvision==0.6.1
• cupy==7.5.0
• pillow==8.2.0
• einops==0.3.0

Train

• Download the Vimeo-90K septuplets dataset.

• Then train SwinCS-VFIT-B using default training configurations:

python main.py --model SwinCS_VFIT_B --dataset vimeo90K_septuplet --data_root <dataset_path>

Training SwinCS-VFIT-S is similiar to above, just change model to SwinCS_VFIT_S.

Test

After training, you can evaluate the model with following command:

python test.py --model SwinCS_VFIT_B --dataset vimeo90K_septuplet --data_root <dataset_path> --load_from checkpoints/SwinCS_VFIT_B/model_best.pth

You can also evaluate SwinCS-VFIT using our weight here.

More datasets for evaluation:

• UCF
• Davis

Interpolate

Folders structure like this:

.
├── inter_data
│   ├── folder1
│   │   ├── im1.jpg
│   │   ├── ...
│   │   └── im7.jpg
│   └── ...
├── out_data
└── SwinCS-VFIT
    ├── interpolate_demo1.py
    └── interpolate_demo2.py

Run interpolation with SwinCS-VFIT-B:

python interpolate_demo1.py --model SwinCS_VFIT_B --load_from checkpoints/SwinCS_VFIT_B/model_best.pth

Or specify your own input and output dir:

python interpolate_demo1.py --model SwinCS_VFIT_B --load_from checkpoints/SwinCS_VFIT_B/model_best.pth --img_path <path/to/inter_data> --out_path <path/to/out_data>

Interpolating frame with SwinCS-VFIT-S is similiar.

References

Some other great video interpolation resources that we benefit from:

• VFIT: Video Frame Interpolation Transformer, CVPR 2022 Code
• VFIformer: Video Frame Interpolation with Transformer, CVPR 2022 Code
• FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation, arXiv 2021 Code
• QVI: Quadratic Video Interpolation, NeurIPS 2019 Code
• AdaCoF: Adaptive Collaboration of Flows for Video Frame Interpolation, CVPR 2020 Code

Thanks a lot!