test: add cppapi test to give TorchCodecConfig.cmake a try

[dvrogozh]

That's a test for the TorchCodecConfig.cmake we've introduced in dc87228. Test is integrated into TorchCodec pytest suite and marked with @pytest.mark.devel which is disabled by default. Test assumes that ffmpeg is installed and can be used for both runtime and compilation (i.e. .pc and include files are available). Test builds a .so which is linked to TorchCodec core library. There are 2 kind of tests executed:

1. Building with ffmpeg being found via pkg-config
2. Building with ffmpeg being pointed to by TORCHCODEC_FFMPEGn_INSTALL_PREFIX environment variable

Test which relies on environment variable assumes that the ffmpeg install layout will match expectations hardcoded in the test and in the TorchCodecConfig.cmake. It potentially might fail - we need to check in the CI.

@NicolasHug, please, help to review paying attention on the above. I hope this will allow us to have test naturally integrated into TorchCodec test suite. If not, let's discuss what we need to modify.

CC: @NicolasHug

[dvrogozh]

Thank you for triggering CI. Ok, we have the whole bunch of errors which are circling around the pattern that we need not only runtime, but development environment as well, i.e. we need pkg-config, CUDA dev. files and ffmpeg dev. files available. Depending on the workflow we get somewhat different errors:

1. linux_cuda_wheel.yaml - we are missing CUDA which causes failure to import torch target in cmake:

CMake Error at /__w/_temp/conda_environment_19876943476/lib/python3.10/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:90 (message):
  Your installed Caffe2 version uses CUDA but I cannot find the CUDA

2. linux_wheel.yaml and macos_wheel.yaml - we are missing ffmpeg dev. files (are they available with conda installed ffmpeg?)

CMake Error at /usr/local/share/cmake-3.31/Modules/FindPkgConfig.cmake:645 (message):
  The following required packages were not found:
   - libavdevice

3. windows_wheel.yaml - we are missing pkg-config

CMake Warning at C:/Miniconda3/envs/test/Lib/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:22 (message):
  static library kineto_LIBRARY-NOTFOUND not found.

Across these errors the most worrisome seems to be missing CUDA. @NicolasHug, is that reasonable to request having CUDA fully installed on the runner environment for these tests or we better change a strategy? For example, we can consider to avoid integrating this test into regular torchcodec pytest suite and run it separately within cpp_tests.yaml which is less demanding in terms of pre-installed environment?

[dvrogozh]

is that reasonable to request having CUDA fully installed on the runner environment

Actually, it's installed, but was not detected. I guess that due to OS differences in cmake behavior. On ubuntu it get's detected from /usr/local, on AlmaLinux it seems not:

$ docker run -it --rm pytorch/manylinux2_28-builder:cuda12.6 ls /usr/local/cuda/include
bootstrap_device_host        cudaTypedefs.h                 cusparseLt.h                              nv_decode.h
builtin_types.h              cudaVDPAU.h                    cusparse_v2.h                             nvFatbin.h

[NicolasHug]

Thanks for the PR @dvrogozh ! I have a few questions / comments:

• Do we want all this to work when users are building torchcodec from source, or do we want it to work when they install a pre-built version of torchcodec (e.g. via pip install torchcodec ...) ? I assume we want the latter?
• Would it be acceptable to only support Linux for now? I feel like Windows and MacOS support might be lower priority for the feature, and focusing on linux could make it easier.

we need not only runtime, but development environment as well, i.e. we need pkg-config, CUDA dev. files and ffmpeg dev. files available.

Right now, the newly added test are ran under this install-and-test job:

torchcodec/.github/workflows/linux_wheel.yaml

Line 60 in 38fa96c

install-and-test:

. The point of this job is to have a minimal environment that looks like what an end-user would have (typically without development dependencies!). If we need dev stuff, it's best to run the new test in a separate job. That is, a new job, similar in nature to install-and-test, that still depends on the build job above, but where we can install the required dev dependencies safely. This new job doesn't even need to rely on pytest, we could have some inline bash commands directly calling into cmake, which might be simpler too? Does that make sense?

[dvrogozh]

Do we want all this to work when users are building torchcodec from source, or do we want it to work when they install a pre-built version of torchcodec (e.g. via pip install torchcodec ...) ? I assume we want the latter?

Yes, ultimate goal is to support the latter - that's the intended scenario. This being said, it still should work if someone builds from source.

Would it be acceptable to only support Linux for now?

On XPU side we plan to support Windows for we are not their yet (with torchcodec). To be honest, I don't feel that right now we are at the point when focusing just on Linux will make things easier. I still believe we can make things just work across all 3 OSes fluently.

That is, a new job, similar in nature to install-and-test, that still depends on the build job above <...>

I think that connecting things to pytest will just make things for developers so much easier. Otherwise they will start seeing issues not on the point when they've prepared a code locally, but when CI completed after a push.

Creating a separate job won't actually much help if we want to give test a run on the CUDA environment because as of now install-and-test is executed under pytorch/manylinux2_28-builder:cudaX.Y container. I checked this container - it has CUDA installed. I think the issue is that it just was not found due to missing CUDA_HOME environment variable.

I still propose to try finish with the idea to integrate test into pytest. I believe current workflow .yml files just need 2 minor changes to make things running:

1. Need to install pkg-config from the conda environment
2. Need to set CUDA_HOME=/usr/local/cuda when running tests on pytorch with cuda

I've did these changes in the last update of the PR. @NicolasHug, don't you mind to trigger CI once again to see if it works now?

[dvrogozh]

@NicolasHug, I reworked the test trying to take into account your comments. Key points:

1. Added new tests marked with @pytest.mark.devel
2. Tests are skipped by default and require explicit -m devel on the command line
3. Added new test job only for linux/torch-cpu to give test a run

Please, help to review.

[NicolasHug]

Thanks @dvrogozh , this LGTM! I made some minor changes, mainly:

• used the norecursedirs feature of pytest instead of the addition of a new pytest mark, which I think is more convenient since we don't need to change all the pytest invocations in the other workflows
• renamed cppapi into ThirdPartyInterface. I wanted to avoid the perception that we support a C++ API for the decoder (we don't!)

Nothing more to do on your side, I'll make sure the PR is green and then I'll merge it. I just had a quick question for you below in the comment.

Thanks!

[NicolasHug]

@dvrogozh I'm not 100% sure what you meant by this comment, could you help me understand?

[dvrogozh]

User may point to the location of one or few FFmpeg versions with TORCHCODEC_FFMPEG{ver}_INSTALL_PREFIX. The TorchCodec assumes certain hardcode layout of the these FFmpeg versions (how include, libdir, etc. folders are named). If user has custom layout not aligned with our expectations, then test might fail as it will fail to find certain ffmpeg targets we are using here.

[dvrogozh]

That works for me. Thank you @NicolasHug for the change.