Question about how to compile dependencies with conan + shared=True

[davidsanfal]

Hi @tttapa!

I've been testing your backend implementation based on the conan branch of py-build-cmake-example. In my case, I added the fmt dependency to try building a package with some dependencies. Honestly, I found it very easy to understand and use.

The problem came up when I tried to compile fmt with the shared=True option.

[tool.py-build-cmake.conan]
args = ["--build=missing", "-o fmt/*:shared=True", "-o fmt/*:header_only=False"]

In this case, the wheel seems to build successfully.

pip install .               
Processing /Users/davidsanfal/repos/pep517/pep517-conan/example
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: py-conan-add
  Building wheel for py-conan-add (pyproject.toml) ... done
  Created wheel for py-conan-add: filename=py_conan_add-0.6.0a2.dev0-cp313-cp313-macosx_10_13_x86_64.whl size=59290 sha256=5b248b4e10c758fda42a9fc09a6c07427dea787441840c7a1d826429969e050f
  Stored in directory: /private/var/folders/x8/4lczj5ps1hzdb_3_f7h70wlm0000gn/T/pip-ephem-wheel-cache-qfudjm05/wheels/e9/2a/fb/a19497391fa407f6efa60fc550acb8f6bfd4bc5a71b2cca970
Successfully built py-conan-add
Installing collected packages: py-conan-add
  Attempting uninstall: py-conan-add
    Found existing installation: py-conan-add 0.6.0a2.dev0
    Uninstalling py-conan-add-0.6.0a2.dev0:
      Successfully uninstalled py-conan-add-0.6.0a2.dev0
Successfully installed py-conan-add-0.6.0a2.dev0

but it hasn't actually copied the fmt library into it, so it fails when using the package.

pytest                      
=================================================================================================================== test session starts ====================================================================================================================
platform darwin -- Python 3.13.4, pytest-8.4.2, pluggy-1.6.0
rootdir: /Users/davidsanfal/pep517-conan/example
configfile: pyproject.toml
collected 0 items / 1 error                                                                                                                                                                                                                                

========================================================================================================================== ERRORS ==========================================================================================================================
_________________________________________________________________________________________________________ ERROR collecting test/test_add_module.py _________________________________________________________________________________________________________
ImportError while importing test module '/Users/davidsanfal/pep517-conan/example/test/test_add_module.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/importlib/__init__.py:88: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
test/test_add_module.py:2: in <module>
    from py_conan_add.add_module import add
../../venv/lib/python3.13/site-packages/py_conan_add/add_module.py:6: in <module>
    from py_conan_add._add_module import *
E   ImportError: dlopen(/Users/davidsanfal/venv/lib/python3.13/site-packages/py_conan_add/_add_module.cpython-313-darwin.so, 0x0002): Library not loaded: @rpath/libfmt.12.dylib
E     Referenced from: <D376EC0A-7EFC-3A7D-B59B-B99640E1D02C> /Users/davidsanfal/venv/lib/python3.13/site-packages/py_conan_add/_add_module.cpython-313-darwin.so
E     Reason: no LC_RPATH's found
================================================================================================================= short test summary info ==================================================================================================================
ERROR test/test_add_module.py

I guess there's some step I'm missing or some setting I'm adding incorrectly. I have created a test repository with the example I am using in case it can help to reproduce my issue. Thanks!

[tttapa]

Hi, thanks for reaching out!

At the moment, py-build-cmake does not attempt to automatically include any dependencies in the Wheel. There are multiple possible approaches to add such a feature to py-build-cmake:

• Manual install(...) rules in the user's CMake scripts (e.g. using RUNTIME_DEPENDENCY_SET or BundleUtilities).
  Pro: Full control by the user, possibility to patch up RUNPATH/@loader_path.
  Con: Cumbersome, requires some (platform-specific) boilerplate in each project.
• Using a Conan deployer (e.g. runtime_deploy).
  Pro: Less boilerplate for the user.
  Con: Experimental feature. Limited user control, bound by Conan's behavior (which could change in the future).
• Using a custom Conan deployer.
  Pro: Full control, full access to dependency graph.
  Con: Possible maintenance burden. Unclear how to best allow users to exclude certain libraries (that may be in Conan's graph, but shouldn't end up in the wheel).
• External tool (such as auditwheel/delocate-wheel).
  Pro: Possible to re-use standard Python tools, consistency with other build systems.
  Con: Requires an extra build step, which requires access to Conan's runtime paths to correctly locate the libraries.

It is not clear to me what the best approach would be. I intend to experiment with some of these ideas at some point, but I've been quite busy with other things, unfortunately.
If you have any comments or ideas, let me know!

[davidsanfal]

Hello @tttapa ! I have been working on a solution for this issue based on your comments. They have been very helpful, thanks!

I'm working on a branch in my fork. For now, it's just a POC: add a profile detect if it's your first time using conan or a post_wheel to manipulate it if necessary.

The post_wheel is now inside the conan class, but I would like to move it to a separate configuration where you define the path to a .py file with your hook (or even have some built-in hook).
The idea is to have general purpose prehooks or posthooks where you can do some actions (for example, delocalte on mac) before and after building, but before installing the packages.

This is the branch I'm working on. As soon as I have something more definite, I'll let you know.

[tttapa]

Thanks a lot!

In the mean time, I've looked at auditwheel (the Linux counterpart of delocate-wheel) in a bit more detail, and a major limitation seems to be that it does not support cross-compilation: pypa/auditwheel#244 (comment)
Cross-compilation is one of the core features of py-build-cmake (to ensure maximum portability and reproducibility), so this may be a deal breaker.

That said, I think the approach with a Python hook for deploying libraries or delocating the wheel is a good idea. That still leaves the option of directly performing a runtime_deploy to the wheel folder.