Extension module can't find shared library from same Meson project

[sampotter]

I develop a numerical library in C called butterfly which is built using Meson and have more recently been using meson-python to build a thin wrapper for it written in Cython. By default, the C library is built as a shared library and the Python module just consumes this shared library as a dependency in the natural way in Meson. This has worked fine for testing when have I run scripts etc "locally" from builddir in the hackiest way possible.

I've just recently tried running meson install to see whether import butterfly will work. It doesn't. When I run meson install I see something like:

Installing subdir /home/sfp/Dropbox/Research/Butterfly/butterfly/include/bf to /usr/local/include/bf
Installing /home/sfp/Dropbox/Research/Butterfly/butterfly/include/bf/backends.h to /usr/local/include/bf
...
... many irrelevant lines ...
...
Installing /home/sfp/Dropbox/Research/Butterfly/butterfly/include/bf/vectors.h to /usr/local/include/bf
Installing libbutterfly.so to /usr/local/lib64
Installing wrappers/python/butterfly.cpython-311-x86_64-linux-gnu.so to /usr/local/lib64/python3.11/site-packages/
Installing /home/sfp/Dropbox/Research/Butterfly/butterfly/builddir/meson-private/butterfly.pc to /usr/local/lib64/pkgconfig

So, the shared library and Python module seem to get put somewhere reasonable. But, when I fire up Python and try to import the shared library, it can't be found:

[sfp@fedora ~]$ python
Python 3.11.7 (main, Dec 18 2023, 00:00:00) [GCC 13.2.1 20231011 (Red Hat 13.2.1-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import butterfly
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: libbutterfly.so: cannot open shared object file: No such file or directory

Any help would be appreciated.

I should say that I'm very green when it comes to packaging. I would like to make this work but it's not clear to me what tool in this complicated process is even responsible.

Thanks!

[rgommers]

Hi @sampotter, thanks for the question. This has come up multiple times before, so I guess it's really time we write some docs that address this use case. (TODO: https://mesonbuild.com/meson-python/explanations/internal-dependencies.html)

First a few general pointers:

• You say you are running meson install, which means you are not using meson-python at all I think. meson-python is a "build backend" in Python packaging terminology, which means it gets used under the hood when building an sdist or wheel via commands like python -m build or pip install . (locally) or pip install butterfly (when butterfly is on PyPI)
  • If you avoid meson-python and indeed install with meson setup/compile/install, then you should be able to make things work mostly, you just won't get Python packaging metadata and you cannot install with pip & co.
• shared libraries are difficult for Python packaging; in particular it's important to realize that (a) pip/build are still able to install things into <prefix>/lib/ but that that is strongly discouraged, and (b) project-external shared libraries are not handled by default at all, it needs an external tool like auditwheel to bundle them into a wheel.
  • For project-internal libraries things should work through meson-python though, except for the documented limitation on Windows (currently): https://mesonbuild.com/meson-python/reference/limitations.html#shared-libraries-on-windows
  • Also import: there is no fixed relationship in directory structure between <prefix>/lib/ and Python's site-packages, so if you install via a wheel it's difficult to keep an RPATH working

To say more, it'd be useful to know what exactly you're aiming for here:

1. Installing with meson install only, and making your Python wrapper work that way
2. Using meson-python to produce wheels including the shared library
3. Using meson-python to produce wheels excluding the shared library, but only the Cython wrapper and relying on the user or a Linux distro to install that shared library separately.

And if (2) or (3): do you need it to work on Windows? And do you intend to distribute via PyPI, or only for users cloning your repo?

[sampotter]

Hi @rgommers. Thanks for the speedy and very educational response. I truly appreciate it! I did see the linked stub in the documentation and assumed it would be pertinent.

Re: your first general pointer: you can see the depth of my misunderstanding. :-) I had assumed that all Python-related functionality was captured by meson-python, but it's clear now that .extension_module etc are provided by Meson's Python module. Are there long term plans for meson-python to merge into Meson proper?

And thanks for the explanation regarding shared libraries. I have seen this come up enough times in other contexts that I figured it would be kind of nasty to resolve. If I could trouble you: do you have any recommendations for something to read about RPATHs? I'm not very familiar with them or how they work and they seem central to this business.

To explain what I'm after:

1. The core C library is where all important functionality lives. Since it is a numerical library, the long-term goal is to provide thin wrappers for the main scientific languages (Python, Julia, MATLAB, etc... even C++).
2. I would like to make it so that whoever is using it can install it in the way that seems easiest and most convenient for them: i.e., clone + install following README, pip install ..., etc. I don't care what these build steps look like. If using meson install to install everything makes the most sense, fine; or if running meson and pip is necessary, also fine. My goal is to make things work consistently and reliably. Happy to add an extra step or two to the process if it means things will work consistently.

Since I would like someone to be able to just pip install butterfly (or for another Python project to depend on butterfly), it sounds like PyPI + wheels is mandatory. And in this case, since it's discouraged for pip to install a dependency itself, the shared library should be bundled, or the C library should even just be built statically and linked...

Butterfly also depends on several other numerical libraries: OpenBLAS, ARPACK, UMFPACK, GSL, etc, which get picked up either using Meson's find_library or dependency. It sounds like this may also not play nicely with Python packaging...

[dnicolodi]

Looking at the output from ldconfig and at the meson install log you posted previously, the problem is that Meson installed the shared library in /usr/local/lib64/ but this path is not in the list of the paths where the dynamic linker looks for shared libraries. It is customary to install software that is not packaged by the distribution into /usr/local/, and for the dynamic linker to be configured to look into /usr/local/lib/ or /usr/local/lib64/. I don't know why your distribution is not doing that.

What Linux distribution are you using? You may ask on the support channel for your distribution for guidance.

Or you can simply drop a file in /etc/ld.so.conf.d/ with a line containing /usr/local/lib64 and issue ldconfig as the root user.

[sampotter]

I'm running Fedora 38. It appears that having /usr/local/lib and /usr/local/lib64 globally in the dynamic linker path might be discouraged on Fedora for some reason. Searching around, I was able to turn up some comments related to this, but couldn't find a good explanation. I would assume Meson would get this right by default since Fedora is a very widely used distribution...

[sampotter]

For now I've fixed this by passing --prefix=/usr when running meson setup ....

[dnicolodi]

Which path is used on Fedora for locally installed software that is not installed as a distribution package?

I would assume Meson would get this right by default since Fedora is a very widely used distribution...

I don't understand what you mean. /usr/local/ is the default installation prefix used by Meson on Unix-like systems. If you install in this prefix, libraries go in /usr/local/lib/ or /usr/local/lib64/. If you want to use another installation prefix, just specify it. There isn't anything Meson is doing wrong or that Meson could be doing differently.

[sampotter]

It does appear that Fedora uses /usr/local for software that isn't a distribution package.

It also looks like Fedora omits /usr/local/lib or /usr/local/lib64 on purpose, while many other distributions don't.

Because Meson's (reasonable) default prefix is /usr/local, it appears to me I will need to include a notice with my library directed to Fedora users that they need to either pass --prefix=/usr (maybe problematic?) or add something to /etc/ld.so.conf.d (annoying).