difference between isolated and non-isolated builds

[rdenham]

I've run into a problem when running a python meson build project. It's one of those cases in which it used to work, but no longer does, so I suspect its something to do with change in versions of some unpinned dependencies.

In my code, using the python build package, if I do:

python3 -m pip install meson-python patchelf ninja numpy<2
python3 build . --no-isolation

everything works ok. But if I do

python3 build . 

then the build looks the same, no errors, but the library built doesn't perform the same as the 'no-isolation' build, and worse, can throw segfaults.

I thought it might have been due to the introduction of numpy 2, so in my pyproject.toml I have:

[build-system]
build-backend = 'mesonpy'
requires = ['meson-python', 'numpy==1.26.4']

to try to match to the same version of python as in the non-isolated build, but the problem persists. I looked at the logs for both the isolated and non-isolated builds, but I couldn't see anything that stood out.

Can anybody suggest what to do to isolate the effect, and suggest a fix for this? I'd prefer to use build isolation if possible.

Here's a diff of the logs:

diff noisolation.log isolation.log 
0a1,4
> v* Creating isolated environment: venv+pip...
> * Installing packages in isolated environment:
>   - meson-python
>   - numpy==1.26.4
3c7
< + meson setup /export/home/denhamr/tmp/test1/rscutils /export/home/denhamr/tmp/test1/rscutils/.mesonpy-c5pbzed1 -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/export/home/denhamr/tmp/test1/rscutils/.mesonpy-c5pbzed1/meson-python-native-file.ini
---
> + meson setup /export/home/denhamr/tmp/test1/rscutils /export/home/denhamr/tmp/test1/rscutils/.mesonpy-hzwk_gb8 -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/export/home/denhamr/tmp/test1/rscutils/.mesonpy-hzwk_gb8/meson-python-native-file.ini
7c11
< Build dir: /export/home/denhamr/tmp/test1/rscutils/.mesonpy-c5pbzed1
---
> Build dir: /export/home/denhamr/tmp/test1/rscutils/.mesonpy-hzwk_gb8
19c23
< Program python3 found: YES (/export/home/denhamr/tmp/test1/venvsys/bin/python3)
---
> Program python3 found: YES (/tmp/build-env-wot6w9gq/bin/python)
23c27
< Message: /export/home/denhamr/tmp/test1/venvsys/bin/python3
---
> Message: /tmp/build-env-wot6w9gq/bin/python
32c36
<     Native files: /export/home/denhamr/tmp/test1/rscutils/.mesonpy-c5pbzed1/meson-python-native-file.ini
---
>     Native files: /export/home/denhamr/tmp/test1/rscutils/.mesonpy-hzwk_gb8/meson-python-native-file.ini
43c47
< Created /export/home/denhamr/tmp/test1/rscutils/.mesonpy-c5pbzed1/meson-dist/rsc-2.0a.1.tar.gz
---
> Created /export/home/denhamr/tmp/test1/rscutils/.mesonpy-hzwk_gb8/meson-dist/rsc-2.0a.1.tar.gz
44a49,52
> * Creating isolated environment: venv+pip...
> * Installing packages in isolated environment:
>   - meson-python
>   - numpy
47c55
< + meson setup /tmp/build-via-sdist-z97ot6ix/rsc-2.0a1 /tmp/build-via-sdist-z97ot6ix/rsc-2.0a1/.mesonpy-hl8g3gzl -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/tmp/build-via-sdist-z97ot6ix/rsc-2.0a1/.mesonpy-hl8g3gzl/meson-python-native-file.ini
---
> + meson setup /tmp/build-via-sdist-xlooy1g0/rsc-2.0a1 /tmp/build-via-sdist-xlooy1g0/rsc-2.0a1/.mesonpy-_x07ktfk -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/tmp/build-via-sdist-xlooy1g0/rsc-2.0a1/.mesonpy-_x07ktfk/meson-python-native-file.ini
50,51c58,59
< Source dir: /tmp/build-via-sdist-z97ot6ix/rsc-2.0a1
< Build dir: /tmp/build-via-sdist-z97ot6ix/rsc-2.0a1/.mesonpy-hl8g3gzl
---
> Source dir: /tmp/build-via-sdist-xlooy1g0/rsc-2.0a1
> Build dir: /tmp/build-via-sdist-xlooy1g0/rsc-2.0a1/.mesonpy-_x07ktfk
63c71
< Program python3 found: YES (/export/home/denhamr/tmp/test1/venvsys/bin/python3)
---
> Program python3 found: YES (/tmp/build-env-hqled7xa/bin/python)
67c75
< Message: /export/home/denhamr/tmp/test1/venvsys/bin/python3
---
> Message: /tmp/build-env-hqled7xa/bin/python
76c84
<     Native files: /tmp/build-via-sdist-z97ot6ix/rsc-2.0a1/.mesonpy-hl8g3gzl/meson-python-native-file.ini
---
>     Native files: /tmp/build-via-sdist-xlooy1g0/rsc-2.0a1/.mesonpy-_x07ktfk/meson-python-native-file.ini
123a132,134
> {'attrspec': ['intent(in)']}
> In: :vecmedian:../src/rsc/stats/vecmedian.f90:vectormedian
> vars2fortran: No typespec for argument "usenull".
124a136,138
> {'attrspec': ['intent(in)']}
> In: :vecmedian:../src/rsc/stats/vecmedian.f90:selectmedianimage
> vars2fortran: No typespec for argument "medianndx".
140c154
< [4/14] Generating valndx_so with a custom command
---
> [3/14] Generating valndx_so with a custom command
196,197c210,211
< vecmedianmodule.c:200:67: warning: unused parameter ‘capi_self’ [-Wunused-parameter]
<   200 | static PyObject *f2py_rout_vecmedian_vectormedian(const PyObject *capi_self,
---
> vecmedianmodule.c:248:67: warning: unused parameter ‘capi_self’ [-Wunused-parameter]
>   248 | static PyObject *f2py_rout_vecmedian_vectormedian(const PyObject *capi_self,
200,201c214,215
< vecmedianmodule.c:349:72: warning: unused parameter ‘capi_self’ [-Wunused-parameter]
<   349 | static PyObject *f2py_rout_vecmedian_selectmedianimage(const PyObject *capi_self,
---
> vecmedianmodule.c:396:72: warning: unused parameter ‘capi_self’ [-Wunused-parameter]
>   396 | static PyObject *f2py_rout_vecmedian_selectmedianimage(const PyObject *capi_self,
204,205c218,219
< vecmedianmodule.c:517:81: warning: cast between incompatible function types from ‘PyObject * (*)(const PyObject *, PyObject *, PyObject *, void (*)(float *, int *, int *, float *, int *, int *, int *, int *))’ {aka ‘struct _object * (*)(const struct _object *, struct _object *, struct _object *, void (*)(float *, int *, int *, float *, int *, int *, int *, int *))’} to ‘void (*)(int *, npy_intp *, void (*)(char *, npy_intp *), int *)’ {aka ‘void (*)(int *, long int *, void (*)(char *, long int *), int *)’} [-Wcast-function-type]
<   517 |     {"vectormedian",-1,{{-1}},0,0,(char *)  F_FUNC(vectormedian,VECTORMEDIAN),  (f2py_init_func)f2py_rout_vecmedian_vectormedian,doc_f2py_rout_vecmedian_vectormedian},
---
> vecmedianmodule.c:547:81: warning: cast between incompatible function types from ‘PyObject * (*)(const PyObject *, PyObject *, PyObject *, void (*)(float *, int *, float *, float *, int *, int *, int *, int *))’ {aka ‘struct _object * (*)(const struct _object *, struct _object *, struct _object *, void (*)(float *, int *, float *, float *, int *, int *, int *, int *))’} to ‘void (*)(int *, npy_intp *, void (*)(char *, npy_intp *), int *)’ {aka ‘void (*)(int *, long int *, void (*)(char *, long int *), int *)’} [-Wcast-function-type]
>   547 |     {"vectormedian",-1,{{-1}},0,0,(char *)  F_FUNC(vectormedian,VECTORMEDIAN),  (f2py_init_func)f2py_rout_vecmedian_vectormedian,doc_f2py_rout_vecmedian_vectormedian},
207,208c221,222
< vecmedianmodule.c:518:96: warning: cast between incompatible function types from ‘PyObject * (*)(const PyObject *, PyObject *, PyObject *, void (*)(float *, float *, float *, float *, int *, int *, int *, int *))’ {aka ‘struct _object * (*)(const struct _object *, struct _object *, struct _object *, void (*)(float *, float *, float *, float *, int *, int *, int *, int *))’} to ‘void (*)(int *, npy_intp *, void (*)(char *, npy_intp *), int *)’ {aka ‘void (*)(int *, long int *, void (*)(char *, long int *), int *)’} [-Wcast-function-type]
<   518 |     {"selectmedianimage",-1,{{-1}},0,0,(char *)  F_FUNC(selectmedianimage,SELECTMEDIANIMAGE),  (f2py_init_func)f2py_rout_vecmedian_selectmedianimage,doc_f2py_rout_vecmedian_selectmedianimage},
---
> vecmedianmodule.c:548:96: warning: cast between incompatible function types from ‘PyObject * (*)(const PyObject *, PyObject *, PyObject *, void (*)(float *, int *, float *, float *, int *, int *, int *, int *))’ {aka ‘struct _object * (*)(const struct _object *, struct _object *, struct _object *, void (*)(float *, int *, float *, float *, int *, int *, int *, int *))’} to ‘void (*)(int *, npy_intp *, void (*)(char *, npy_intp *), int *)’ {aka ‘void (*)(int *, long int *, void (*)(char *, long int *), int *)’} [-Wcast-function-type]
>   548 |     {"selectmedianimage",-1,{{-1}},0,0,(char *)  F_FUNC(selectmedianimage,SELECTMEDIANIMAGE),  (f2py_init_func)f2py_rout_vecmedian_selectmedianimage,doc_f2py_rout_vecmedian_selectmedianimage},
210,211c224,225
< vecmedianmodule.c:526:5: warning: missing initializer for field ‘ml_flags’ of ‘PyMethodDef’ [-Wmissing-field-initializers]
<   526 |     {NULL,NULL}
---
> vecmedianmodule.c:556:5: warning: missing initializer for field ‘ml_flags’ of ‘PyMethodDef’ [-Wmissing-field-initializers]
>   556 |     {NULL,NULL}
218,220c232,234
< [7/14] Compiling C object valndx.cpython-311-x86_64-linux-gnu...f0a2c061ba4455_site-packages_numpy_f2py_src_fortranobject.c.o
< ../../../../export/home/denhamr/tmp/test1/venvsys/lib/python3.11/site-packages/numpy/f2py/src/fortranobject.c: In function ‘fortran_call’:
< ../../../../export/home/denhamr/tmp/test1/venvsys/lib/python3.11/site-packages/numpy/f2py/src/fortranobject.c:529:23: warning: cast between incompatible function types from ‘f2py_init_func’ {aka ‘void (*)(int *, long int *, void (*)(char *, long int *), int *)’} to ‘PyObject * (*)(PyObject *, PyObject *, PyObject *, void *)’ {aka ‘struct _object * (*)(struct _object *, struct _object *, struct _object *, void *)’} [-Wcast-function-type]
---
> [8/14] Compiling C object valndx.cpython-311-x86_64-linux-gnu...lib_python3.11_site-packages_numpy_f2py_src_fortranobject.c.o
> ../../../build-env-hqled7xa/lib/python3.11/site-packages/numpy/f2py/src/fortranobject.c: In function ‘fortran_call’:
> ../../../build-env-hqled7xa/lib/python3.11/site-packages/numpy/f2py/src/fortranobject.c:529:23: warning: cast between incompatible function types from ‘f2py_init_func’ {aka ‘void (*)(int *, long int *, void (*)(char *, long int *), int *)’} to ‘PyObject * (*)(PyObject *, PyObject *, PyObject *, void *)’ {aka ‘struct _object * (*)(struct _object *, struct _object *, struct _object *, void *)’} [-Wcast-function-type]
223c237
< ../../../../export/home/denhamr/tmp/test1/venvsys/lib/python3.11/site-packages/numpy/f2py/src/fortranobject.c:532:23: warning: cast between incompatible function types from ‘f2py_init_func’ {aka ‘void (*)(int *, long int *, void (*)(char *, long int *), int *)’} to ‘PyObject * (*)(PyObject *, PyObject *, PyObject *, void *)’ {aka ‘struct _object * (*)(struct _object *, struct _object *, struct _object *, void *)’} [-Wcast-function-type]
---
> ../../../build-env-hqled7xa/lib/python3.11/site-packages/numpy/f2py/src/fortranobject.c:532:23: warning: cast between incompatible function types from ‘f2py_init_func’ {aka ‘void (*)(int *, long int *, void (*)(char *, long int *), int *)’} to ‘PyObject * (*)(PyObject *, PyObject *, PyObject *, void *)’ {aka ‘struct _object * (*)(struct _object *, struct _object *, struct _object *, void *)’} [-Wcast-function-type]
226,228c240,242
< [8/14] Compiling C object vecmedian.cpython-311-x86_64-linux-...f0a2c061ba4455_site-packages_numpy_f2py_src_fortranobject.c.o
< ../../../../export/home/denhamr/tmp/test1/venvsys/lib/python3.11/site-packages/numpy/f2py/src/fortranobject.c: In function ‘fortran_call’:
< ../../../../export/home/denhamr/tmp/test1/venvsys/lib/python3.11/site-packages/numpy/f2py/src/fortranobject.c:529:23: warning: cast between incompatible function types from ‘f2py_init_func’ {aka ‘void (*)(int *, long int *, void (*)(char *, long int *), int *)’} to ‘PyObject * (*)(PyObject *, PyObject *, PyObject *, void *)’ {aka ‘struct _object * (*)(struct _object *, struct _object *, struct _object *, void *)’} [-Wcast-function-type]
---
> [9/14] Compiling C object vecmedian.cpython-311-x86_64-linux-...lib_python3.11_site-packages_numpy_f2py_src_fortranobject.c.o
> ../../../build-env-hqled7xa/lib/python3.11/site-packages/numpy/f2py/src/fortranobject.c: In function ‘fortran_call’:
> ../../../build-env-hqled7xa/lib/python3.11/site-packages/numpy/f2py/src/fortranobject.c:529:23: warning: cast between incompatible function types from ‘f2py_init_func’ {aka ‘void (*)(int *, long int *, void (*)(char *, long int *), int *)’} to ‘PyObject * (*)(PyObject *, PyObject *, PyObject *, void *)’ {aka ‘struct _object * (*)(struct _object *, struct _object *, struct _object *, void *)’} [-Wcast-function-type]
231c245
< ../../../../export/home/denhamr/tmp/test1/venvsys/lib/python3.11/site-packages/numpy/f2py/src/fortranobject.c:532:23: warning: cast between incompatible function types from ‘f2py_init_func’ {aka ‘void (*)(int *, long int *, void (*)(char *, long int *), int *)’} to ‘PyObject * (*)(PyObject *, PyObject *, PyObject *, void *)’ {aka ‘struct _object * (*)(struct _object *, struct _object *, struct _object *, void *)’} [-Wcast-function-type]
---
> ../../../build-env-hqled7xa/lib/python3.11/site-packages/numpy/f2py/src/fortranobject.c:532:23: warning: cast between incompatible function types from ‘f2py_init_func’ {aka ‘void (*)(int *, long int *, void (*)(char *, long int *), int *)’} to ‘PyObject * (*)(PyObject *, PyObject *, PyObject *, void *)’ {aka ‘struct _object * (*)(struct _object *, struct _object *, struct _object *, void *)’} [-Wcast-function-type]
303c317
< [62/62] /tmp/build-via-sdist-z97ot6ix/rsc-2.0a1/src/rsc/data/l9olre_p108r070_20220418_da0m2.meta
---
> [62/62] /tmp/build-via-sdist-xlooy1g0/rsc-2.0a1/src/rsc/data/l9olre_p108r070_20220418_da0m2.meta

[rgommers]

Hmm, that's hard to answer without any errors. The one thing I can see is that based on line numbers of vecmedianmodule.c being different, you're probably getting a different numpy version. f2py-generated code should be identical for the same numpy version. It may be worth checking those versions first, and if that doesn't yield a lead then looking at the difference in the generated code.

[rdenham]

Thank you @rgommers yes you are correct, and the problem was due to a lack of understanding on my part.

For posterity, this is the mistake I made:

• code is in a git repository
• originally pyproject.toml specified requires = ['meson-python', 'numpy'] and when I tested that by building and installing, my test failed
• I then thought I should specify that I need numpy <2, and modified the pyproject.toml accordingly but did not commit this change
• I then tried again, but still had the same issue.
• what actually happens is that, at least when using python -m build, you first get a virtual environment with the required settings, so 'numpy<2' is used according to pyproject.toml, and the sdist is created.
• once that's finished, the sdist is used to build the wheel. This is done in a new virtual environment.
• since I hadn't commited pyproject.toml, the version in the sdist tar.gz file had the original, unpinned numpy requirement. So the step in which the real work is done was using numpy 2.2, and my old fortran code was failing.

The solution was just to update pyproject.toml, commit it, and then build.

[rgommers]

The even easier fix is python -m build --wheel, since it avoids going through an sdist, which is not useful and is the thing that doesn't pick up uncommitted changes (rationale: an sdist is only for releases, and you never want to have a source release that doesn't match the tagged commit for that release).