[BUG] Fatal Python error: PyMUTEX_LOCK(gil->mutex) failed

[coreyjadams]

Hi,

I've got a package that uses pybind11 (it's awesome, by the way), and had a few users report the following crash. I've been able to reproduce it myself as well. I've asked on the gitter site and had a good discussion with @quantotto but ultimately we came only down to speculation.

I've reduced the issue to a minimum reproducer, so hopefully this is possible to debug. It seems to be a somewhat hidden issue, doesn't appear in every version of python or compiler.

Issue description

When importing a package built with pybind11, the python libraries fail to load with the following error:

Fatal Python error: PyMUTEX_LOCK(gil->mutex) failed
Python runtime state: unknown

Abort trap: 6

This only seems to appear when using conda's python on Mac OS. I haven't reproduced it elsewhere. Suspect that python is built with clang 10 from conda, while the python package in question is built with clang11, and some incompatibility arises.

Reproducible example code

I am sorry I can not give you a more simple example. I've stripped it down as far as I think I can and still reproduce this.

See the repository here: larcv3-pybind11-example
This uses scikit-build to call cmake and build a package including pybind11-generated python bindings.

Here's a list of instructions to reproduce this:

bash Miniconda3-latest-MacOSX-x86_64.sh #install a fresh conda
source miniconda3/bin/activate # activate it
conda install cmake # install cmake
pip install scikit-build # install scikit-build
git clone https://github.com/coreyjadams/larcv3-pybind11-example.git # clone the example
cd larcv3-pybind11-example/ 
git submodule update --init # clone pybind11 as a submodule
python setup.py build # compile
python setup.py install #install

Then, in a python interpreter you can do:

>>> from larcv import pylarcv
Fatal Python error: PyMUTEX_LOCK(gil->mutex) failed
Python runtime state: unknown

Abort trap: 6

This also appears to be related to this github issue: https://stackoverflow.com/questions/66026520/fatal-python-error-pymutex-lock-pyruntime-ceval-gil-mutex-failed

[melMass]

Can you try otool -L larcv3.dylib and report?
I had the same issue when linking against the wrong Python Framework library

[Jean1995]

Has there been any progress on this issue? I've been experiencing the same bug for our software when people tried to install it in MacOS using anaconda (i.e. pip install proposal and import proposal)

[cqc-alec]

I have also experienced this, building a pybind11 project on MacOS 11.6 with M1 (arm64) architecture and python built locally with pyenv (so no conda or anaconda). It's completely reproducible, but I need to try to reduce it to a minimal example...

[melMass]

I have also experienced this, building a pybind11 project on MacOS 11.6 with M1 (arm64) architecture and python built locally with pyenv (so no conda or anaconda). It's completely reproducible, but I need to try to reduce it to a minimal example...

What does otool -L /path/to/lib returns?

I had the issue that in some instance the python exe used for building was hard linked on Mac!

[coreyjadams]

Thanks @melMass for the reminder. Using the library I posted above which is a reproducer, I get this:

This one is the python bindings:

$ otool -L /Users/corey.adams/miniconda3/lib/python3.8/site-packages/pybind11_test_symbols-1.0.0-py3.8-macosx-10.9-x86_64.egg/larcv/pylarcv.cpython-38-darwin.so
/Users/corey.adams/miniconda3/lib/python3.8/site-packages/pybind11_test_symbols-1.0.0-py3.8-macosx-10.9-x86_64.egg/larcv/pylarcv.cpython-38-darwin.so:
	@rpath/pylarcv.cpython-38-darwin.so (compatibility version 0.0.0, current version 0.0.0)
	@rpath/liblarcv3.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libpython3.8.dylib (compatibility version 3.8.0, current version 3.8.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 800.7.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.0.0)
(base)

And, this is the base C++ library that is getting bound to python:

$ otool -L /Users/corey.adams/miniconda3/lib/python3.8/site-packages/pybind11_test_symbols-1.0.0-py3.8-macosx-10.9-x86_64.egg/larcv/lib/liblarcv3.dylib
/Users/corey.adams/miniconda3/lib/python3.8/site-packages/pybind11_test_symbols-1.0.0-py3.8-macosx-10.9-x86_64.egg/larcv/lib/liblarcv3.dylib:
	@rpath/liblarcv3.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libpython3.8.dylib (compatibility version 3.8.0, current version 3.8.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 800.7.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.0.0)
(base)

[cqc-alec]

In my case:

alec@Mac-mini pytket % otool -L pytket/_tket/circuit.cpython-38-darwin.so
pytket/_tket/circuit.cpython-38-darwin.so:
	@loader_path/libtket.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 905.6.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.100.5)

and

alec@Mac-mini pytket % otool -L pytket/_tket/libtket.dylib               
pytket/_tket/libtket.dylib:
	@loader_path/libtket.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 905.6.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.100.5)

[cqc-alec]

So I have produced a very minimal example to reproduce this issue: https://github.com/cqc-alec/pybind11-3081
The C++ and binder code is utterly trivial.
The build commands are in the Makefile, which includes a hard-coded path to the pybind11 2.7.1 headers installed with conan on a Mac (M1).
The output of make test is:

/Library/Developer/CommandLineTools/usr/bin/c++ -I. -stdlib=libc++ -arch arm64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -fPIC -std=c++2a -o A.cpp.o -c A.cpp
/Library/Developer/CommandLineTools/usr/bin/c++ -stdlib=libc++ -arch arm64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -dynamiclib -Wl,-headerpad_max_install_names -o libA.dylib -install_name @loader_path/libA.dylib A.cpp.o
/Library/Developer/CommandLineTools/usr/bin/c++ -I. -isystem /Users/alec/.conan/data/pybind11/2.7.1/_/_/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/include -isystem /Users/alec/.pyenv/versions/3.8.11/include/python3.8 -arch arm64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -fPIC -fvisibility=hidden -std=c++2a -MD -MT binder.cpp.o -MF binder.cpp.o.d -o binder.cpp.o -c binder.cpp
/Library/Developer/CommandLineTools/usr/bin/c++ -arch arm64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -bundle -Wl,-headerpad_max_install_names -Xlinker -undefined -Xlinker dynamic_lookup -o A.cpython-38-darwin.so binder.cpp.o -L. -lA  /Users/alec/.pyenv/versions/3.8.11/lib/libpython3.8.a
/Library/Developer/CommandLineTools/usr/bin/strip -x A.cpython-38-darwin.so
/Users/alec/.pyenv/versions/3.8.11/bin/python -c "from A import A"
Fatal Python error: PyMUTEX_LOCK(gil->mutex) failed
Python runtime state: unknown

make: *** [test] Abort trap: 6

(In order to construct this example I extracted these commands from a build system that uses conan and cmake under the hood.)

[cqc-alec]

I see that the problem is caused by the hard linkage with libpython3.8.a. If I omit that, it works! So this is looking very much not like a problem with pybind11, but perhaps with either conan or cmake.

[coreyjadams]

Hey @cqc-alec thanks for digging into this too. Interesting result - though, I tried in my reproducer and it's not so trivial to remove the link to python: I have direct calls to pybind11 in my normal code (equivalent to A.hpp and A.cpp) and this has to link to python.

I note that your libraries aren't directly linked to python above either, is that deliberate?

Overall, I'm very confused by this crash.

[cqc-alec]

@coreyjadams I guess my usage is different, in that my core C++ code knows nothing about python or pybind11.
Whatever the actual cause of this crash, I believe my real problem is conan-io/conan-center-index#6605 : the issue arose when I updated to the latest pybind11 conan package, which has the misfeature that it always links against the full list of targets -- including pybind11:embed which could explain the presence of this linkage.

[melMass]

@coreyjadams I guess my usage is different, in that my core C++ code knows nothing about python or pybind11. Whatever the actual cause of this crash, I believe my real problem is conan-io/conan-center-index#6605 : the issue arose when I updated to the latest pybind11 conan package, which has the misfeature that it always links against the full list of targets -- including pybind11:embed which could explain the presence of this linkage.

I had the exact same setup when this error occured for me.
I'll try to explain how I fixed it:

So I'm, like you, using conan to package all of my dependencies.
I'm also using a python virtual env (using poetry) to match the py version I want to target.

Using the classic find_package(pybind11 REQUIRED) & then pybind11_add_module(XX MODULE $SRCS)
lead to python being hardlinked causing the PyMutex_Lock.

To solve this and I'm not sure it's the best way but it works, was to add a custom CMake module:

FindPythonPyEnv.cmake

# Find informations about the current python environment.
# by melMass
#
# Finds the following:
#
# - PYTHON_EXECUTABLE
# - PYTHON_INCLUDE_DIR
# - PYTHON_LIBRARY
# - PYTHON_SITE
# - PYTHON_NUMPY_INCLUDE_DIR
#
# - PYTHONLIBS_VERSION_STRING (The full version id. ie "3.7.4")
# - PYTHON_VERSION_MAJOR
# - PYTHON_VERSION_MINOR
# - PYTHON_VERSION_PATCH
#
#

function(debug_message messages)
  # message(STATUS "")
  message(STATUS "🐍 ${messages}")
  message(STATUS "\n")
endfunction()

if (NOT DEFINED PYTHON_EXECUTABLE)
  execute_process(
    COMMAND which python
    OUTPUT_VARIABLE PYTHON_EXECUTABLE OUTPUT_STRIP_TRAILING_WHITESPACE
  )
endif()

execute_process(
  COMMAND ${PYTHON_EXECUTABLE} -c "from __future__ import print_function; from distutils.sysconfig import get_python_inc; print(get_python_inc())"
  OUTPUT_VARIABLE PYTHON_INCLUDE_DIR OUTPUT_STRIP_TRAILING_WHITESPACE ERROR_QUIET
)

if (NOT EXISTS ${PYTHON_INCLUDE_DIR})
  message(FATAL "Python include directory not found.")
endif()

execute_process(
  COMMAND ${PYTHON_EXECUTABLE} -c "from __future__ import print_function; import os, numpy.distutils; print(os.pathsep.join(numpy.distutils.misc_util.get_numpy_include_dirs()))"
  OUTPUT_VARIABLE PYTHON_NUMPY_INCLUDE_DIR OUTPUT_STRIP_TRAILING_WHITESPACE ERROR_QUIET
)

execute_process(
  COMMAND ${PYTHON_EXECUTABLE} -c "from __future__ import print_function; import distutils.sysconfig as sysconfig; print('-L' + sysconfig.get_config_var('LIBDIR') + '/' + sysconfig.get_config_var('LDLIBRARY'))"
  OUTPUT_VARIABLE PYTHON_LIBRARY OUTPUT_STRIP_TRAILING_WHITESPACE ERROR_QUIET
)

execute_process(
  COMMAND ${PYTHON_EXECUTABLE} -c "from __future__ import print_function; import platform; print(platform.python_version())"
  OUTPUT_VARIABLE PYTHONLIBS_VERSION_STRING OUTPUT_STRIP_TRAILING_WHITESPACE ERROR_QUIET
)

execute_process(
  COMMAND ${PYTHON_EXECUTABLE} -c "from __future__ import print_function; from distutils.sysconfig import get_python_lib; print(get_python_lib())"
  OUTPUT_VARIABLE PYTHON_SITE OUTPUT_STRIP_TRAILING_WHITESPACE ERROR_QUIET
)

set(PYTHON_VIRTUAL_ENV $ENV{VIRTUAL_ENV})
string(REPLACE "." ";" _VERSION_LIST ${PYTHONLIBS_VERSION_STRING})

list(GET _VERSION_LIST 0 PYTHON_VERSION_MAJOR)
list(GET _VERSION_LIST 1 PYTHON_VERSION_MINOR)
list(GET _VERSION_LIST 2 PYTHON_VERSION_PATCH)



debug_message("Found Python ${PYTHON_VERSION_MAJOR} (${PYTHONLIBS_VERSION_STRING})")
debug_message("PYTHON_EXECUTABLE: ${PYTHON_EXECUTABLE}")
debug_message("PYTHON_INCLUDE_DIR: ${PYTHON_INCLUDE_DIR}")
debug_message("PYTHON_LIBRARY: ${PYTHON_LIBRARY}")
debug_message("PYTHON_NUMPY_INCLUDE_DIR: ${PYTHON_NUMPY_INCLUDE_DIR}")

Let's say you put this CMake module in a cmake folder at the CMAKE_SOURCE_DIR root, this is how you make CMake aware of it:

list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake/")

To solve the issue you need to include it before using find_package(pybind11 REQUIRED), like so:

AFAIK this is only happening on macOS
I hope it will help

[melMass]

To be more complete here is how I run the build:

poetry run sh build_osx.sh path/to/installdir

where build_osx.sh is

mkdir -p "_build"
cd "_build"
conan install .. -s build_type=Release --build=missing 
cmake .. -G "Ninja" -DCMAKE_INSTALL_PREFIX="$1" -DCMAKE_BUILD_TYPE=Release
cmake --build . --config release
cmake --install .

[cqc-alec]

Thank you @melMass , I will try this!