[nanvix] E: Build _ssl/_hashlib/_ctypes as .so#19
Conversation
6b31bca to
86e1659
Compare
0533de4 to
9e4ec0f
Compare
86e1659 to
a78b0c8
Compare
9e4ec0f to
9119925
Compare
a78b0c8 to
2f602bf
Compare
cffb2a8 to
a196813
Compare
64c14e9 to
6819a97
Compare
c405ce5 to
b686ee5
Compare
3e503b9 to
c96a209
Compare
Converts the three stdlib extensions whose external dependencies are Nanvix-ported libraries that already ship as .so (libssl/libcrypto from openssl, libffi) from *static* (linked into python.elf) to *shared* (lib/python3.12/lib-dynload/<name>.cpython-312.so), each emitting DT_NEEDED for the corresponding sysroot library: _ctypes.cpython-312.so -> libffi.so _ssl.cpython-312.so -> libssl.so -> libcrypto.so _hashlib.cpython-312.so -> libcrypto.so (shared with _ssl, deduped) This is exactly how upstream cpython builds these modules on a Linux distro with --with-system-* detection enabled (the default). The Nanvix dynamic loader walks the DT_NEEDED chain at dlopen time (the openssl chain forms a diamond at libcrypto.so) and UND symbols in each .so bind to python.elf .dynsym. _ssl and _hashlib previously embedded their own copies of libcrypto (~5.5 MB each); routing both through the single libcrypto.so ends the OpenSSL init-order bug where the two copies maintained separate provider state and the first openssl_sha256() call failed. python.elf LIBS drops -lssl -lcrypto -lffi. _bz2 / _lzma / zlib / _sqlite3 are intentionally NOT moved here: the Nanvix port repos for libbz2 / liblzma / libz / libsqlite3 do not yet ship .so builds, so those four extensions stay statically built into python.elf (cpython upstream default) until the follow-up PR that lands alongside the Wave 6 port-repo .so PRs. Infrastructure -------------- - .nanvix/runtime_sos.py (new): REQUIRED_RUNTIME_SOS single source of truth (libffi.so, libcrypto.so, libssl.so) + stage_runtime_sos(), consumed by .nanvix/test.py and .nanvix/package.py to copy the .so files from the buildroot into the test / release sysroot. - .nanvix/setup_local.py: _ssl / _hashlib / _ctypes shared entries. - .nanvix/test.py: CPYTHON_TEST_EXTERNAL_DEPS smoke checks. - .nanvix/z.py: _DEP_EXPECTED_LIBS validates libffi.so + libssl.so / libcrypto.so installed into the buildroot at `./z setup` time. - Makefile.nanvix: LIBS drops -lssl -lcrypto -lffi. Runtime dependencies (already merged upstream) ---------------------------------------------- - nanvix/nanvix#2473 -- dlfcn init-array + DT_RUNPATH support. - nanvix/nanvix#2478 -- diamond DT_NEEDED handling (openssl chain). - nanvix/nanvix#2481 -- pthread_once (critical: OpenSSL routes every internal one-time init through pthread_once). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
c96a209 to
d1070cb
Compare
|
Updated: removed all lxml content from this PR. It now contains only the The lxml integration on Nanvix (a flat-name built-in Commit retitled |
Summary
Converts the three stdlib extensions whose external dependencies are Nanvix-ported libraries that already ship as
.so(libssl/libcryptofrom openssl,libffi) from*static*(linked intopython.elf) to*shared*(lib/python3.12/lib-dynload/<name>.cpython-312.so), each emittingDT_NEEDEDfor the corresponding sysroot library.This is exactly how upstream CPython builds these modules on a Linux distro with
--with-system-*detection enabled (the default).DT_NEEDED chains
_ctypes.cpython-312.so→DT_NEEDED libffi.so_ssl.cpython-312.so→DT_NEEDED libssl.so→DT_NEEDED libcrypto.so_hashlib.cpython-312.so→DT_NEEDED libcrypto.soThe Nanvix dynamic loader walks the chain at
dlopen()time (the openssl chain forms a diamond atlibcrypto.so), and UND symbols in each.sobind topython.elf .dynsym. Same flow Linux distros use._ssland_hashlibpreviously embedded their own copies oflibcrypto(~5.5 MB each); routing both through the singlelibcrypto.soends the OpenSSL init-order bug where the two copies maintained separate provider state and the firstopenssl_sha256()call failed.Scope
python.elfLIBS drops-lssl -lcrypto -lffi._bz2/_lzma/zlib/_sqlite3are intentionally not moved here — the Nanvix port repos forlibbz2/liblzma/libz/libsqlite3do not yet ship.sobuilds, so those four extensions stay statically built intopython.elf(CPython upstream default) until those port repos ship.sobuilds and a follow-up PR switches the four extensions toDT_NEEDED.lxml is not part of this PR. lxml is a third-party package, not a CPython stdlib module, and the way it is currently integrated on Nanvix (a flat-name built-in shim) diverges from how upstream CPython loads third-party extensions. That divergence is addressed separately — this PR stays focused on the stdlib modules that match upstream's system-library
.soflow.Mechanics
.nanvix/runtime_sos.py(new)REQUIRED_RUNTIME_SOSsingle source of truth (libffi.so,libcrypto.so,libssl.so) +stage_runtime_sos(), consumed by.nanvix/test.pyand.nanvix/package.pyto copy the.sofiles from the buildroot into the test / release sysroot..nanvix/setup_local.py_ssl/_hashlib/_ctypesshared entries..nanvix/test.pyCPYTHON_TEST_EXTERNAL_DEPSsmoke checks (import+ trivial attribute access)..nanvix/z.py_DEP_EXPECTED_LIBSvalidateslibffi.so+libssl.so/libcrypto.soinstalled into the buildroot at./z setuptime.Makefile.nanvix-lssl -lcrypto -lffi.Dependencies
Base branch:
feat/wave5-pr-a-stdlib-so(nanvix/cpython#732).Port-repo PRs that produce the required
.sofiles (already filed upstream):.solibffi.solibcrypto.so,libssl.soRuntime dependencies (already merged upstream):
dlfcninit-array + DT_RUNPATH support.libcrypto.soboth vialibssl.soand directly from_hashlib).pthread_once(critical: OpenSSL routes every internal one-time init throughpthread_once; without it everyEVP_*call fails).