Skip to content

[nanvix] E: Build _ssl/_hashlib/_ctypes as .so#19

Open
esaurez wants to merge 1 commit into
feat/wave5-pr-a-stdlib-sofrom
feat/wave5-pr-b-unbundle-externals
Open

[nanvix] E: Build _ssl/_hashlib/_ctypes as .so#19
esaurez wants to merge 1 commit into
feat/wave5-pr-a-stdlib-sofrom
feat/wave5-pr-b-unbundle-externals

Conversation

@esaurez

@esaurez esaurez commented Jun 9, 2026

Copy link
Copy Markdown
Owner

Summary

Converts the three stdlib extensions whose external dependencies are Nanvix-ported libraries that already ship as .so (libssl/libcrypto from openssl, libffi) from *static* (linked into python.elf) to *shared* (lib/python3.12/lib-dynload/<name>.cpython-312.so), each emitting DT_NEEDED for the corresponding sysroot library.

This is exactly how upstream CPython builds these modules on a Linux distro with --with-system-* detection enabled (the default).

DT_NEEDED chains

  • _ctypes.cpython-312.soDT_NEEDED libffi.so
  • _ssl.cpython-312.soDT_NEEDED libssl.soDT_NEEDED libcrypto.so
  • _hashlib.cpython-312.soDT_NEEDED libcrypto.so

The Nanvix dynamic loader walks the chain at dlopen() time (the openssl chain forms a diamond at libcrypto.so), and UND symbols in each .so bind to python.elf .dynsym. Same flow Linux distros use.

_ssl and _hashlib previously embedded their own copies of libcrypto (~5.5 MB each); routing both through the single libcrypto.so ends the OpenSSL init-order bug where the two copies maintained separate provider state and the first openssl_sha256() call failed.

Scope

python.elf LIBS drops -lssl -lcrypto -lffi. _bz2 / _lzma / zlib / _sqlite3 are intentionally not moved here — the Nanvix port repos for libbz2 / liblzma / libz / libsqlite3 do not yet ship .so builds, so those four extensions stay statically built into python.elf (CPython upstream default) until those port repos ship .so builds and a follow-up PR switches the four extensions to DT_NEEDED.

lxml is not part of this PR. lxml is a third-party package, not a CPython stdlib module, and the way it is currently integrated on Nanvix (a flat-name built-in shim) diverges from how upstream CPython loads third-party extensions. That divergence is addressed separately — this PR stays focused on the stdlib modules that match upstream's system-library .so flow.

Mechanics

File Change
.nanvix/runtime_sos.py (new) REQUIRED_RUNTIME_SOS single source of truth (libffi.so, libcrypto.so, libssl.so) + stage_runtime_sos(), consumed by .nanvix/test.py and .nanvix/package.py to copy the .so files from the buildroot into the test / release sysroot.
.nanvix/setup_local.py _ssl / _hashlib / _ctypes shared entries.
.nanvix/test.py CPYTHON_TEST_EXTERNAL_DEPS smoke checks (import + trivial attribute access).
.nanvix/z.py _DEP_EXPECTED_LIBS validates libffi.so + libssl.so / libcrypto.so installed into the buildroot at ./z setup time.
Makefile.nanvix LIBS drops -lssl -lcrypto -lffi.

Dependencies

Base branch: feat/wave5-pr-a-stdlib-so (nanvix/cpython#732).

Port-repo PRs that produce the required .so files (already filed upstream):

Required .so Produced by
libffi.so nanvix/libffi#235
libcrypto.so, libssl.so nanvix/openssl#252

Runtime dependencies (already merged upstream):

@esaurez esaurez force-pushed the feat/wave5-pr-a-stdlib-so branch from 6b31bca to 86e1659 Compare June 9, 2026 22:55
@esaurez esaurez force-pushed the feat/wave5-pr-b-unbundle-externals branch from 0533de4 to 9e4ec0f Compare June 9, 2026 22:56
@esaurez esaurez changed the title [nanvix] E: Unbundle external libs into DT_NEEDED .so chains (Phases 3/3b/4) [nanvix] E: Build external-dep stdlib + lxml extensions as .so Jun 9, 2026
@esaurez esaurez force-pushed the feat/wave5-pr-a-stdlib-so branch from 86e1659 to a78b0c8 Compare June 9, 2026 23:07
@esaurez esaurez force-pushed the feat/wave5-pr-b-unbundle-externals branch from 9e4ec0f to 9119925 Compare June 9, 2026 23:11
@esaurez esaurez force-pushed the feat/wave5-pr-a-stdlib-so branch from a78b0c8 to 2f602bf Compare June 10, 2026 02:32
@esaurez esaurez force-pushed the feat/wave5-pr-b-unbundle-externals branch 2 times, most recently from cffb2a8 to a196813 Compare June 10, 2026 03:03
@esaurez esaurez force-pushed the feat/wave5-pr-a-stdlib-so branch from 64c14e9 to 6819a97 Compare June 12, 2026 23:52
@esaurez esaurez force-pushed the feat/wave5-pr-b-unbundle-externals branch 2 times, most recently from c405ce5 to b686ee5 Compare June 13, 2026 00:06
@esaurez esaurez changed the title [nanvix] E: Build external-dep stdlib + lxml extensions as .so [nanvix] E: Build _ssl/_hashlib/_ctypes + lxml extensions as .so Jun 13, 2026
@esaurez esaurez force-pushed the feat/wave5-pr-b-unbundle-externals branch 2 times, most recently from 3e503b9 to c96a209 Compare June 13, 2026 01:44
Converts the three stdlib extensions whose external dependencies are
Nanvix-ported libraries that already ship as .so (libssl/libcrypto
from openssl, libffi) from *static* (linked into python.elf) to
*shared* (lib/python3.12/lib-dynload/<name>.cpython-312.so), each
emitting DT_NEEDED for the corresponding sysroot library:

  _ctypes.cpython-312.so   -> libffi.so
  _ssl.cpython-312.so      -> libssl.so -> libcrypto.so
  _hashlib.cpython-312.so  -> libcrypto.so  (shared with _ssl, deduped)

This is exactly how upstream cpython builds these modules on a Linux
distro with --with-system-* detection enabled (the default). The
Nanvix dynamic loader walks the DT_NEEDED chain at dlopen time (the
openssl chain forms a diamond at libcrypto.so) and UND symbols in
each .so bind to python.elf .dynsym.

_ssl and _hashlib previously embedded their own copies of libcrypto
(~5.5 MB each); routing both through the single libcrypto.so ends the
OpenSSL init-order bug where the two copies maintained separate
provider state and the first openssl_sha256() call failed.

python.elf LIBS drops -lssl -lcrypto -lffi. _bz2 / _lzma / zlib /
_sqlite3 are intentionally NOT moved here: the Nanvix port repos for
libbz2 / liblzma / libz / libsqlite3 do not yet ship .so builds, so
those four extensions stay statically built into python.elf (cpython
upstream default) until the follow-up PR that lands alongside the
Wave 6 port-repo .so PRs.

Infrastructure
--------------
- .nanvix/runtime_sos.py (new): REQUIRED_RUNTIME_SOS single source of
  truth (libffi.so, libcrypto.so, libssl.so) + stage_runtime_sos(),
  consumed by .nanvix/test.py and .nanvix/package.py to copy the .so
  files from the buildroot into the test / release sysroot.
- .nanvix/setup_local.py: _ssl / _hashlib / _ctypes shared entries.
- .nanvix/test.py: CPYTHON_TEST_EXTERNAL_DEPS smoke checks.
- .nanvix/z.py: _DEP_EXPECTED_LIBS validates libffi.so + libssl.so /
  libcrypto.so installed into the buildroot at `./z setup` time.
- Makefile.nanvix: LIBS drops -lssl -lcrypto -lffi.

Runtime dependencies (already merged upstream)
----------------------------------------------
- nanvix/nanvix#2473 -- dlfcn init-array + DT_RUNPATH support.
- nanvix/nanvix#2478 -- diamond DT_NEEDED handling (openssl chain).
- nanvix/nanvix#2481 -- pthread_once (critical: OpenSSL routes every
  internal one-time init through pthread_once).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@esaurez esaurez force-pushed the feat/wave5-pr-b-unbundle-externals branch from c96a209 to d1070cb Compare June 16, 2026 21:22
@esaurez esaurez changed the title [nanvix] E: Build _ssl/_hashlib/_ctypes + lxml extensions as .so [nanvix] E: Build _ssl/_hashlib/_ctypes as .so Jun 16, 2026
@esaurez

esaurez commented Jun 16, 2026

Copy link
Copy Markdown
Owner Author

Updated: removed all lxml content from this PR. It now contains only the _ssl / _hashlib / _ctypes.so conversion, which matches upstream CPython's system-library .so flow exactly.

The lxml integration on Nanvix (a flat-name built-in _lxml_etree shim that forwards PyInit) diverges from how upstream CPython loads third-party packages, so it does not belong in a PR whose goal is "match upstream." lxml is reverted here to its pre-existing static state and will be removed entirely from CPython in a separate PR (it will return properly once the nanvix/lxml port emits native CPython extension modules — tracked in lxml-port-ship-as-native-cpython-extensions.md).

Commit retitled [nanvix] E: Build _ssl/_hashlib/_ctypes as .so; #20 and #21 rebased onto the cleaned commit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant