[nanvix] E: Phase 3 — build 7 Tier-3 stdlib extensions as .so (with Group A unbundling)

[esaurez]

Summary

Phase 3 of the .a → .so migration (see nanvix-todo/cpython-static-to-shared-migration.md section 7). Promotes 7 modules with external Nanvix-ported .a dependencies (sysroot libs) from statically linked into python.elf to dlopen-loaded shared objects under lib/python3.12/lib-dynload/.

For 4 of the 7 modules (zlib, _bz2, _lzma, _sqlite3 — "Group A"), this PR additionally does the full Linux-style split: the underlying library lives once in python.elf via --whole-archive --export-dynamic, and each .so resolves its lib symbols at dlopen time. Same architectural pattern as Phase 1B-drop-libm (#7) established for libm.

For the other 3 modules (_ssl, _hashlib, _ctypes — "Group B"), each .so still embeds its own copy of the underlying library. Group B is blocked by pre-existing Nanvix sysroot bugs (see below) and will be optimized in a separate follow-up PR once those bugs are fixed.

Modules moved to *shared*

Group A — fully unbundled

Module	External lib	.so size
zlib	libz	181 KB (was 236)
_bz2	libbz2	68 KB (was 128)
_lzma	liblzma	120 KB (was 212)
_sqlite3	libsqlite3	473 KB (was 1448 — biggest single .so win)
binascii (carrier)	libz crc32	149 KB

Group B — still bundled (deferred follow-up)

Module	External lib	.so size	Blocker
_ssl	libssl + libcrypto	6398 KB	libcrypto.a contains bss_log.o referencing unimplemented POSIX openlog/syslog/closelog
_hashlib	libcrypto (dups _ssl's copy)	4795 KB	same as _ssl
_ctypes	libffi	744 KB	libffi.a contains nested archives (libposix.a, libc.a, libm.a as ar members) that break --whole-archive

These three blockers are tracked in nanvix-todo/phase2-3-unbundle-bundled-libs.md. The fixes are sysroot-side (strip bss_log.o from libcrypto.a; fix or strip nested archives from libffi.a) — not invasive once tackled, but kept separate to avoid mixing sysroot changes into this cpython PR.

Group A implementation

Changes in Makefile.nanvix CONFIGURE_ENV:

1. python.elf's LIBS is extended with a --whole-archive ... --no-whole-archive group containing libsqlite3.a, libz.a, libbz2.a, liblzma.a. The four libs now live exactly once in python.elf with every public symbol exported via --export-dynamic.

2. The per-module LIBS env vars used by cpython's configure to inject -l<lib> into individual .so links are cleared (ZLIB_LIBS="", BZIP2_LIBS="", LIBLZMA_LIBS="", LIBSQLITE3_LIBS=""). The .so files now reference compress2 / BZ2_bzCompress / lzma_code / sqlite3_exec etc. as UND symbols; the dynamic loader resolves them at dlopen time against python.elf's .dynsym.

Correctness (not a libm-style issue)

Unlike the libm case that required nanvix#26 (Rust compiler_builtins shadows libm's WEAK HIDDEN symbols, breaking visibility merge), the Phase 3 bundled-lib duplication is a pure size issue:

• All four Group A libs (libz, libbz2, liblzma, libsqlite3) and the three Group B libs (libssl, libcrypto, libffi) are pure C with no Rust compiler_builtins shadows.
• None carry shared mutable per-process state in the way libm does. The Group B libs DO carry init state (OpenSSL algorithm registry, libffi closure caches) but each .so copy initializes its own copy — wasteful but not incorrect.

For OpenSSL specifically, each .so copy initializes its own OpenSSL state. Observable side-effect today: _hashlib.openssl_sha256() called BEFORE any other OpenSSL init returns "unsupported hash type sha256"; calls from regrtest's test_hashlib succeed because OpenSSL has been initialized via _ssl by then. The deferred Group B unbundling fixes this for free by ensuring exactly one libcrypto init runs in python.elf before any module dlopens.

Size impact

Metric	Before (Phase 2)	After (this PR)	Delta
python.elf	16.67 MB	9.50 MB	−7.17 MB (largest single-phase reduction)
_ssl.so + _hashlib.so + _ctypes.so (still bundled)	0 (were static)	~12 MB	+12 MB
Group A .so total	0 (were static)	~840 KB	+840 KB
Total ramfs			+~5.7 MB on disk; per-process the .so files are loaded on demand

When the Group B unbundling lands, the _ssl.so / _hashlib.so / _ctypes.so will shrink to ~30/30/80 KB each, with libssl/libcrypto/libffi added to python.elf once. Net ramfs at that point will be substantially smaller than today's all-static baseline.

Test coverage

New phase3_snippet in .nanvix/test.py imports each module, asserts non-builtin status, exercises one trivial API call (zlib.crc32, _bz2.BZ2Compressor, _hashlib.openssl_sha256 or _hashlib.new, _sqlite3.connect, _ctypes.dlopen, etc.), and prints __file__. Phase 1A/1B/1C/2 probes retained.

The _hashlib probe is intentionally lenient (hasattr(m, 'openssl_sha256') or hasattr(m, 'new')) because actually computing a hash from a freshly-loaded .so hits the OpenSSL init-order issue described above. Real hash functionality is exercised by regrtest's test_hashlib, which passes 160/160.

Validation

Tested on phase0-llfix toolchain overlay (containing the patched libposix.a from esaurez/nanvix#26):

• All 7 .so files installed under lib-dynload/.
• nm python.elf no longer shows PyInit_<name> for any of the 7.
• python.elf size: 16.67 MB (Phase 2) → 9.50 MB (Phase 3), −7.17 MB.
• Full regrtest 160/160 modules PASS, including test_hashlib, test_ssl, test_zlib, test_bz2, test_lzma, test_sqlite3, test_ctypes — they exercise the dlopen-resolved symbols end-to-end (real compression/decompression and SQL).
• All Phase 1A/1B/1C/2/3 import probes pass.
• Hello + lxml + HTTP server smoke pass.

Prerequisites

Stacked on Phase 2 (esaurez/cpython#9). Requires esaurez/nanvix#26 merged AND the toolchain image rebuilt — same as PR #7.

Risk

Mechanical configuration change — 7 entries added to *shared* block of Setup.local, 4 sysroot .a paths added to --whole-archive in Makefile.nanvix, 4 per-module LIBS env vars cleared. All tests verify behavior is unchanged.

Cumulative status after this PR

Phases 1 + 2 + 3 combined: 47 of 47 Tier-1/2/3 modules moved from static to .so. python.elf has dropped from ~20 MB at Phase 0 baseline to 9.50 MB now.