[nanvix] E: Unbundle libsqlite3 / libz / libbz2 / liblzma from Phase 3 .so#12
Closed
esaurez wants to merge 3 commits into
Closed
Conversation
Phase 3 of the .a -> .so migration (see nanvix-todo/cpython-static-to-shared-migration.md section 7). Promotes 7 modules with external Nanvix-ported .a dependencies from statically linked into python.elf to dlopen-loaded shared objects. Modules moved to *shared* in Modules/Setup.local generation (.nanvix/docker.py): - _bz2 (libbz2) - _lzma (liblzma) - zlib (libz) - _ssl (libssl + libcrypto) - _hashlib (libcrypto) - _sqlite3 (libsqlite3) - _ctypes (libffi) Same trade-off as Phase 2: each .so currently embeds its own copy of the underlying library, via the existing cpython per-module LDFLAGS mechanism. The .so files are larger than ideal but functionally complete: - _ssl.so: 6398 KB (libssl + libcrypto baked in) - _hashlib.so: 4795 KB (libcrypto baked in, duplicates _ssl's copy) - _sqlite3.so: 1448 KB (libsqlite3 baked in) - _ctypes.so: 744 KB (libffi baked in) - zlib.so: 236 KB, _lzma.so: 212 KB, _bz2.so: 128 KB Optimizing all 11 Phase 2 + Phase 3 modules to resolve bundled-lib symbols at dlopen time against python.elf's .dynsym is captured in nanvix-todo/phase2-3-unbundle-bundled-libs.md as a deferred follow-up PR. The optimization requires either a two-pass configure or linker-script-based approach, AND requires fixing two pre-existing sysroot bugs: 1. libffi.a contains nested archives (libposix.a, libc.a, libm.a) that block --whole-archive. Either fix Nanvix's libffi build or post-process to remove the nested members. 2. libcrypto.a contains bss_log.o that references POSIX openlog / syslog / closelog (not implemented in Nanvix's newlib + libposix). Either strip bss_log.o from libcrypto.a or provide stub implementations. Importantly, the bundled-lib duplication is a pure size issue, not a correctness issue (unlike the libm case that required nanvix#26): mpdec / expat / HACL / zlib / lzma / bz2 / sqlite / ffi are all pure C with no Rust compiler_builtins shadows, and none carry shared mutable per-process state. The visibility-merge correctness fix required for libm does NOT apply. For OpenSSL, each .so copy initializes its own copy of OpenSSL state (algorithm registry, BIO method list, RAND state). Observed side-effect: _hashlib.openssl_sha256() called BEFORE any other OpenSSL init returns "unsupported hash type sha256"; calls from regrtest's test_hashlib succeed because OpenSSL has been initialized via _ssl elsewhere by then. The deferred unbundling fixes this for free by ensuring exactly one libcrypto init runs in python.elf before any module dlopens. Validation on local toolchain (phase0-llfix): - All 7 .so files produced and installed under lib-dynload/. - nm python.elf no longer shows PyInit_<name> for any of the 7. - python.elf size: 16.67 MB (Phase 2) -> 11.20 MB (Phase 3), -5.47 MB. Largest single-phase reduction so far. - Hello + Phase 1A + 1B + 1C + 2 + 3 import probes + lxml + HTTP smoke + full regrtest 160/160 PASS in standalone mode. - test_hashlib (regrtest) computes real sha256 successfully via dlopen'd _hashlib, confirming OpenSSL works end-to-end (the init-order quirk only affects callers that hit OpenSSL before any other initialization). Phase 1 + Phase 2 + Phase 3 cumulative: 47 of 47 Tier-1/2/3 modules moved from static to .so. python.elf has dropped from ~20 MB at Phase 0 baseline to 11.2 MB now. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Phase 4 of the .a -> .so migration (see nanvix-todo/cpython-static-to-shared-migration.md section 8). Promotes the 2 lxml C extension modules (_lxml_etree, _lxml_elementpath) from statically linked into python.elf to dlopen-loaded shared objects. Modules moved from *static* to *shared* in Modules/Setup.local generation (.nanvix/docker.py): - _lxml_etree (links liblxml_etree.a + libxslt.a + libexslt.a + libxml2.a + libz.a from the sysroot) - _lxml_elementpath (links liblxml_elementpath.a + libxml2.a + libz.a) Both modules use the same thin-shim approach the static variant used: lxml_etree_builtin.c / lxml_elementpath_builtin.c each register the C extension's PyInit_ entrypoint after pulling in the Cython-generated implementation from liblxml_etree.a / liblxml_elementpath.a in the sysroot. The shim files are unchanged. The lxml/etree.py Python-level shim that re-exports symbols from `_lxml_etree` continues to work as-is, because Python's import system finds .cpython-312.so files in lib-dynload/ exactly like static modules — the import name `_lxml_etree` resolves either way. Same trade-off as Phase 2 and Phase 3: each .so embeds its own copy of the underlying libraries (libxslt, libexslt, libxml2, libz). _lxml_etree.so is 3645 KB primarily because libxslt and libxml2 are sizable. _lxml_elementpath.so is 175 KB because it only needs libxml2. This trade-off is tracked alongside the Phase 2 + Phase 3 bundled libs in nanvix-todo/phase2-3-unbundle-bundled-libs.md as a deferred follow-up optimization PR. Note that libz is already shared with Phase 3's zlib.so today (each carries its own copy); the same applies to libxml2 (only used by lxml; if libxml2 had to be in python.elf for other consumers we'd see the same visibility merge issue as libm, but it has no Rust compiler_builtins shadows so we're safe). Validation on local toolchain (phase0-llfix): - Both .so files produced and installed under lib-dynload/. - nm python.elf no longer shows PyInit__lxml_etree or PyInit__lxml_elementpath. - python.elf size: 11.20 MB (Phase 3) -> 8.48 MB (Phase 4), -2.72 MB. python.elf now meets the migration plan's "<=10 MB stripped" success criterion from Section 10. - Hello + all Phase 1/2/3 import probes + lxml import-and-parse smoke + HTTP server smoke + full regrtest 160/160 PASS in standalone mode. - test_nanvix_lxml (regrtest's lxml-specific test) included in the 160/160 and passes. This completes the migration plan's Phase 4 scope — only Phase 5 (optional dormant module revival for numpy / regex) remains. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…3 .so
Group A of the deferred Phase 2/3 unbundling work (see
nanvix-todo/phase2-3-unbundle-bundled-libs.md). Promotes 4 of the
Phase 3 sysroot-ported libraries from "duplicated into each .so" to
"single copy in python.elf, .so resolves at dlopen time".
Affected modules: zlib, _bz2, _lzma, _sqlite3 (and binascii, which
uses libz for crc32 via the same ZLIB_LIBS chain).
Changes in Makefile.nanvix CONFIGURE_ENV:
1. python.elf's LIBS is extended with a --whole-archive ...
--no-whole-archive group containing:
$(SYSROOT_PATH)/lib/libsqlite3.a
$(SYSROOT_PATH)/lib/libz.a
$(SYSROOT_PATH)/lib/libbz2.a
$(SYSROOT_PATH)/lib/liblzma.a
The four libs now live exactly once in python.elf, with every
public symbol exported via --export-dynamic.
2. The per-module LIBS env vars used by cpython's configure to
inject -l<lib> into individual .so links are cleared:
ZLIB_LIBS="" (also feeds BINASCII_LIBS via configure)
BZIP2_LIBS=""
LIBLZMA_LIBS=""
LIBSQLITE3_LIBS=""
This drops the underlying .a from each .so's link command. The
.so files now reference compress2 / BZ2_bzCompress / lzma_code /
sqlite3_exec etc. as UND symbols; the dynamic loader resolves
them at dlopen time against python.elf's .dynsym.
This is the same architectural pattern that Phase 1B-drop-libm (#7)
established for libm. It is now extended to the four "clean" sysroot
libs that have no pre-existing sysroot bugs.
NOT included in this PR:
- libcrypto/libssl (used by _ssl, _hashlib): blocked by
libcrypto.a containing bss_log.o which references unimplemented
POSIX openlog/syslog/closelog. To be addressed in a follow-up PR
that either strips bss_log.o from libcrypto.a or stubs syslog.
- libffi (used by _ctypes): blocked by libffi.a containing nested
archives (libposix.a, libc.a, libm.a as ar members) that ld
refuses to process under --whole-archive. To be addressed in a
follow-up PR that either fixes the libffi build or strips the
nested members.
- libmpdec / libexpat / libHacl_Hash_SHA2 (used by _decimal,
pyexpat, _sha2): blocked by the chicken-and-egg that these .a
files are built by cpython's own Makefile AFTER configure runs,
so they cannot be referenced in LIBS at configure conftest time.
To be addressed in a follow-up PR that uses either a two-pass
configure or a linker-script INPUT() approach.
- libxml2 / libxslt / libexslt / liblxml_etree / liblxml_elementpath
(used by _lxml_etree, _lxml_elementpath): same shape as Group A
but kept as a separate Group D PR to keep diffs small.
Validation on local toolchain (phase0-llfix overlay containing the
patched libposix.a from esaurez/nanvix#26):
- .so size reductions:
zlib.so 236 KB -> 181 KB (-55 KB)
_bz2.so 128 KB -> 68 KB (-60 KB)
_lzma.so 212 KB -> 120 KB (-92 KB)
_sqlite3.so 1448 KB -> 473 KB (-975 KB, biggest single win)
- python.elf size: 8.48 MB -> 9.50 MB (+1.02 MB)
(libsqlite3 + libz + libbz2 + liblzma now baked in once via
--whole-archive instead of duplicated into each .so).
- Net ramfs delta: ~150 KB savings, single-copy architecture
established for these four libs.
- Functional: full regrtest 160/160 modules pass, including
test_zlib, test_bz2, test_lzma, test_sqlite3 (real
compression/decompression and SQL operations exercise the
dlopen-resolved symbols end-to-end).
- All Phase 1A/1B/1C/2/3/4 import probes continue to pass.
- Hello + lxml + HTTP server smoke continue to pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
esaurez
force-pushed
the
feat/phase4-tier4-lxml-shared
branch
from
7c7e39f to
453b1e3
Compare
Owner
Author
|
Folded into PR #10 (Phase 3) per esaurez review preference — easier to review the .so move and the unbundling together. The squashed Phase 3 commit on eat/phase3-tier3-external-shared now contains both changes (4 modules unbundled with --whole-archive, 3 modules left bundled awaiting Group B sysroot fixes). See updated PR #10 description. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Group A of the deferred Phase 2/3 unbundling work (see
nanvix-todo/phase2-3-unbundle-bundled-libs.md). Promotes 4 of the Phase 3 sysroot-ported libraries from "duplicated into each.so" to "single copy inpython.elf,.soresolves at dlopen time".This is the same architectural pattern that Phase 1B-drop-libm (#7) established for
libm. Group A extends it to the four "clean" sysroot libs that have no pre-existing sysroot bugs.Affected modules
zlib,_bz2,_lzma,_sqlite3(andbinascii, which useslibzfor CRC32 via the sameZLIB_LIBSchain).Changes in
Makefile.nanvixCONFIGURE_ENVpython.elf's LIBS is extended with a--whole-archive ... --no-whole-archivegroup containinglibsqlite3.a,libz.a,libbz2.a,liblzma.a. The four libs now live exactly once inpython.elf, with every public symbol exported via--export-dynamic.Per-module
LIBSenv vars used by cpython's configure to inject-l<lib>into individual.solinks are cleared (ZLIB_LIBS="",BZIP2_LIBS="",LIBLZMA_LIBS="",LIBSQLITE3_LIBS=""). The.sofiles now referencecompress2/BZ2_bzCompress/lzma_code/sqlite3_execetc. asUNDsymbols; the dynamic loader resolves them at dlopen time againstpython.elf's.dynsym.NOT included in this PR (deferred to follow-ups)
libcrypto/libssl(used by_ssl,_hashlib): blocked bylibcrypto.acontainingbss_log.owhich references unimplemented POSIXopenlog/syslog/closelog.libffi(used by_ctypes): blocked bylibffi.acontaining nested archives (libposix.a,libc.a,libm.aas ar members).libmpdec/libexpat/libHacl_Hash_SHA2(used by_decimal,pyexpat,_sha2): blocked by chicken-and-egg at configure time (cpython builds these.afiles later).libxml2/libxslt/libexslt/liblxml_etree/liblxml_elementpath(used by_lxml_etree,_lxml_elementpath): kept as a separate Group D PR to keep diffs small.Size impact
zlib.so_bz2.so_lzma.so_sqlite3.sopython.elfThe bigger win is architectural: each library now exists exactly once in the image, matching the canonical Linux pattern (libm.so.6, libsqlite3.so.0, etc. are single instances).
Validation
Tested on
phase0-llfixtoolchain overlay (containing the patchedlibposix.afromesaurez/nanvix#26):test_zlib,test_bz2,test_lzma,test_sqlite3(real compression/decompression and SQL operations exercise the dlopen-resolved symbols end-to-end).Prerequisites
Stacked on Phase 4 (esaurez/cpython#11). Requires
esaurez/nanvix#26merged AND the toolchain image rebuilt — same as Phase 1B-drop-libm.Risk
Low. The change is mechanical: 4 sysroot
.apaths added to--whole-archive, 4 per-module LIBS env vars cleared. All tests verify behavior is unchanged. Reversible by undoing the Makefile.nanvix diff.