[nanvix] E: Phase 4 — build lxml C extensions as .so (unbundled)

[esaurez]

Summary

Phase 4 of the .a → .so migration (roadmap). Switches the lxml extension chain from being statically bundled into python.elf to being loaded at runtime via dlopen of liblxml_etree.so, with a proper DT_NEEDED chain pulling in libxslt.so → libexslt.so → libxml2.so transitively.

Architecture

_lxml_etree.cpython-312.so          ~10 KB shim   (dlopen + dlsym PyInit_etree)
  └─> liblxml_etree.so              1.7 MB        (lxml Cython output)
        ├─> libxslt.so              296 KB
        │     └─> libxml2.so        1.5 MB        (libz embedded)
        └─> libexslt.so             92 KB
              ├─> libxslt.so        (shared with above)
              └─> libxml2.so        (shared with above)

The shim modules (Modules/lxml_etree_builtin.c, Modules/lxml_elementpath_builtin.c) are tiny C wrappers that:

1. dlopen(RTLD_NOW | RTLD_GLOBAL) the underlying .so from /lib/python3.12/lib-dynload/.
2. dlsym PyInit_etree / PyInit__elementpath.
3. Forward the call. Leak the handle intentionally (process-lifetime); dlclose only on the failure path.

The Setup.local entries (*shared* block) consume only the shim .c files — no -L/-l flags, no --whole-archive, no MODLIBS piggyback. All resolution happens at dlopen time against python.elf's .dynsym (which already exports the full POSIX/libc/libm runtime via PR #1).

Size impact

Artifact	Before (Phase-4 static)	After (Phase-4 dlopen)
python.elf	12.26 MB	9.98 MB
_lxml_etree.cpython-312.so	3.78 MB (vendored copy)	10 KB shim
liblxml_etree.so	(none)	1.7 MB (shared, on-disk)

~2.3 MB reclaimed from the binary; the lxml code now ships once on disk rather than being duplicated per-process.

What changed

File	Change
Modules/lxml_etree_builtin.c	Rewrote as dlopen-shim (was --whole-archive-embedded module). Leaks handle on success, dlclose on dlsym failure.
Modules/lxml_elementpath_builtin.c	Same pattern for _elementpath.
.nanvix/lxml.py	_SETUP_LOCAL_TEMPLATE rewritten to bare _lxml_etree lxml_etree_builtin.c (no link flags). generate_setup_local keeps its signature for source-compat with build.py.
.nanvix/docker.py	_generate_setup_local_cmd mirrors lxml.py — _nanvix line no longer carries MODLIBS-piggyback flags.
.nanvix/package.py	Stages liblxml_etree.so, liblxml_elementpath.so, libxslt.so, libexslt.so, libxml2.so into the release tarball's lib/python3.12/lib-dynload/. Hard-fails with FileNotFoundError if any are missing.
.nanvix/test.py	Same staging + hard-fail for the test ramfs.

Validation

End-to-end test on nanvix-dev:

STEP_1:python_started (3, 12, 3)
STEP_2:about to import _lxml_etree (chain: libxslt+libexslt+libxml2)
STEP_3:_lxml_etree imported /lib/python3.12/lib-dynload/_lxml_etree.cpython-312.so
STEP_4:about to import lxml.etree
STEP_5:lxml.etree imported /lib/python3.12/lib-dynload/_lxml_etree.cpython-312.so
STEP_6:parsed XML, root= root child= lxml-ok
LXML_CHAIN_PASS

The 3-deep DT_NEEDED chain plus the diamond at libexslt.so → {libxslt, libxml2} ∪ liblxml_etree.so → {libxslt, libexslt, libxml2} resolves cleanly thanks to the loader fixes in esaurez/nanvix#27 and esaurez/nanvix#28.

Independent regrtest run: 122 / 185 PASS, 0 regression vs the pre-rework baseline. Remaining failures (44 ERROR + 19 TIMEOUT) are all pre-existing Nanvix issues independent of this PR (see nanvix-todo/cpython-libregrtest-compile-time-import.md and the unittest-teardown hang).

Code review

A code-review pass found three issues — all fixed in the commit pushed here:

1. (Critical) .nanvix/lxml.py's template still carried the old MODLIBS-piggyback -L/-l flags, which would silently break local (non-docker) builds.
2. (Medium) dlopen handle was leaked on the dlsym-failure path; now dlclose'd.
3. (Medium) test.py / package.py previously printed WARNING and continued when required .so files were missing, producing silently broken artifacts. Now they raise FileNotFoundError.

Runtime dependencies (must ship together)

• [syscall] E: Run dlopen ctors/dtors and DT_RUNPATH nanvix#27 — [syscall] E: Run dlopen ctors/dtors and DT_RUNPATH. Required for the lxml chain to initialize (Cython modules rely on .init_array).
• [syscall] B: Fix diamond DT_NEEDED handling nanvix#28 — [syscall] B: Fix diamond DT_NEEDED handling. The lxml chain forms a diamond at libexslt and liblxml_etree; without this fix dlopen deadlocks or double-loads.
• [build] E: Build libxml2.so alongside libxml2.a libxml2#1 — produces libxml2.so (bottom of the chain).
• [build] E: Build libxslt.so and libexslt.so libxslt#1 — produces libxslt.so + libexslt.so with DT_NEEDED libxml2.so.
• [build] E: Build lxml C extensions as .so lxml#1 — produces liblxml_etree.so + liblxml_elementpath.so with full DT_NEEDED chain.

Build-time dependencies (in this repo)

• [nanvix] E: Embed C/C++ runtime in python.elf for .so dlopen support #1 — embeds the full C/C++ runtime in python.elf and re-exports via --export-dynamic. The dlopen'd .so chain leaves libc/libm/libposix/libstdc++ symbols undefined and binds them against python.elf at load time.
• [nanvix] E: Link libnvx_crt0.a into python.elf #2 — libnvx_crt0.a startup (required for any cpython build with the current toolchain).
• [nanvix] E: Bake ninja and Cython into toolchain image #4 — bakes ninja and Cython into the toolchain image (required by the lxml port-lib builds).

Sequenced rollout

1. Merge [syscall] E: Run dlopen ctors/dtors and DT_RUNPATH nanvix#27 and [syscall] B: Fix diamond DT_NEEDED handling nanvix#28 → cut a nanvix release.
2. Merge [build] E: Build libxml2.so alongside libxml2.a libxml2#1 → cut release.
3. Bump [build] E: Build libxslt.so and libexslt.so libxslt#1's pin to the new libxml2 → merge → cut release.
4. Bump [build] E: Build lxml C extensions as .so lxml#1's pins to the new libxml2 + libxslt → merge → cut release.
5. Bump this PR's pin to the new lxml release → merge.

Until step 5, this repo's CI can still produce a working python.elf against the existing static-only release tarballs (the shim modules degrade to dlopen-fails-at-import, which is a self-contained failure rather than a build break).