[nanvix] E: Build stdlib extensions as .so by esaurez · Pull Request #18 · esaurez/cpython

esaurez · 2026-06-09T22:42:31Z

Summary

Builds 39 CPython stdlib extension modules as runtime-loaded .so files under lib/python3.12/lib-dynload/<name>.cpython-312.so instead of statically linking them into python.elf. Continues the work started by feat/phase0-array-so (the array proof of concept), generalising the *shared* Setup.local flow across the cpython stdlib.

Base branch: feat/phase0-array-so — this PR will be filed against nanvix/cpython with nanvix/cpython#690 as its base. Local tip 2b881a11521 matches that PR's HEAD exactly.

Why

Before this change, every cpython extension that Nanvix builds was forced into *static* in Modules/Setup.local, so the entire stdlib extension surface was linked into a monolithic python.elf. lib-dynload/ was empty. Side effects:

python.elf carried every extension's code even when a workload uses only a fraction of it.
Third-party C extensions could not be loaded via dlopen() at run time because the cpython build was not actually producing extension .so files. This blocked any out-of-tree extension story.
Stdlib extensions that bundle their own copies of vendored C libraries (_decimal/libmpdec, pyexpat/libexpat, the SHA2 hash module / libHacl_Hash_SHA2) re-paid the bundling cost per extension.

This PR enables the dlopen flow for the stdlib by making 39 modules *shared* — matching upstream cpython's default ./configure behavior on Linux.

What changed

39 stdlib modules are migrated from *static* (linked into python.elf) to *shared* (built as lib/python3.12/lib-dynload/<name>.cpython-312.so):

Data primitives, no external deps: _bisect, _heapq, _struct, _random, _opcode, _queue, _csv, binascii, _json, _pickle, _zoneinfo.
Math and memory, libm symbols resolved against python.elf: math, cmath, _statistics, mmap, _contextvars.
Text codecs, no external deps: unicodedata, _multibytecodec, _codecs_cn, _codecs_hk, _codecs_iso2022, _codecs_jp, _codecs_kr, _codecs_tw.
With bundled-in-cpython C deps, each .so bundles its own .a: _asyncio, _datetime, _decimal, pyexpat, _elementtree, _md5, _sha1, _sha2, _sha3, _blake2, select, _socket, _posixsubprocess, fcntl, termios.

Architecture

For modules with bundled-in-cpython C deps (_decimal, pyexpat, _sha2), each .so bundles its own copy of the vendored .a — exactly the behavior of cpython's default ./configure run on Linux without --with-system-libmpdec / --with-system-expat:

Module	Bundled `.a`	How it gets in
`_decimal.cpython-312.so`	`libmpdec.a`	cpython's configure sets `MODULE__DECIMAL_LDFLAGS=-lm $(LIBMPDEC_A)` automatically
`pyexpat.cpython-312.so`	`libexpat.a`	cpython's configure sets `MODULE_PYEXPAT_LDFLAGS=-lm $(LIBEXPAT_A)` automatically
`_sha2.cpython-312.so`	`libHacl_Hash_SHA2.a`	the `_sha2` line in `Setup.local` references the `.a` explicitly, matching upstream `Modules/Setup.stdlib.in:81`
`_elementtree.cpython-312.so`	(none — calls into `pyexpat` via `PyExpat_CAPI` capsule)	upstream pattern: `_elementtree` `import pyexpat` and pulls a `PyCapsule` of function pointers, so libexpat lives in `pyexpat.so` only

mpdec, expat, and libHacl_Hash_SHA2 are stateless C APIs with at most one consumer each. Duplicating their code into python.elf would buy nothing — none of them maintain process-global state that needs to be shared across multiple loaded extensions.

--with-libm= is cleared so libm symbols stay in python.elf — math / cmath resolve them via the existing --whole-archive of libm.a in LIBNVX_CRT0 at dlopen time.

Modules without external deps:
  <module>.cpython-312.so
      └── UND symbols → resolved against python.elf .dynsym at dlopen
                         (--export-dynamic on python.elf LDFLAGS)

Modules with bundled-in-cpython C deps:
  _decimal.cpython-312.so  (bundles libmpdec.a; no external symbol deps)
  pyexpat.cpython-312.so   (bundles libexpat.a; no external symbol deps)
  _sha2.cpython-312.so     (bundles libHacl_Hash_SHA2.a)
  _elementtree.cpython-312.so (uses pyexpat's PyExpat_CAPI capsule)

Mechanics

File	Change
`.nanvix/setup_local.py` (new)	`SETUP_LOCAL_ENTRIES` data table + `render_setup_local()` — single source of truth for `Modules/Setup.local` body, consumed by both the host (`.nanvix/lxml.py`) and Docker (`.nanvix/docker.py`) build paths.
`.nanvix/lxml.py`	`generate_setup_local()` renders via `setup_local.render_setup_local()`.
`.nanvix/docker.py`	`_generate_setup_local_cmd()` renders via the same helper and single-quotes each line for a `printf '%s\n' ... > Setup.local` invocation inside the container.
`.nanvix/test.py`	`_SO_MODULE_SANITY_CHECKS` table + `_render_so_sanity_snippets()` generator emit smoke-test snippets that exercise every migrated module via `import` + a trivial method call, and assert the module is no longer in `sys.builtin_module_names`.
`Makefile.nanvix`	Sets `--with-libm=` (empty) so libm symbols stay in `python.elf`. No other configure-time or post-configure changes are needed; cpython's default `MODULE__DECIMAL_LDFLAGS` / `MODULE_PYEXPAT_LDFLAGS` handle the bundling automatically.

Dependencies

Base branch: feat/phase0-array-so (the array Phase 0 proof of concept).

Runtime dependencies (already merged upstream):

nanvix/nanvix#2472 — libm visibility fix. Required so math.so / cmath.so can resolve libm symbols against python.elf .dynsym at dlopen time.
nanvix/nanvix#2473 — dlfcn init-array + DT_RUNPATH support. Required so dlopen() runs initialisers for the new .so modules and the loader can find dependencies.

esaurez · 2026-06-12T18:55:51Z

 {
    static char buf[32];
-    DWORD dw = GetLastError(); 
+    DWORD dw = GetLastError();


Let's create a separate PR to fix this

esaurez · 2026-06-12T18:56:26Z

 };

-static struct cffi_tls_s *get_cffi_tls(void);   /* in misc_thread_posix.h 
+static struct cffi_tls_s *get_cffi_tls(void);   /* in misc_thread_posix.h


This change also belong to the same cleanup separate PR, as well as the change in line 315 in this file.

esaurez · 2026-06-12T18:56:40Z

        return x;
    }
-    /* this hack is for Python 3.5, and also to give a more 
+    /* this hack is for Python 3.5, and also to give a more


Another change to the separate PR for cleaning

esaurez · 2026-06-12T18:56:55Z


 #if PY_MAJOR_VERSION >= 3
-    /* add manually 'module_name' in sys.modules: it seems that 
+    /* add manually 'module_name' in sys.modules: it seems that


Another change for the separate PR for cleaning

esaurez · 2026-06-12T18:57:28Z

        return PyInt_AS_LONG(ob);
    }
-    else 
+    else


All changes in this fil should be in the separate PR for cleaning

esaurez · 2026-06-12T22:58:53Z

+# Kept as a single shell statement so it embeds cleanly inside `sh -c
+# '...'` and inside a regular recipe line without backslash-continuation
+# gymnastics.
+NANVIX_VISIBILITY_DEFAULT_PATCH = \


is this the cleanest way to patch it? Is there a more fundamental way to achieve the same?

Builds 39 CPython stdlib extension modules as runtime-loaded .so files under lib/python3.12/lib-dynload/<name>.cpython-312.so instead of statically linking them into python.elf. Generalises the *shared* Setup.local flow first proven by feat/phase0-array-so. Modules built as .so -------------------- - Data primitives (no external deps): _bisect, _heapq, _struct, _random, _opcode, _queue, _csv, binascii, _json, _pickle, _zoneinfo. - Math and memory (libm symbols resolved against python.elf): math, cmath, _statistics, mmap, _contextvars. - Text codecs (no external deps): unicodedata, _multibytecodec, _codecs_cn/hk/iso2022/jp/kr/tw. - With bundled-in-cpython C deps (each .so bundles its own .a, same as upstream cpython's default ./configure run): _asyncio, _datetime, _decimal, pyexpat, _elementtree, _md5, _sha1, _sha2, _sha3, _blake2, select, _socket, _posixsubprocess, fcntl, termios. For the modules with bundled-in-cpython C deps (_decimal, pyexpat, _sha2), each .so bundles its own copy of the vendored .a via cpython's normal MODULE_*_LDFLAGS machinery -- exactly the behavior of cpython's default ./configure run on Linux without --with-system-libmpdec / --with-system-expat. mpdec / expat / libHacl_Hash_SHA2 are stateless C APIs with at most one consumer each (pyexpat exposes a PyCapsule that _elementtree calls through, so libexpat lives in pyexpat.so only), so duplication into python.elf would buy nothing. --with-libm= is cleared so libm symbols stay in python.elf -- math / cmath resolve them via the existing --whole-archive of libm.a in LIBNVX_CRT0. Infrastructure -------------- - .nanvix/setup_local.py (new): SETUP_LOCAL_ENTRIES data table + render_setup_local() -- single source of truth for Modules/Setup.local, consumed by both the host (.nanvix/lxml.py) and Docker (.nanvix/docker.py) build paths. - .nanvix/test.py: _SO_MODULE_SANITY_CHECKS table + _render_so_sanity_snippets() emit smoke-test snippets that exercise every migrated module via import + a trivial method call and assert each module is no longer in sys.builtin_module_names. - Makefile.nanvix: --with-libm= cleared (Phase 1B math/cmath modules resolve libm symbols against python.elf .dynsym at dlopen time via the existing --whole-archive of libm.a in LIBNVX_CRT0). Runtime dependencies (already merged upstream) ---------------------------------------------- - nanvix/nanvix#2472 -- libm visibility fix. - nanvix/nanvix#2473 -- dlfcn init-array + DT_RUNPATH support. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…FLAGS When --with-system-libmpdec / --with-system-expat is *not* given, the bundled-library code paths in configure.ac (lines 3831, 3915) hardcoded a literal `-lm` in: LIBEXPAT_LDFLAGS="-lm $(LIBEXPAT_A)" LIBMPDEC_LDFLAGS="-lm $(LIBMPDEC_A)" On Linux this happens to work because LIBM defaults to `-lm`, so the result is a no-op. On Nanvix we pass `--with-libm=` (empty) to keep libm.a out of every Setup.local `.so` -- libm is whole-archived into python.elf instead. The literal `-lm` defeats that and bundles a full copy of libm.a into both `_decimal.so` and `pyexpat.so` -- about ~400 KB of redundant code per .so, plus a symbol-collision risk that forces --allow-multiple-definition at every other .so link. Switching the literal to `$(LIBM)` is equivalent on Linux (LIBM=-lm by default) and correctly drops the duplicate libm when LIBM is empty. Both `configure.ac` and the generated `configure` are patched in lockstep so no autoreconf is required. Bug surfaced by the cpython-on-Nanvix .so management audit; tracked for a follow-up upstream contribution to python/cpython once the Nanvix port stabilizes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

esaurez · 2026-06-13T01:45:10Z

Update: folded in audit finding B-1 as a new commit
(configure: use $(LIBM) instead of literal -lm in LIBMPDEC/LIBEXPAT LDFLAGS).

Without this fix, the stdlib .so files this PR introduces would ship
with a ~400 KB copy of libm.a bundled into both _decimal.so and
pyexpat.so -- the audit caught this via the literal -lm hardcoded
in configure.ac:3834,3918 instead of the $(LIBM) substitution that
the rest of the build system uses.

The patch is symmetric on Linux (LIBM=-lm by default there, so it
is a no-op) and correctly drops the duplicate libm on Nanvix (where
--with-libm= is passed empty so libm.a is whole-archived into
python.elf only).

Both configure.ac and the generated configure are patched in
lockstep (no autoreconf required). Tracked for a possible future
upstream contribution to python/cpython, but suitable to carry as
a downstream patch indefinitely.

Stack force-pushed to keep PRs #19 / #20 / #21 atop the new base.

esaurez mentioned this pull request Jun 9, 2026

[nanvix] E: Build _ssl/_hashlib/_ctypes as .so #19

Open

esaurez force-pushed the feat/wave5-pr-a-stdlib-so branch from 6b31bca to 86e1659 Compare June 9, 2026 22:55

esaurez changed the title ~~[nanvix] E: Build stdlib extensions as .so (Phases 1A/1B/1C/2)~~ [nanvix] E: Build stdlib extensions as .so Jun 9, 2026

esaurez force-pushed the feat/wave5-pr-a-stdlib-so branch 2 times, most recently from a78b0c8 to 2f602bf Compare June 10, 2026 02:32

esaurez commented Jun 12, 2026

View reviewed changes

esaurez force-pushed the feat/wave5-pr-a-stdlib-so branch from 64c14e9 to 6819a97 Compare June 12, 2026 23:52

esaurez mentioned this pull request Jun 16, 2026

[nanvix] E: Remove lxml built-in shim from cpython #22

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[nanvix] E: Build stdlib extensions as .so#18

[nanvix] E: Build stdlib extensions as .so#18
esaurez wants to merge 2 commits into
feat/phase0-array-sofrom
feat/wave5-pr-a-stdlib-so

esaurez commented Jun 9, 2026 •

edited

Loading

Uh oh!

esaurez Jun 12, 2026

Uh oh!

esaurez Jun 12, 2026

Uh oh!

esaurez Jun 12, 2026

Uh oh!

esaurez Jun 12, 2026

Uh oh!

esaurez Jun 12, 2026

Uh oh!

esaurez Jun 12, 2026

Uh oh!

esaurez commented Jun 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

esaurez commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why

What changed

Architecture

Mechanics

Dependencies

Uh oh!

esaurez Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

esaurez Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

esaurez Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

esaurez Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

esaurez Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

esaurez Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

esaurez commented Jun 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

esaurez commented Jun 9, 2026 •

edited

Loading