Skip to content

[nanvix] E: Build bz2/lzma/zlib/sqlite3 as DT_NEEDED .so chains#20

Open
esaurez wants to merge 1 commit into
feat/wave5-pr-b-unbundle-externalsfrom
feat/wave5-pr-c-unbundle-group-a
Open

[nanvix] E: Build bz2/lzma/zlib/sqlite3 as DT_NEEDED .so chains#20
esaurez wants to merge 1 commit into
feat/wave5-pr-b-unbundle-externalsfrom
feat/wave5-pr-c-unbundle-group-a

Conversation

@esaurez

@esaurez esaurez commented Jun 10, 2026

Copy link
Copy Markdown
Owner

Summary

Completes the external-library unbundling started in nanvix/cpython#738 by moving the remaining four stdlib extensions (_bz2, _lzma, zlib, _sqlite3) from being statically linked into python.elf to runtime-loaded .so files emitting DT_NEEDED for libbz2.so / liblzma.so / libz.so / libsqlite3.so.

After this PR, python.elf bundles no external Nanvix-ported libraries at all — every .so wrapper (_ssl, _hashlib, _ctypes, _sqlite3, zlib, _bz2, _lzma) consumes its underlying library through DT_NEEDED. This matches upstream cpython's default behavior on every Linux distro (system-library detection enabled).

Why

The previous PR (nanvix/cpython#738) took the libffi / openssl / lxml consumers to DT_NEEDED .so chains but left these four extensions statically linked into python.elf because the Nanvix port repos for bzip2 / zlib / xz / sqlite did not yet ship .so builds. The four port-repo PRs listed under Dependencies below add those .so builds, which finally lets this PR finish the migration.

What changed

4 stdlib modules migrated from *static* (linked into python.elf) to *shared* (loaded via dlopen). The corresponding LIBSQLITE3_LIBS / ZLIB_LIBS / BZIP2_LIBS / LIBLZMA_LIBS env vars already use -L/-l form in nanvix/cpython#738 (no change needed there); cpython's normal MODULE_*_LDFLAGS machinery wires them into each .so's link line as DT_NEEDED.

DT_NEEDED .so chains added by this PR:
  _bz2.cpython-312.so       -> libbz2.so
  _lzma.cpython-312.so      -> liblzma.so
  zlib.cpython-312.so       -> libz.so
  _sqlite3.cpython-312.so   -> libsqlite3.so

The Nanvix dynamic loader walks each DT_NEEDED chain at dlopen() time, and UND symbols in any of those .so files bind to python.elf .dynsym.

Mechanics

File Change
.nanvix/setup_local.py Adds 4 *shared* entries (_bz2, _lzma, zlib, _sqlite3).
Makefile.nanvix Drops -lsqlite3 -lz -lbz2 -llzma from python.elf LIBS; the -Wl,--start-group ... -Wl,--end-group block is gone. Per-module *_LIBS env vars unchanged (already in -L/-l form from nanvix/cpython#738).
.nanvix/runtime_sos.py REQUIRED_RUNTIME_SOS gains libbz2.so / liblzma.so / libz.so / libsqlite3.so so .nanvix/test.py and .nanvix/package.py stage them into sysroot/lib/ at run time.
.nanvix/z.py _DEP_EXPECTED_LIBS for bzip2, zlib, sqlite, xz extended with the .so siblings so ./z setup validates that the new shared-library port builds installed correctly into buildroot/lib.
.nanvix/test.py Extends CPYTHON_TEST_EXTERNAL_DEPS smoke test with the 4 new modules.

Dependencies

Base branch: nanvix/cpython#738.

Runtime dependencies — the four port-repo PRs that produce the .so files this PR consumes:

These four PRs must land and their port-lib releases must be pinned in cpython's nanvix.toml before this PR is mergeable upstream.

Also inherits the runtime deps already required by nanvix/cpython#738:

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

@esaurez esaurez force-pushed the feat/wave5-pr-b-unbundle-externals branch from 9119925 to cffb2a8 Compare June 10, 2026 02:34
@esaurez esaurez force-pushed the feat/wave5-pr-c-unbundle-group-a branch from 176e8cb to ab96af7 Compare June 10, 2026 02:34
@esaurez esaurez force-pushed the feat/wave5-pr-b-unbundle-externals branch from cffb2a8 to a196813 Compare June 10, 2026 03:03
@esaurez esaurez force-pushed the feat/wave5-pr-c-unbundle-group-a branch from ab96af7 to 2b4aaff Compare June 10, 2026 03:03
@esaurez esaurez force-pushed the feat/wave5-pr-b-unbundle-externals branch from a196813 to c405ce5 Compare June 12, 2026 23:54
@esaurez esaurez force-pushed the feat/wave5-pr-c-unbundle-group-a branch from 2b4aaff to c830f95 Compare June 12, 2026 23:54
@esaurez esaurez force-pushed the feat/wave5-pr-b-unbundle-externals branch from c405ce5 to b686ee5 Compare June 13, 2026 00:06
@esaurez esaurez force-pushed the feat/wave5-pr-c-unbundle-group-a branch from c830f95 to 8f420b1 Compare June 13, 2026 00:06
@esaurez esaurez changed the title [nanvix] E: Switch bz2/lzma/zlib/sqlite3 to DT_NEEDED .so chains [nanvix] E: Build bz2/lzma/zlib/sqlite3 as DT_NEEDED .so chains Jun 13, 2026
@esaurez esaurez force-pushed the feat/wave5-pr-b-unbundle-externals branch from b686ee5 to 3e503b9 Compare June 13, 2026 00:08
@esaurez esaurez force-pushed the feat/wave5-pr-c-unbundle-group-a branch from 8f420b1 to 712e51a Compare June 13, 2026 00:08
@esaurez esaurez force-pushed the feat/wave5-pr-b-unbundle-externals branch from 3e503b9 to c96a209 Compare June 13, 2026 01:44
@esaurez esaurez force-pushed the feat/wave5-pr-c-unbundle-group-a branch from 712e51a to cf8e25c Compare June 13, 2026 01:44
Completes the external-library unbundling started in the parent PR by
moving the remaining four stdlib extensions (_bz2, _lzma, zlib,
_sqlite3) from being statically linked into python.elf to runtime-
loaded .so files that emit DT_NEEDED for libbz2.so / liblzma.so /
libz.so / libsqlite3.so.

After this PR, python.elf bundles no external Nanvix-ported libraries
at all -- every .so wrapper (_ssl, _hashlib, _ctypes, _sqlite3, zlib,
_bz2, _lzma) consumes its underlying library through DT_NEEDED. This
matches upstream cpython's default behavior on every Linux distro
(system-library detection enabled).

Changes
-------
- .nanvix/setup_local.py: add 4 *shared* entries (_bz2, _lzma, zlib,
  _sqlite3). The corresponding LIBSQLITE3_LIBS / ZLIB_LIBS /
  BZIP2_LIBS / LIBLZMA_LIBS already use -L/-l form (no change needed
  there); cpython's normal MODULE_*_LDFLAGS machinery wires them into
  each .so's link line as DT_NEEDED.

- Makefile.nanvix: drop -lsqlite3 -lz -lbz2 -llzma from python.elf's
  LIBS (those symbols come from the standalone .so files now). LIBS
  segment 2 becomes empty.

- .nanvix/runtime_sos.py: REQUIRED_RUNTIME_SOS gains libbz2.so /
  liblzma.so / libz.so / libsqlite3.so so .nanvix/test.py and
  .nanvix/package.py stage them into sysroot/lib/ at run time.

- .nanvix/z.py: _DEP_EXPECTED_LIBS for bzip2 / zlib / sqlite / xz
  extended with the .so siblings so `./z setup` validates that the
  new shared-library port builds installed correctly into
  buildroot/lib.

- .nanvix/test.py: extend CPYTHON_TEST_EXTERNAL_DEPS smoke test with
  the 4 new modules.

Runtime dependencies
--------------------
- nanvix/nanvix#2472 -- libm visibility fix (inherited from PR-A).
- nanvix/nanvix#2473 -- dlfcn init-array + DT_RUNPATH support
  (inherited from PR-B).

The 4 new .so files this PR consumes come from the Wave 6 port-repo
PRs that must land first:
- esaurez/bzip2#2 -- libbz2.so build target
- esaurez/zlib#1 -- libz.so build target
- esaurez/xz#1 -- liblzma.so build target
- esaurez/sqlite#1 -- libsqlite3.so build target

These four port-lib releases must be pinned in cpython's nanvix.toml
before this PR is mergeable upstream.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant