Add the option to compile key modules to C using mypyc by chadrik · Pull Request #2134 · AcademySoftwareFoundation/rez

chadrik · 2026-06-15T05:41:51Z

Here is an interesting payoff from my recent type annotations PR that was merged: we can now use mypyc to compile key libraries into C for added performance.

Running the benchmarks locally I see a speedup of about ~2.6x.

Notes:

Successfully building rez as a C-library requires a custom build of mypyc that fixes several issues encountered during this project.
Most of the changes in this PR are safe typing changes that could be split off into a separate PR to release first
This PR changed the lint job to use the latest mypy because it needs to match the version used for building with mypyc, and there are a number of fixes in the newer versions. The primary ramification of this is that mypy won't flag errors if the code uses language features that are not compatible with python 3.8 or 3.9. I think these same issues should be flagged by flake8, though. The other effect is that mypyc can't compile rez for python 3.8 or 3.9.
One fundamental change is that the schemas are now used to generate accessor functions in codegen step, rather than using dynamic lookups -- this is a prerequisite for fast performance.
I have some additional modifications that eke out some more performance, but they're a bit uglier, so I will save them for a followup

Obviously, if this gets released we'll need a plan for gradual deployment/adoption in order to release something this radical. Gradual adoption is one of the big advantages of using something like mypyc over rewriting specific modules in rust: the business logic and API interface remains the same between pure python and C-extension, compilation is just a installation setting. Here are some thoughts:

We should scour the diff looking for edits that might introduce breaking changes. I believe they're minimal, but we should document them all.
Any initial release of this should be optional at first. Advanced users who build their own rez can begin opting into it, while tests continue to enforce that it works.

codecov · 2026-06-15T15:47:39Z

Codecov Report

❌ Patch coverage is 72.54174% with 444 lines in your changes missing coverage. Please review.
✅ Project coverage is 61.44%. Comparing base (6654418) to head (f19c6f2).

Files with missing lines	Patch %	Lines
src/rez/utils/data_utils.py	26.42%	177 Missing and 4 partials ⚠️
src/rez/resolved_context.py	57.57%	34 Missing and 8 partials ⚠️
src/rez/packages.py	83.11%	37 Missing and 2 partials ⚠️
src/rez/package_resources.py	88.88%	36 Missing ⚠️
src/rez/utils/resources.py	82.90%	11 Missing and 9 partials ⚠️
src/rez/solver.py	78.37%	11 Missing and 5 partials ⚠️
src/rez/package_order_list.py	75.00%	12 Missing and 3 partials ⚠️
src/rezplugins/package_repository/filesystem.py	90.06%	14 Missing and 1 partial ⚠️
src/rez/package_order.py	75.55%	7 Missing and 4 partials ⚠️
src/rez/resolver.py	35.29%	11 Missing ⚠️
... and 22 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2134      +/-   ##
==========================================
+ Coverage   61.32%   61.44%   +0.11%     
==========================================
  Files         164      166       +2     
  Lines       20568    21528     +960     
  Branches     3575     3689     +114     
==========================================
+ Hits        12613    13227     +614     
- Misses       7084     7386     +302     
- Partials      871      915      +44

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

chadrik · 2026-06-15T15:42:57Z

+
+
+def get_latest_package(name: str, *, range_: VersionRange | None = None,
+                       paths: list[str] | None = None,


This is a breaking change: range_, paths, and error are now keyword-only arguments. You can no longer pass values by position. This was done because the alternative is a very long list of overloads covering all the permutations of positional and keyword scenarios.

Signed-off-by: Chad Dombrova <chadrik@gmail.com>

The bulk of the mypyc port: replace custom metaclasses with __init_subclass__ plus explicitly written-out resource members, migrate from the home-grown cached_property to functools.cached_property, add mypyc_attr directives, native-class type-narrowing asserts/casts, split PackageOrderList out into package_order_list.py, and assorted runtime fixes. Signed-off-by: Chad Dombrova <chadrik@gmail.com>

Add a 'mypyc' axis to the installation and tests workflows so CI exercises both the pure-python build (REZ_MYPYC=0) and the mypyc-compiled build (REZ_MYPYC=1) across the full OS x Python matrix. fail-fast is off, so each configuration reports independently and we can see exactly which ones the compiled build supports. Point the pyproject build requirement at our mypy fork (github.com/chadrik/mypy@rez-mypyc-fixes), which carries the fixes needed to compile rez. This makes an isolated build (pip install ., install.py, ...) automatically pick up the right mypy when building with mypyc, with no extra CI steps. Verified end to end: an isolated REZ_MYPYC=1 build pulls the fork and produces the compiled extensions. Signed-off-by: Chad Dombrova <chadrik@gmail.com>

Bump the mypy used by the static-analysis workflow from 1.14.1 to 2.1.0, the version our mypyc fork is based on, so type-checking matches what the compiled build sees. mypy 2.1 requires the type-check target (tool.mypy python_version) to be 3.10+, so it can no longer be rez's lowest supported runtime (3.8); set it to 3.10. Regenerate mypy-baseline.txt against mypy 2.1.0 (the error set differs from 1.14.1: 594 -> 234 baselined entries). Verified the baseline keeps the mypy workflow green (mypy | mypy-baseline filter reports no new errors). Signed-off-by: Chad Dombrova <chadrik@gmail.com>

Add a mypyc=true/false matrix axis to the benchmark workflow so the resolving benchmark runs for both the compiled and pure-python builds. mypyc requires Python >= 3.10, so the run moves from 3.7 to 3.10 and both variants run there for an apples-to-apples comparison. REZ_MYPYC is wired into the install step, and result artifacts are named per-variant. The store job keeps the pure (mypyc=false) result as the canonical baseline, since store_benchmark.py keys results by python+rez version only. Signed-off-by: Chad Dombrova <chadrik@gmail.com>

The @atexit.register decorator on _atexit (and the atexit import) had been commented out during the mypyc work. util.py is not compiled, so there is no reason to disable it; restore it so the tmpdir cleanup runs at exit as it does on main. Signed-off-by: Chad Dombrova <chadrik@gmail.com>

self.vcs is Optional. The mypyc work added an `if self.vcs:` guard around the changelog lookup to satisfy the type checker. Confirm it is also safe at runtime: get_release_data only reaches get_changelog after an early vcs-is-None return, and the rez-release caller already catches and nulls a failed changelog. Record that reasoning in a comment so the guard is not mistaken for an unexplained behaviour change. Signed-off-by: Chad Dombrova <chadrik@gmail.com>

range_, paths and error became keyword-only during the typing work (the error value determines the return type, and covering every positional and keyword permutation with overloads is impractical). This is a breaking change, so record it and the rationale in the docstring rather than leaving it as an unannounced signature change. Signed-off-by: Chad Dombrova <chadrik@gmail.com>

The auto-generated Package.version accessor had been hand-corrected from the schema-derived 'Version | None' to 'Version': a package always exposes a Version, but 'version' is Optional in the pod schema so the generator emitted 'Version | None'. Hand-editing generated source is fragile (it is wiped on regeneration), so add a declarative FORWARDER_TYPE_OVERRIDES map to the generator and register ("Package", "version") -> "Version". The generator now emits the correct type itself, and the manual-correction note is removed. Signed-off-by: Chad Dombrova <chadrik@gmail.com>

ResolverStatus stores a human-readable description as the first (and only) element of each member's tuple value, but the accessor for it was left as a commented-out Enum __init__. Expose it as a `description` property returning value[0] instead, and drop the dead __init__ stub. Signed-off-by: Chad Dombrova <chadrik@gmail.com>

Mirror ResolverStatus: SolverStatus members carry their human-readable description as the first element of their tuple value. Expose it via a `description` property so callers can read it without reaching into .value. Signed-off-by: Chad Dombrova <chadrik@gmail.com>

The @pool_memcached_connections decorator had been removed from iter_package_families and iter_packages during the mypyc work. Both are generators, and mypyc cannot compile a decorated generator method that overrides a base-class method: it raises KeyError in handle_ext_method while reconciling the override signature. That is why the decorator was dropped. Restore the pooling in a mypyc-compatible way by inlining a memcached_client() context manager around each generator body, so a single client is pooled for the whole iteration while the method stays a plain, compilable generator. Verified by a full mypyc build plus selftest. Signed-off-by: Chad Dombrova <chadrik@gmail.com>

chadrik requested a review from a team as a code owner June 15, 2026 05:41

chadrik force-pushed the mypyc branch from a3196ca to f7f9409 Compare June 15, 2026 15:26

chadrik commented Jun 15, 2026

View reviewed changes

Add use of Generic

e8a9f42

Signed-off-by: Chad Dombrova <chadrik@gmail.com>

chadrik force-pushed the mypyc branch 7 times, most recently from 77f594b to adfd9e0 Compare June 16, 2026 01:45

chadrik added 8 commits June 15, 2026 19:22

Safe code changes to reduce mypy errors

0e0c7ce

Signed-off-by: Chad Dombrova <chadrik@gmail.com>

asserts and riskier changes

e88514c

Signed-off-by: Chad Dombrova <chadrik@gmail.com>

Add generics for package resources

519a5c9

Signed-off-by: Chad Dombrova <chadrik@gmail.com>

riskier change: return NotImplemented from __eq__

0e39c66

Signed-off-by: Chad Dombrova <chadrik@gmail.com>

chadrik force-pushed the mypyc branch from adfd9e0 to ec95bcf Compare June 16, 2026 02:24

chadrik added 7 commits June 15, 2026 19:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add the option to compile key modules to C using mypyc#2134

Add the option to compile key modules to C using mypyc#2134
chadrik wants to merge 16 commits into
AcademySoftwareFoundation:mainfrom
chadrik:mypyc

chadrik commented Jun 15, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Jun 15, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chadrik Jun 15, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant



		def get_latest_package(name: str, *, range_: VersionRange \| None = None,
		paths: list[str] \| None = None,

Conversation

chadrik commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chadrik Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

chadrik commented Jun 15, 2026 •

edited

Loading

codecov Bot commented Jun 15, 2026 •

edited

Loading