Add experimental Windows wheel build support#1359
Open
zym1998year wants to merge 3 commits intoGalSim-developers:releases/2.8from
Open
Add experimental Windows wheel build support#1359zym1998year wants to merge 3 commits intoGalSim-developers:releases/2.8from
zym1998year wants to merge 3 commits intoGalSim-developers:releases/2.8from
Conversation
setuptools 72.0+ removed setuptools.command.test; guard the import. Add IS_WINDOWS sentinel for subsequent platform branches. Normalize the mmgr.cpp source-list match via os.path.normpath so glob's backslash paths still resolve. Use os.pathsep when splitting LIBRARY_PATH / C_INCLUDE_PATH / PATH-style env vars so the colon in Windows drive letters isn't treated as a separator.
- copt/lopt: MSVC entry with /O2 /std:c++14 /EHsc /openmp /Zc:__cplusplus /utf-8 /DNOMINMAX (/openmp wins MSVC's old runtime but suffices for the loops GalSim parallelizes today). - get_compiler_type: short-circuit MSVC via compiler_type before touching compiler_so (Unix-only attribute). - try_compile: route MSVC probes through compiler.compile() and compiler.link_executable() instead of hand-built cc -c / -o lines. - fix_compiler: skip ccache, -msse2, -stdlib=libc++ removal and linker_so editing on MSVC; force single-process compile on Windows (parallel_compile pool path is Unix-shaped; MSVC can later regain per-extension parallelism via /MP). - find_fftw_lib: search %CONDA_PREFIX%\Library\lib and the vcpkg layout, accept fftw3.lib / libfftw3.lib / libfftw3-3.lib, and skip ctypes.LoadLibrary on Windows (the located file is the import library, not the runtime DLL). - find_eigen_dir: same conda/vcpkg additions for include dirs. - Extension: use libraries=['fftw3'] on Windows; -lfftw3 stays for GCC/Clang. - my_build_ext.run: skip GALSIM_BUILD_SHARED on Windows; that path uses os.symlink and bakes in .so/.dylib naming.
- include/galsim/Std.h: define NOMINMAX before <Windows.h> so std::min / std::max (and Eigen members) survive unshadowed. The MSVC build also passes /DNOMINMAX, but defining it here protects direct header consumers. - include/galsim/Stopwatch.h: switch to std::chrono::steady_clock from gettimeofday. No GalSim TU includes this header today, but keeping the public surface portable avoids surprises. - src/Image.cpp, src/Polygon.cpp: replace alternative tokens (`or`) with `||`. MSVC parses these as identifiers without /Za + ciso646. - src/Random.cpp: split the POSIX <sys/time.h> / <unistd.h> / <fcntl.h> includes behind #ifndef _WIN32 and add _WIN32 branches for seedurandom() (std::random_device, which delegates to BCryptGenRandom/CryptGenRandom on Windows CRTs) and seedtime() (std::chrono::system_clock with the same microsecond-modulo seed). - src/SBInterpolatedImage.cpp, src/WCS.cpp, src/SBTransform.cpp: replace GCC variable-length arrays with std::vector<...>. Pass .data() to consumers expecting raw pointers (Horner2D, memset, reinterpret_cast onto std::complex<T>, KValueInnerLoop). The std::vector path keeps the original observable behaviour: heap allocation per call instead of stack, but the inner loops dominate any allocation cost.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft PR against
releases/2.8(baselineee177aa50, v2.8.4).Branch on file:
windows-msvc-port.Status
setup.py build_clib + build_ext,pip wheel,delvewheel repair, fresh-venv install (no conda PATH),import galsim,Gaussian.drawImage, FFT-backed convolve,BaseDeviate(42)determinism,hsm.EstimateShear.multiprocessingconfigflow,
download_cosmossymlink fallback, full Windows pytest pass,GitHub Actions CI. See Follow-ups.
IS_WINDOWSbranches and_WIN32C++ guards. I have not re-run theexisting CI from this branch; the diff stays under
if IS_WINDOWS:so no GCC/Clang behavioural change is intended.Summary
Three small commits make GalSim build, install, and import on native
Windows x64 with the MSVC toolchain. The result is a self-contained
win_amd64wheel that installs into a fresh venv with no condadependency at runtime, and that produces correct numbers on a small
smoke surface (
drawImage, FFT convolve, random determinism, HSMEstimateShear).
The PR stays on the minimal-diff setuptools path. A
scikit-build-corerewrite is a larger discussion this PR does notattempt.
What this PR changes
setup.pysetup.pyinclude/,src/Branch tip:
9f4db829c.Per-commit detail (click to expand)
1. setup metadata and paths
setup.pyonly. Surgical:from setuptools.command.test import testimport.setuptools 72.0+ removed it; the cmdclass below never registered a
testcommand.IS_WINDOWS = sys.platform == 'win32'.cpp_sources.remove('src/mmgr.cpp')-> compare viaos.path.normpath, so theglob-returnedsrc\mmgr.cppon Windowsstill matches.
':'->os.pathsepwhen splittingLIBRARY_PATH,LD_LIBRARY_PATH,DYLD_LIBRARY_PATH,C_INCLUDE_PATH, and theinstalled-script PATH check. Avoids treating drive-letter colons as
separators.
2. MSVC and Windows FFTW / Eigen
setup.pyonly. The Windows build path:copt/loptgain an'msvc'entry:/O2 /std:c++14 /EHsc /openmp /Zc:__cplusplus /utf-8 /DNOMINMAX.get_compiler_typeshort-circuits viagetattr(compiler, 'compiler_type', None) == 'msvc'before touchingcompiler_so(Unix-only on
MSVCCompiler).try_compileadds an MSVC branch usingcompiler.compile()/compiler.link_executable()instead of constructingcc -c -ocommand lines.
fix_compilerskips ccache,-msse2,-stdlib=libc++removal, andthe
-Llinker_so rewrite on MSVC; forces single-process compile onWindows (
parallel_compileis shaped around the Unix invocationpattern).
find_fftw_libsearches%CONDA_PREFIX%\Library\lib, vcpkg'sinstalled/x64-windows/lib, and acceptsfftw3.lib/libfftw3.lib/libfftw3-3.lib. Thectypes.cdll.LoadLibraryprobe is skipped on Windows because the resolved path is the import
library, not a runnable DLL.
find_eigen_dirgets the same conda / vcpkg additions for includedirectories.
libraries=['fftw3']on Windows, keepsextra_link_args=['-lfftw3']on the GCC/Clang path.my_build_ext.runskipsGALSIM_BUILD_SHAREDon Windows: thatcodepath uses
os.symlinkand.so/.dylibnaming.3. C++ sources for MSVC
8 files:
include/galsim/Std.h:#define NOMINMAXbefore<Windows.h>sostd::min/std::max(and Eigen members) survive.include/galsim/Stopwatch.h: switch fromgettimeofdaytostd::chrono::steady_clock. No GalSim TU includes this headertoday; the rewrite is hygiene.
src/Image.cpp,src/Polygon.cpp: replace alternative tokenorwith
||. MSVC parsesoras an identifier without/Zaplus<ciso646>.src/Random.cpp: split the POSIX<sys/time.h>/<unistd.h>/<fcntl.h>includes behind#ifndef _WIN32; add_WIN32branchesfor
seedurandom()(std::random_device, which delegates toBCryptGenRandom on the Windows CRT) and
seedtime()(
std::chrono::system_clockwith the same microsecond-modulo seed).src/SBInterpolatedImage.cpp,src/WCS.cpp,src/SBTransform.cpp:replace GCC variable-length arrays with
std::vector<...>; pass.data()to consumers expecting raw pointers (Horner2D,memset/std::fill,reinterpret_castontostd::complex<T>,KValueInnerLoop).Verification
Built and tested on Windows 11 x64, Python 3.11.15 (conda-forge), VS
2026 Community (MSVC 14.50.35717), conda-forge
fftw 3.3.10/eigen 3.4.0/pybind11 3.0.3.Build / wheel
fftw3.dllvcruntime140.dll,vcruntime140_1.dllmsvcp140.dllpython311.dll,kernel32.dll,api-ms-win-crt-*vcomp140.dllFresh-venv smoke
import galsim2.8.4,_galsim.cp311-win_amd64.pydresolved from venv site-packagesGaussian(sigma=1.2).drawImage(nx=16, ny=16, scale=0.2)(16, 16), sum0.6684Convolve([Sersic(n=2.5), Gaussian])0.9737(flux=1 retained ~3%)BaseDeviate(42)firstGaussianDeviatesample-0.23898784173300305reproducible across instanceshsm.EstimateShearong1=0.2, g2=0.1status=0,e1=0.381,e2=0.190The cleaned PATH does not include
%CONDA_PREFIX%\Library\bin, sothe bundled
fftw3.dllis the one being loaded.Reproduction transcript (full local commands)
phase3_smoke.pyruns the five tests in the table above and exitsnon-zero on any failure.
Wheel notes
fftw3fromconda-forge::fftw,msvcp140/vcomp140from the env's MSVCredistributable mirror.
fftw3.dllinside a wheel ispermitted under GPL-2.0-or-later but pushes the binary distribution
under GPL constraints. If MIT-only binary distribution is preferred,
an alternative is to ship two flavours (FFTW vs. KissFFT or
pocketfft); out of scope for this PR. Reviewer input requested.
vcomp140.dllis the older Microsoft OpenMP runtime;/openmp:llvmwould pulllibomp.dllinstead. Reviewer inputrequested.
cp311-cp311-win_amd64. cibuildwheel will produce onewheel per Python minor version.
Follow-ups
Deliberately not in this PR. Each is a candidate for a separate PR
once this one lands.
galsim/config/util.pyusesmultiprocessing.get_context('fork'),which Windows lacks. Needs a
spawnfallback plus an audit ofconfig processing / phase-screen code for picklability under
spawn. Not exercised by the smoke tests above; surfaces as soon asa config-driven run hits the multiprocessing branch.
galsim/download_cosmos.pyusesos.symlink, which non-elevatedWindows users cannot create unless Developer Mode is on. Needs
copy/ directory-junction /--nolinkfallback.tests/for fork-only or symlink-onlyassumptions, mark with
pytest.mark.skipif(sys.platform == 'win32')where appropriate, and run the full pytest on
windows-latest.(
cp39 .. cp313 -win_amd64). I have a working local recipe but wantto align with project preferences (FFTW source, OpenMP runtime,
delvewheel command) before sending it; see
Reviewer questions below.
Risk and compatibility
IS_WINDOWS(insetup.py) or_WIN32(in C++). I expect nobehavioural regression on those platforms but did not re-run their
CI from this branch.
std::vectorinSBInterpolatedImage.cpp,WCS.cpp,SBTransform.cpp: per-call adds heap allocation; the loops affectedare inner enough that I expect dominance from per-element ray
operations, but I have not benchmarked. If micro-benchmarks regress
noticeably, an alternative is
_alloca/allocaon the Windowspath with VLAs preserved on GCC. I went with
std::vectorforcross-platform clarity.
Random.cppWindows seeding usesstd::random_device. OnMSVC's CRT this delegates to the OS CSPRNG, matching the
/dev/urandomcontract. POSIX builds are unchanged.delvewheelwarns when thebundled
msvcp140.dllis older than the toolset that built the.pyd. I built locally with VS 2026 (14.50) and bundledconda-forge's 14.44 redistributable. The wheel still loaded and
passed the smoke tests; CI on
windows-latest(VS 2022 14.4x)would not see this warning. No code change required.
Reviewer questions
or fftw.org pre-built. Affects the GPL-redistribution note.
/openmp(MS) vs/openmp:llvm. The latter aligns betterwith modern OpenMP but adds a
libomp.dllredistribution step.std::vectorchange: keep it global (current PR), orkeep VLAs on GCC and switch only on MSVC.
submit
multiprocessing/symlink/ Windows test markersseparately, or fold in.
windows-latest+ cibuildwheel workflow asa follow-up PR once questions 1-2 are resolved. A draft strategy
doc is available locally if helpful.
Happy to split, rebase, or extend as the project prefers.