You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The "zero-copy memory-mapped reads" advertised for compression=none do not exist: there is no pa.memory_map anywhere, the File backend os.reads the whole file then slices (two full heap copies), and the CK frame unwrap copies again. Setting compression=none today only inflates the payload with no read-RSS benefit. This epic builds the real path (measured target ~0.32x read RSS).
File backend mmap read — FileBackend.get() returns a buffer backed by pa.memory_map/mmap + memoryview slice past the 14-byte header, instead of os.read+slice. backends/file/backend.py:134,167
Lazy / per-batch checksum — the eager full-body xxHash3 (arrow_serializer.py:279) faults every page and erases the mmap RSS win; make it optional/streaming for the mmap path
compression=none default for the File backend (plaintext-scoped) while wire backends keep zstd — wire the coupling at create_cache_wrapper (decorators/wrapper.py:375 where config.backend is known) and extend the name-keyed serializer cache key (serializers/__init__.py:96) so File and Redis decorators don't share one zstd instance
Use the dead File header FLAGS field (backend.py:235, hardcoded 0) to record "uncompressed Arrow, mmappable"
Constraints
Plaintext only. Encrypted values (@cache.secure) can never be mmap-read — AES-GCM decrypt reads the whole ciphertext into an owned buffer. Keep zstd for encrypted.
Summary
The "zero-copy memory-mapped reads" advertised for
compression=nonedo not exist: there is nopa.memory_mapanywhere, the File backendos.reads the whole file then slices (two full heap copies), and the CK frameunwrapcopies again. Settingcompression=nonetoday only inflates the payload with no read-RSS benefit. This epic builds the real path (measured target ~0.32x read RSS).Sequenced sub-tasks
SerializationWrapper.unwrapreturns a memoryview (prerequisite; standalone win)FileBackend.get()returns a buffer backed bypa.memory_map/mmap+ memoryview slice past the 14-byte header, instead ofos.read+slice.backends/file/backend.py:134,167arrow_serializer.py:279) faults every page and erases the mmap RSS win; make it optional/streaming for the mmap pathcompression=nonedefault for the File backend (plaintext-scoped) while wire backends keep zstd — wire the coupling atcreate_cache_wrapper(decorators/wrapper.py:375whereconfig.backendis known) and extend the name-keyed serializer cache key (serializers/__init__.py:96) so File and Redis decorators don't share one zstd instancemax_value_mb(currently rejects the ~300 MB motivating payload,file/config.py:75)backend.py:235, hardcoded 0) to record "uncompressed Arrow, mmappable"Constraints
@cache.secure) can never be mmap-read — AES-GCM decrypt reads the whole ciphertext into an owned buffer. Keep zstd for encrypted.