Summary
@cache.secure inherits the default serializer and the encryption guard rejects any non-default serializer, so it always wraps StandardSerializer (msgpack), which hard-rejects DataFrames. The entire Arrow large-object optimization is unreachable for encrypted DataFrames — the exact PII/medical/financial use case secure mode targets.
Evidence
- Guard:
src/cachekit/cache_handler.py:351-362 (rejects non-default serializer when encryption on)
config/decorator.py:191,420-445 — secure() never sets a serializer (inherits "default")serializers/standard_serializer.py:106-107 — TypeError on DataFrame- Capability exists:
tests/integration/test_encryption_composability.py:62-82 shows EncryptionWrapper(serializer=ArrowSerializer()) round-trips a DataFrame
Impact
Fails loud (ConfigurationError/TypeError) — no data loss — but blocks the headline OOM optimization for large encrypted DataFrames. Note: encrypted values can also never be mmap-read (see #171); compress-then-encrypt forces inner LZ4 always-on. Related: #128.
Fix
Thread EncryptionWrapper(serializer=ArrowSerializer()) through the decorator (gated on the cross-SDK interop constraint), or document the limitation + the manual workaround.
Summary
@cache.secureinherits the default serializer and the encryption guard rejects any non-default serializer, so it always wrapsStandardSerializer(msgpack), which hard-rejects DataFrames. The entire Arrow large-object optimization is unreachable for encrypted DataFrames — the exact PII/medical/financial use case secure mode targets.Evidence
src/cachekit/cache_handler.py:351-362(rejects non-default serializer when encryption on)config/decorator.py:191,420-445—secure()never sets a serializer (inherits"default")serializers/standard_serializer.py:106-107—TypeErroron DataFrametests/integration/test_encryption_composability.py:62-82showsEncryptionWrapper(serializer=ArrowSerializer())round-trips a DataFrameImpact
Fails loud (ConfigurationError/TypeError) — no data loss — but blocks the headline OOM optimization for large encrypted DataFrames. Note: encrypted values can also never be mmap-read (see #171); compress-then-encrypt forces inner LZ4 always-on. Related: #128.
Fix
Thread
EncryptionWrapper(serializer=ArrowSerializer())through the decorator (gated on the cross-SDK interop constraint), or document the limitation + the manual workaround.