cache: shared-memory-backed Dir for fast restart#13328
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces an opt-in “fast restart” cache directory implementation by hosting each stripe’s in-memory directory (raw_dir) in POSIX shared memory, allowing subsequent traffic_server starts to attach the prior directory quickly instead of rebuilding from disk. It also adds operator tooling (traffic_ctl cache shm status|clear), configuration records, extensive AuTest coverage, unit tests for trust gates, and design/admin documentation.
Changes:
- Add shared-memory directory infrastructure (
CacheShm*) plus integration into cache startup, stripe directory allocation, and shutdown/clean-marking paths. - Add
traffic_ctl cache shm status|clearcommands and newproxy.config.cache.shm.*records. - Add unit + AuTest suites and documentation for the shm fast-restart feature.
Reviewed changes
Copilot reviewed 32 out of 32 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/gold_tests/logging/ts_process_handler.py | Hardens psutil cmdline handling for macOS/permission-restricted processes. |
| tests/gold_tests/cache/shm_poke.py | Adds Linux-only helper to tamper with /dev/shm segments for trust-gate testing. |
| tests/gold_tests/cache/replay/cache-shm-fast-restart.replay.yaml | Proxy Verifier replay traffic used to validate hit-after-restart behavior. |
| tests/gold_tests/cache/gold/cache_shm_state_after_shutdown.gold | Gold output for traffic_ctl cache shm status validation. |
| tests/gold_tests/cache/cache_shm_unclean_shutdown.test.py | AuTest for rejecting dirty segments after SIGKILL and rebuilding from disk. |
| tests/gold_tests/cache/cache_shm_storage_mismatch.test.py | AuTest for partial attach behavior when storage layout changes. |
| tests/gold_tests/cache/cache_shm_schema_mismatch.test.py | Linux-only AuTest that pokes schema_version and verifies drop+rebuild. |
| tests/gold_tests/cache/cache_shm_purge_on_disable.test.py | AuTest for purge-on-disabled-start behavior and traffic_ctl exit codes. |
| tests/gold_tests/cache/cache_shm_fast_restart.test.py | End-to-end AuTest for clean shutdown -> shm attach -> cache HIT without origin contact. |
| tests/gold_tests/cache/cache_shm_concurrent_attach.test.py | AuTest for concurrent-attach guard (flock vs owner_pid liveness backstop). |
| tests/gold_tests/cache/cache_shm_bad_disk_dropped.test.py | AuTest for partial attach and orphan reclaim when a disk is removed from storage. |
| src/traffic_ctl/traffic_ctl.cc | Adds `traffic_ctl cache shm status |
| src/traffic_ctl/CMakeLists.txt | Builds the new traffic_ctl command source and adds cache include path for shared headers. |
| src/traffic_ctl/CacheShmCommand.h | Declares the traffic_ctl cache shm command handler. |
| src/traffic_ctl/CacheShmCommand.cc | Implements shm status/clear via direct shm_open/mmap, shared purge primitive, and exit codes. |
| src/records/RecordsConfig.cc | Registers proxy.config.cache.shm.* configuration records. |
| src/iocore/cache/unit_tests/test_CacheShm.cc | Adds unit tests for ABI hash, storage signature, layout round-trip, prefix normalization, and liveness checks. |
| src/iocore/cache/StripeSM.cc | Adds fast-attach path that can skip disk dir read + recovery when shm directory is trusted; adjusts shutdown behavior. |
| src/iocore/cache/Stripe.h | Adds _shm_directory_is_valid() and strengthens flush API via [[nodiscard]]. |
| src/iocore/cache/Stripe.cc | Uses shm-backed directory allocation; adds shm bounds validation; detaches shm mappings safely in destructor. |
| src/iocore/cache/CMakeLists.txt | Builds new CacheShm.cc and adds test_CacheShm to the cache unit tests. |
| src/iocore/cache/CacheShmPurge.h | Introduces shared header-only purge/enumerate/unlink primitive for server startup and traffic_ctl. |
| src/iocore/cache/CacheShmLayout.h | Defines the shared control-segment layout and prefix normalization utilities. |
| src/iocore/cache/CacheShm.h | Declares the CacheShm facade for lifecycle + stripe attach/create + trust gates. |
| src/iocore/cache/CacheShm.cc | Implements shm control segment lifecycle, trust gates, stripe attach/create, orphan reclaim, and clean-shutdown marking. |
| src/iocore/cache/CacheProcessor.cc | Calls CacheShm initialization before stripe construction and finalization after cache init. |
| src/iocore/cache/CacheDir.cc | Marks shm control segment clean only after shutdown sync has quiesced writers. |
| src/iocore/cache/AggregateWriteBuffer.h | Marks flush() as [[nodiscard]] to encourage handling short-write failures. |
| src/iocore/cache/AggregateWriteBuffer.cc | Converts flush failure from an assertion to a false return for graceful handling. |
| doc/developer-guide/cache-architecture/shm-fast-restart.en.rst | Adds detailed design doc: layout, gates, attach modes, shutdown semantics, tooling, and platform notes. |
| doc/developer-guide/cache-architecture/index.en.rst | Hooks shm fast-restart document into the cache architecture index. |
| doc/admin-guide/files/records.yaml.en.rst | Documents new proxy.config.cache.shm.* records and operator-facing behavior/tooling. |
Cold-start cache initialization rebuilds each stripe's in-memory directory from disk on every restart -- multi-minute on large caches. Host the directory in POSIX shared memory so the next process start attaches the existing segment in milliseconds instead of rebuilding it. Recovery stays binary and fail-safe: when the segment cannot be trusted (crash, reboot, ABI/schema or storage mismatch, failed validation) the start drops it and rebuilds via the existing disk path, and reads still validate Doc magic + key so a stale entry is a miss, never corruption. Opt-in behind proxy.config.cache.shm.enabled (default 0), where it is a functional no-op.
masaori335
force-pushed
the
asf-master-dir-on-shm
branch
from
8358d37 to
e0ee0af
Compare
POSIX shm names permit only the leading '/', so a misconfigured name_prefix like "foo/bar" would build a name shm_open rejects with EINVAL. Strip embedded '/' during normalization instead of preserving it.
On glibc < 2.34 (e.g. CentOS 7) shm_open/shm_unlink live in librt, so traffic_ctl and inkcache fail to link. Add an optional rt::rt target that is a no-op where the library is folded into libc (modern glibc, macOS).
There was a problem hiding this comment.
Approving. This is well built and safe to land. It is opt-in and off by default, the shared-memory attach path validates the mapping size before use and bounds the stripe count, and every failing system call is logged with its name and error string and then propagated back to the caller.
One correction to my earlier note. I wrote that a fast-attach bug returning a wrong offset or length would still pass CI. That is not accurate, and I withdraw it. The fast_restart test wires the origin to return a 502 on the hit transaction, so if the shared-memory directory failed to resolve the cached object, ATS would fall through to the origin and the 502 would fail the test. The real gap is narrower than I first stated.
The items below are all non-blocking for an off-by-default feature, but worth closing out before this is relied on in production.
-
The testing table in
doc/developer-guide/cache-architecture/shm-fast-restart.en.rstlists acache_shm_data_integrityautest that is not in this PR. Please either add the test or drop the row. -
The objects the tests serve are small and single-fragment. A multi-fragment or large-object case after a fast attach would cover the one path the 502 trick does not fully exercise.
-
The rejection branches in
_shm_directory_is_valid()and the invalidate-on-shutdown paths have no direct unit coverage, as the caveats note. They are close to pure header math, so each branch is a cheap unit test.
I also agree with the open Copilot threads on this PR.
-
In
CacheShm.cc:229,unlink_all_known_segments()unlinks stripe names read from the control segment. On the path that drops an untrusted or corrupt control segment those names are untrusted, so a corrupt segment could lead toshm_unlinkon unrelated objects. Since the names are always built fromname_prefixviabuild_stripe_shm_name(), gating the unlink loop on a prefix match, and only running it when the control magic is valid, closes this with no effect on the normal path. -
The three
storage.configreferences in the design doc at lines 132, 340, and 430 should bestorage.yamlto match the tests and the rest of this PR.
Thanks for the clean, well-documented change. The approval stands.
A corrupt control segment (invalid magic) left the stripe-name bytes untrusted, yet the drop path unlinked them, so a bad segment could shm_unlink unrelated POSIX objects. Only walk the stripe table when the header magic is intact and restrict unlinks to our own name prefix, matching the guard already in purge_segments(). Also correct storage.config -> storage.yaml in the design doc and drop the testing-table row for the not-yet-added cache_shm_data_integrity autest.
|
Thank you for taking a look. I addressed 1. 4. and 5. by 0cc3b52. |
| if (addr == MAP_FAILED) { | ||
| int e = errno; | ||
| Warning("mmap(%s, %zu) failed: %s", name.c_str(), size, strerror(e)); | ||
| if (out_errno != nullptr) { | ||
| *out_errno = e; | ||
| } | ||
| return nullptr; | ||
| } |
| ++reclaimed; | ||
| } | ||
| if (reclaimed > 0) { | ||
| Note("cache shm: reclaimed %u orphaned stripe segment(s) after storage change", reclaimed); |
| // All writers are now stopped (every stripe's mutex was held and released by the | ||
| // shutdown threads above), so the directory is final -- only now is it safe to | ||
| // mark the shm control segment clean. | ||
| CacheShm::mark_clean_shutdown(); |
There was a problem hiding this comment.
I don't think "All writers are now stopped" is right here. shutdown() only holds/releases each stripe's mutex, which excludes concurrent writers, not future ones. An ET_NET thread can still complete a write after a stripe's snapshot, and this runs before shut_down_event_system().
Was the intention that a host would need to be fully drained before the cache dir can be persisted?
There was a problem hiding this comment.
That's true, this comment is wrong. The intention is set the "clean flag" to indicate next process that previous process did not crash (did shutdown) and cache dir on shm is available.
| if (!this->_write_buffer.is_empty()) { | ||
| Dbg(dbg_ctl_cache_dir_sync, "Dir %s: flushing agg buffer first", this->hash_text.get()); | ||
| this->flush_aggregate_write_buffer(this->fd); | ||
| if (!this->flush_aggregate_write_buffer(this->fd)) { |
There was a problem hiding this comment.
What happens if the agg buffer write is pending in AIO? Before this PR, the double write was probably harmless, but now the wrong write_pos would get picked up by the next ATS pid.
There was a problem hiding this comment.
Good point. When double write (flush agg bufer + in-flight AIO) happens, write_pos is advanced twice. So next process will start writing from there. This is wasting disk space. Let me figure out.
There was a problem hiding this comment.
I'm going to change shutdown sequence of cache, but it'll be big change, so as a workaround, I invalidate the stripe when AIO is in progress. ( 54a2ec3 )
if (shm_backed && this->is_io_in_progress()) {
Dbg(dbg_ctl_cache_dir_sync, "Dir %s: AIO write in flight -- invalidating shm copy, syncing dir to disk", this->hash_text.get());
CacheShm::invalidate_stripe_directory(this->directory.raw_dir);
shm_backed = false;
}
Address review feedback on the shm-backed Dir. Create shm segments with O_EXCL so a create never adopts a pre-existing object; otherwise a segment planted by another user could hand ATS attacker-controlled permissions and contents. Both create paths already unlink stale names first, so this only rejects races and pre-planted files. Enabling shm previously preempted the MAP_HUGETLB dir allocation silently, dropping a box that relied on the global hugepage allocator to base tmpfs pages. shm segments cannot use MAP_HUGETLB, so inherit the allocator's intent by advising MADV_HUGEPAGE (THP) when shm.use_hugepages is left at default, honor an explicit opt-out, and log the substitution once.
masaori335
force-pushed
the
asf-master-dir-on-shm
branch
from
91899f8 to
161f39c
Compare
A storage.yaml change does not discard the shm segments; storage_ signature is informational, not a trust gate. Each stripe attaches by its own identity, so unchanged stripes fast-attach, added or resized stripes rebuild from disk, and vanished stripes are reclaimed. Drop the misleading discard bullet and describe the partial attach instead.
Huge pages back the shm directory to cut page-table teardown cost at process exit -- the restart-speed motivation for the feature -- not merely to reduce TLB pressure. Align the admin guide with the design doc and code.
At graceful shutdown a shm-backed stripe with an aggregate write still in flight would have write_pos advanced twice -- once by the shutdown flush, again by the deferred aggWriteDone -- so the fast-attach path would trust an imprecise cursor verbatim. Invalidate the shm copy and fall through to a fresh on-disk directory write, so the next start recovers via recover_data() with a short scan instead of a stale periodic-sync copy. Also reword the mark_clean_shutdown comment, which overstated that all writers are stopped; the real guarantee is the read-path magic+key check that turns any late write into a miss, never served corruption.
Fold the PR 13328 review fixes into the feature: - unlink the shm segment on mmap failure in the create path, so a failed create does not leak it and wedge the next O_EXCL create on EEXIST (silently disabling shm). - harden purge_segments(): filter stripe unlinks to the configured prefix so a corrupt-but-magic-valid table cannot touch unrelated shm, and report an fstat failure instead of unlinking the control object. - retry the control flock on EINTR so a signal does not drop the guard to the owner-pid backstop; reword the orphan-reclaim log to "after attach" since it also fires on disk-open failure. - drop the redundant cmdline guard and stale comment in the logging test helper, and tighten the longest shm comment blocks. No functional change to the fast-restart path itself.
| @@ -49,7 +49,6 @@ AggregateWriteBuffer::flush(int fd, off_t write_pos) const | |||
| { | |||
| int r = pwrite(fd, this->_buffer, this->_buffer_pos, write_pos); | |||
Motivation
Cold-start cache initialization rebuilds each stripe's in-memory directory from disk on every
restart — multi-minute on large caches.
Approach
Host the directory (
raw_dir) in POSIX shared memory so the next process start attaches theexisting segment in milliseconds instead of rebuilding it. Recovery is binary and fail-safe:
anything untrustworthy (crash, reboot, ABI/schema mismatch, failed validation, bad disk) falls
back to the existing disk-rebuild path, and reads still validate
Docmagic + key so a staleentry is a miss, never served corruption.
New configs & traffic_ctl commands
Opt-in behind
proxy.config.cache.shm.enabled(default0, a functional no-op). Also addsproxy.config.cache.shm.{name_prefix,use_hugepages,purge_stale_on_start}and atraffic_ctl cache shm status|clearcommand.Details
Design, recovery model, configuration, and platform notes are in
doc/developer-guide/cache-architecture/shm-fast-restart.en.rst(added in this PR) and theproxy.config.cache.shm.*entries inrecords.yaml.Testing
test_CacheShm(ABI hash, storage signature, control round-trip, name length, prefixnormalization, process liveness).
cache_shm_*suites — fast restart, unclean shutdown, schema/storage mismatch,bad-disk drop + orphan reclaim, concurrent-attach refusal, purge-on-disable.
Caveats
_shm_directory_is_valid()rejection branches or thebad-disk / flush-failure
invalidate_stripe_directory()paths (planned follow-up).flockguard is authoritative on Linux but ano-op on macOS POSIX shm, where it falls back to an owner-pid liveness check.