Header Parsing Optimizations#13376
Draft
moonchen wants to merge 14 commits into
Draft
Conversation
tools/benchmark/benchmark_HdrParse provides a Catch2 A/B stats mode and a --profile loop (for perf) over realistic, adversarial, and file-loaded header corpora. It drives the production parse path -- zero-copy (must_copy_strings=false) under strict_uri_parsing=2 -- so header-parse optimizations are measured where they actually run.
hdrtoken_tokenize returned a well-known-string index whenever the 32-bit name hash and length matched, without comparing the bytes. An equal-length name that collided in the hash could therefore be mis-interned, taking on the WKS's presence bits and slot accelerators (e.g. treated as Content-Length). Confirm the candidate with an exact ASCII-case-insensitive byte comparison before accepting the WKS index.
hdrtoken_hash ran the name through ATSHash32FNV1a with an ATSHash::nocase transform that calls libc toupper() once per byte: an out-of-line, locale-sensitive call in the hot path of every field name, method, and URL scheme. The polymorphic ATSHash32FNV1a also cannot inline its ctor/final/get/dtor across translation units, so each hash additionally pays for those out-of-line calls. Replace it with an inlined FNV-1a loop over an ASCII-folded name, folding each byte through a small hdrtoken_hash_step() helper (later reused by the fused field-name scan, keeping the seed and step single-copy). The fold matches toupper() under the C locale for all byte values, so the well-known-string table's hash values and collision pattern are unchanged, but the result is locale-independent and call-free. Tokenizing the field-name corpus is ~2x faster in the header-parse benchmark; realistic request/response parsing drops ~17%.
url_is_mostly_compliant runs under strict_uri_parsing mode 2 -- the config default -- so it validates every request target. It called isspace() and isprint() once per byte: both are out-of-line and locale-sensitive, and passing a possibly-negative char is undefined for bytes above 0x7F. Mode 2 accepts exactly the printable non-space range 0x21..0x7E, so replace the pair with one well-defined range check, reclassifying on the cold reject path to preserve the diagnostics. Validating a 4 KB target is ~2.8x faster in the benchmark; realistic request parsing drops ~9%.
During parsing every field was attached with a duplicate check that, for a non-well-known name, scanned all prior fields case-insensitively -- O(n^2) for a header with many distinct custom names. hdrtoken_tokenize already computes a hash of each name; return it (on every path) and keep a parse-local Bloom of the non-well-known names seen so far. When a name's bit is clear it cannot duplicate an earlier field, so attach skips the search, taking the same no-duplicate path a null find would. The Bloom is keyed to the header it was seeded from, so a parser reused on a different header without an intervening clear reseeds instead of missing duplicates; it is also seeded from any fields already present, so appending into a non-empty header stays correct. Parsing 100 distinct custom-header fields is ~36% faster and 50 non-well-known fields ~53%; typical all-well-known headers are unchanged.
REQ_REALISTIC uses only classic well-known header names, so it does not exercise the non-WKS name paths (tokenize miss, name validation, the duplicate filter). Add REQ_MODERN, a 2026 Chrome navigation request carrying the client-hint and fetch-metadata headers (sec-ch-ua*, sec-fetch-*, priority, upgrade-insecure-requests) that are not well-known strings, so the benchmark reflects how those paths cost on real traffic.
The request-target validation calls three out-of-line URL getters and a branch tree that reduces to a single member-field test. Compute it directly, dropping the getter calls on every request. Origin-, asterisk-, absolute-, and authority-form behavior is unchanged.
url_is_mostly_compliant checked the request target one byte at a time. Replace that loop with a single branchless pass that OR-reduces an out-of-range flag over the whole target; with no early exit or data-dependent branch the compiler auto-vectorizes it to the build's SIMD (ATS builds -O3, where clang and GCC both do). This merges the scan into one loop and drops the cold-path whitespace/non-printable debug logging -- the false return already signals a rejected target. An exhaustive differential test checks the scan against a scalar reference for every byte value at every position across lengths spanning a sub-vector target, a whole vector, and the scalar remainder.
A non-well-known field name was walked three times: locating the colon, computing the FNV hash, and validating each byte. Scan once, accumulating the hash and a validity flag, and reuse the hash for both the WKS lookup and the duplicate Bloom. The WKS hash/lookup stays single-copy via a prehashed tokenize helper. A parity test checks the fused scan against references for the colon position and per-byte validity, and confirms the prehashed lookup fed the fused hash returns the well-known index the standalone tokenizer does.
Field names arrive in canonical case, so an exact memcmp usually settles the well-known-name collision check without the per-byte ASCII fold; the fold still runs on a mismatch to catch case variants. memcmp-equal implies fold-equal, so the accepted set is unchanged.
The parse-local dup filter was a single 64-bit word; for header blocks with many custom names its false-positive rate drove wasted O(n) searches. Reclaim three long-dead MIMEParser scratch fields (m_field, m_field_flags, m_value) to hold a second word, halving the false-positive rate without growing the parser.
The parse-time duplicate-search skip covered only non-well-known names. For a well-known name a clear presence bit is exactly the negative result mime_hdr_field_find would return, so the O(n) walk can be bypassed too. Mask-zero well-known names keep the search on. This extends the skip to ordinary all-well-known headers, not just custom-heavy ones.
A duplicate of the immediately-preceding attached field belongs at that field's dup-chain tail, so splice it directly instead of running attach's O(n) duplicate search and slot-number walk. Any non-adjacent case (a mismatched or non-tail predecessor) falls back to the full attach path, so the fast path is self-validating and safe across CONT re-entry. An equivalence test builds consecutive, interleaved, and well-known duplicates both through the parser and through explicit create+attach and asserts the resulting dup chains are identical.
moonchen
force-pushed
the
header-parse-optimization
branch
from
dd3ee23 to
0442cf1
Compare
hdrtoken_hash_init builds the WKS lookup table at startup and used to guard against two strings landing in the same hash slot by printing an error and abort()ing. Make the fold/hash chain (hdrtoken_ascii_toupper, hdrtoken_hash_step, hdrtoken_hash, hash_to_slot) and the string table constexpr, and check the same slot-uniqueness invariant with a static_assert instead. A colliding table -- from editing the well-known-string list or changing the hash -- is now a build error rather than a startup abort, so the runtime scan and abort() are dropped as unreachable.
moonchen
force-pushed
the
header-parse-optimization
branch
from
0442cf1 to
656735c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft — putting this up so folks can try it if they want. Still pending my own production testing; not proposing to merge yet.
A stack of scalar optimizations to the HTTP header-parsing hot path (
src/proxy/hdrs/), plus a micro-benchmark harness. Portable C++ only — no SIMD/Highway dependency, and no changes to the well-known-string table (those would invalidate cache).Changes
tools/benchmark/benchmark_HdrParse.cc, gated by-DENABLE_BENCHMARKS=ON) covering the request/response/mime/url/wks parse paths, with realistic and adversarial corpora.toupper()and the out-of-line hash object from the hot path. Hash values are unchanged.static_assertthat no two well-known strings share a hash slot (replacing the startup collision scan +abort()).isspace/isprint→ a single branchless range-check pass the compiler auto-vectorizes to the build's SIMD (no explicit SIMD library).Set-Cookie).validate_hdr_request_targetfolded to a truth table.Correctness
test_proxy_hdrs: 436,859 assertions / 41 cases pass.Performance (measured on two desktop-class computers — independent validation welcome)
Parse-path micro-benchmark (clang, RelWithDebInfo): realistic request −30%, modern-browser request −41%, URL −41%, MIME fields −25%, WKS tokenize −58%, response −14%.
End-to-end (10GbE, RAM-cache hit, ATS pinned to isolated cores; metric = CPU per 1k requests): HTTP/1.1 plain −3.7%, HTTP/1.1 TLS −3.9%, HTTP/2 TLS −6.3%. Parsing is a fraction of the request path, so the micro-bench delta dilutes end-to-end as expected.
Not included (on purpose)
sec-ch-ua*,sec-fetch-*) as new WKS entries — that changes the WKS table and would invalidate cache; also contentious on perf.