Increase REALITY target TLS record buffer to 17 KiB

[fanyangCS]

Fixes XTLS/Xray-core#6356.

Summary

Increase REALITY's target TLS record buffer from 8192 bytes to 16384 bytes.

This fixes a reproducible REALITY failure when the legitimate target server returns a TLS Certificate record slightly larger than 8192 bytes. In the linked Xray-core issue, www.microsoft.com can return a Certificate record with total record length 8273 bytes when OCSP/status is included:

RecordHeader: 17 03 03 20 4c
Certificate handshake length: 0x203b
0x204c + 5 = 8273

The current REALITY code rejects it because:

size = 8192
...
if handshakeLen > size { // too long
    break f
}

The resulting user-facing/server log error is only:

REALITY: processed invalid connection ... handshake did not complete successfully

Validation

I reproduced the failure with Xray-core v26.3.27 using:

• VLESS + TCP + REALITY + Vision
• REALITY dest: www.microsoft.com:443
• serverNames: www.microsoft.com
• client fingerprint: chrome

Unpatched Xray failed locally with:

curl: (35) Recv failure: Connection reset by peer
Certificate: 8273
hs.c.isHandshakeComplete.Load(): false
handshake did not complete successfully

After this patch, the same localhost REALITY server/client setup succeeds:

https://www.google.com/generate_204 -> HTTP 204

A production deployment using the patched binary was also verified by the reporter.

Notes

TLS records can be up to around 16 KiB, so 8192 is too tight for real-world OCSP-stapled Certificate records from some large sites/CDN edges. This patch keeps the change minimal and avoids changing protocol behavior beyond allowing larger legitimate target handshake records.

[fanyangCS]

Update: changed the proposed bound from 16384 to 17 * 1024.

Reason: TLS plaintext records are commonly described as 16 KiB, but TLS 1.3 encrypted records can exceed 16 KiB slightly due to overhead (RFC allows TLSCiphertext length up to 2^14 + 256). A 17 KiB bounded buffer covers valid ~16 KiB TLS records plus overhead while still remaining small and bounded.

The original reproducer (www.microsoft.com Certificate record total length 8273) is still covered; this just makes the fix less edge-case-prone.

[jcdong98]

I met the same issue. LGTM👍

Nit: Please update the PR title accordingly to align with your change.

[papka1974]

Is this fix included in v26.7.11? As i remember it was promised to include it to next release...

[Szer]

Really interested in merging this, otherwise clients (which I can't change) are broken for dest with big certs