sort:Optimize sort collation for long lines

[mattsu2020]

What changed

• Avoid precomputing ICU collation sort keys for lines larger than 65,535 bytes.
• Store optional collation key ranges so very long lines can fall back to lazy locale comparison during sorting.

Why

Fixes #12138. In UTF-8 locales, sort precomputed ICU collation keys for every input line. For inputs with a small number of very large lines, such as 26 lines of 200 MiB each, the cost of generating and storing multi-GiB collation keys dominated runtime.

Impact

Small and normal-sized lines keep the existing precomputed-key fast path. Very long lines skip the expensive key materialization and use locale_cmp when compared.

Validation

• cargo check -p uu_sort
• cargo test -p uu_sort
• cargo test -p coreutils --test tests test_sort::test_default_unsorted_ints -- --exact
• Compared output against GNU sort with cmp for 52 MiB and 130 MiB reproducer inputs.
• Hyperfine on the issue-sized 5.1 GiB input with LC_ALL=en_US.UTF-8 --parallel 1 --buffer-size 8G:
  • uutils release: 5.054 s
  • GNU gsort 9.11: 33.685 s

[github-actions]

GNU testsuite comparison:

Skip an intermittent issue tests/cut/bounded-memory (fails in this run but passes in the 'main' branch)
Skip an intermittent issue tests/tail/tail-n0f (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/misc/io-errors (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/tail/retry (passes in this run but fails in the 'main' branch)
Congrats! The gnu test tests/dd/no-allocate is now passing!

[codspeed-hq]

Merging this PR will not alter performance

✅ 70 untouched benchmarks
⏩ 299 skipped benchmarks¹

---

_{Comparing mattsu2020:fix_sort_performance (3e6f8d6) with main (80cc829)²}

Footnotes

1. 299 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

2. No successful run was found on main (47d7c76) during the generation of this report, so 80cc829 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[xtqqczze]

Out of interest, why choose 1 MiB as the limit, rather than something lower like u16::MAX?

[mattsu2020]

Out of interest, why choose 1 MiB as the limit, rather than something lower like u16::MAX?

Since measurements using 64 KiB showed performance that was at least equivalent for the issue workload, we will change the threshold to u16::MAX.

[xtqqczze]

@mattsu2020 Could you also add a benchmark (in separate PR)?

[mattsu2020]

@mattsu2020 Could you also add a benchmark (in separate PR)?

Sure, I’ll keep this PR focused on the fix and open a separate PR adding a benchmark for long-line locale collation.

[sylvestre]

Codspeed is not happy

[mattsu2020]

Codspeed is not happy

The memory usage is within expectations, but I'll look into the CPU.