feat: accuracy issuer inherits perf concurrency in online mode (#357)#379
Conversation
When the performance phase runs the CONCURRENCY load pattern (online), the accuracy phase now mirrors that same fixed concurrency instead of always bursting at MAX_THROUGHPUT, so evaluation exercises the endpoint the same way as the performance run. All other patterns are unchanged: POISSON and offline MAX_THROUGHPUT perf phases keep the accuracy phase at MAX_THROUGHPUT, since inheriting POISSON would silently rate-limit evaluation to the perf QPS (no accuracy QPS-budgeting yet). The gate is purely load_pattern.type == CONCURRENCY, which the schema already constrains to online mode. Also logs the accuracy issuer's chosen load mode (pattern + target_concurrency) per accuracy dataset. Adds unit tests for the concurrency-inheritance, POISSON-stays-max-throughput, offline-stays-max-throughput, and logging cases. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the benchmark execution logic so that the accuracy phase mirrors the fixed concurrency of the performance phase when a CONCURRENCY load pattern is used, while continuing to default to MAX_THROUGHPUT for other patterns (such as POISSON). It also adds logging for the accuracy issuer's load mode and includes comprehensive unit tests to verify these behaviors. There are no review comments, and I have no additional feedback to provide.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
Signed-off-by: arekay-nv <230885705+arekay-nv@users.noreply.github.com>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Signed-off-by: Rashid Kaleem <230885705+arekay-nv@users.noreply.github.com>
4 participants
When the performance phase runs an online load pattern, the accuracy phase now mirrors that same pattern instead of always bursting at MAX_THROUGHPUT, so evaluation exercises the endpoint the same way as the performance run:
target_concurrency.target_qps.It falls back to MAX_THROUGHPUT only when there is no online pattern to mirror: offline (MAX_THROUGHPUT) perf, AGENTIC_INFERENCE perf (which can't drive the non-agentic accuracy datasets —
create_load_strategyrejects it), or no perf pattern at all.Caveat: inheriting POISSON rate-limits accuracy to the perf QPS (there is no separate accuracy QPS budget yet). This is intentional so accuracy is measured under the same offered load as performance; a dedicated accuracy load-pattern control is planned as a follow-up (see the thread on the agentic case).
Also logs the accuracy issuer's chosen load mode (pattern + parameter) per accuracy dataset, via a new
LoadPattern.__str__. Adds unit tests for concurrency-inheritance, poisson-inheritance, offline→max_throughput, agentic→max_throughput, none→max_throughput, and the logging cases.What does this PR do?
Makes the online accuracy phase reuse the performance phase's load pattern (concurrency or poisson) rather than always running MAX_THROUGHPUT, so accuracy is measured under the same offered load as performance. Offline / agentic / unset perf patterns keep the MAX_THROUGHPUT fallback.
Type of change
Related issues
Related: #357
Testing
Checklist