[evaluation] Name both SDK and asset aliases in TaskNavigationEfficiency validator errors#48048
Open
m7md7sien wants to merge 5 commits into
Open
[evaluation] Name both SDK and asset aliases in TaskNavigationEfficiency validator errors#48048m7md7sien wants to merge 5 commits into
m7md7sien wants to merge 5 commits into
Conversation
…ncy validator errors TaskNavigationEfficiencyValidator accepts either the SDK input names (response/ground_truth) or the azureml-assets names (actions/expected_actions), but its user-facing validation messages referenced only response/ground_truth. A caller who correctly supplied actions/expected_actions could get an error naming ground_truth/response - parameters they never used and that the eval service does not accept as data-mapping keys - which was confusing. Update every validation message that names response/ground_truth to also cite the accepted alias, e.g. 'ground_truth' (also accepted as 'expected_actions') and 'response' (also accepted as 'actions'). Message text only; no changes to categories, logic, or control flow.
|
Azure Pipelines: Successfully started running 1 pipeline(s). 9 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
github-actions
Bot
added
the
Evaluation
Contributor
There was a problem hiding this comment.
Pull request overview
Clarifies task-navigation validation errors by naming both SDK inputs and service aliases.
Changes:
- Updates
response/ground_trutherrors to mentionactions/expected_actions. - Documents the fix in the unreleased changelog.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
CHANGELOG.md |
Records the validation-message fix. |
_task_navigation_efficiency_validator.py |
Adds accepted aliases to user-facing errors. |
…eml-assets names
Swap the alias order in every TaskNavigationEfficiencyValidator error message so the azureml-assets input names ('actions'/'expected_actions') are primary and the SDK names ('response'/'ground_truth') are the parenthetical alias. Update the CHANGELOG entry to match. Message text only; no logic, category, or control-flow changes.
…ub.com/m7md7sien/azure-sdk-for-python into eval/tne-validator-dual-name-error-msg
ahmad-nader
approved these changes
Jul 15, 2026
shrutiyer
approved these changes
Jul 15, 2026
|
Azure Pipelines: Successfully started running 1 pipeline(s). 9 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
TaskNavigationEfficiencyValidator(added to the SDK in Azure/azureml-assets#5199, wired up in #5203) accepts either the SDK input names (response/ground_truth) or the azureml-assets / eval-service names (actions/expected_actions) via_normalize_input_aliases. Thebuiltin.task_navigation_efficiencyevaluator declares its evaluation-service data-mapping input names asactions/expected_actions— the Foundry eval service / ACA / C# require these and reject the SDK-styleresponse/ground_truthmapping keys withMissingRequiredDataMapping.However, the validator's user-facing error messages referenced only the SDK names. So a caller who correctly supplied
expected_actions(empty/None) got:— naming a parameter (
ground_truth) they never used and that the service does not accept as a mapping key. Confusing UX/consistency bug.Fix
Update every user-facing validation message in
TaskNavigationEfficiencyValidatorthat namesresponseorground_truthto also cite the accepted alias:'ground_truth'→'ground_truth' (also accepted as 'expected_actions')'response'→'response' (also accepted as 'actions')Applies to the required/empty, type-mismatch, and empty-list messages. Message text only — no changes to error categories, validation logic, or control flow.
Notes
Bugs Fixednote to the unreleased CHANGELOG section.tests/unittests/test_common_validators.pyassert onErrorCategory/pytest.raises(EvaluationException), not on message substrings, so they remain green.