docs: fix evaluator variable mappings that silently break judge scores#31
Open
tkykenmt wants to merge 1 commit into
Open
docs: fix evaluator variable mappings that silently break judge scores#31tkykenmt wants to merge 1 commit into
tkykenmt wants to merge 1 commit into
Conversation
Verified two ways: by inspecting the compiled judge prompts in the
langfuse-llm-as-a-judge execution traces on Langfuse Cloud, and by
reproducing locally with jsonpath-plus@10.3.0 invoked the same way as
packages/shared/src/features/evals/utilities.ts (eval: false).
- 06 Experiments (Correctness, learner + instructor): the mapping used
$.messages[-1].content for {{query}}, but the evaluator's JsonPath
engine does not resolve bare negative indexes - the variable silently
compiles to empty and the judge returns a plausible-looking score with
no query. Restore the slice form $.messages[-1:].content, which langfuse#15
had rewritten to the bare form while renaming the table columns. Also
warn about leading whitespace in JsonPaths (saved as-is, resolves to
nothing) and add a tip to debug via the judge execution trace.
- 04 Monitoring (Out-of-Scope Request, learner + instructor):
{{last_user_message}} was mapped to $.messages[-1:].content, but this
evaluator targets the final generation of the tool-calling loop, whose
input transcript ends with tool results ([system, user, assistant
tool-calls, tool result]). The judge received the tool payload as the
user message and mis-scored obvious out-of-scope requests as in-scope.
Map to $.messages[1].content instead (the transcript always starts
[system, user, ...]). The instructor notes already listed this pitfall
under Watch for; both docs now carry the fix.
- 06 Experiments (keyword_overlap): note that the out-of-scope items'
expectedKeywords test literal wording, so a good refusal phrased
differently can score 0 - read the answer and compare with correctness.
9e59501 to
ab37a34
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #30
Summary
Fixes three evaluator variable mappings in the workshop docs that break silently — the judge runs with an empty (or wrong) variable and still returns a plausible-looking score.
$.messages[-1:].contentfor{{query}}. Clarify experiment evaluator variable mapping #15 rewrote the table from the slice form to a bare$.messages[-1].contentwhile renaming the columns, and the evaluator's JsonPath engine (jsonpath-plus,eval: false) silently resolves bare negative indexes to nothing. Also adds warnings about leading whitespace in JsonPaths and a tip to debug via thelangfuse-llm-as-a-judgeexecution traces.{{last_user_message}}to$.messages[1].content. The evaluator targets the final generation of the tool-calling loop, whose input ends with tool results, so the[-1:]slice handed the judge the tool payload and obvious out-of-scope requests were scored as in-scope. The instructor notes already listed this pitfall; both docs now carry the fix.expectedKeywordstest literal wording, so a good refusal phrased differently can score 0 on them.Verification
Confirmed two ways:
langfuse-llm-as-a-judgeexecution traces before/after each mapping change (emptyQuery:/Ground truth:before; filled and scoring correctly after).jsonpath-plus@10.3.0invoked the same way aspackages/shared/src/features/evals/utilities.ts.