Skip to content

docs: fix evaluator variable mappings that silently break judge scores#31

Open
tkykenmt wants to merge 1 commit into
langfuse:mainfrom
tkykenmt:fix/evaluator-mapping-docs
Open

docs: fix evaluator variable mappings that silently break judge scores#31
tkykenmt wants to merge 1 commit into
langfuse:mainfrom
tkykenmt:fix/evaluator-mapping-docs

Conversation

@tkykenmt

Copy link
Copy Markdown

Fixes #30

Summary

Fixes three evaluator variable mappings in the workshop docs that break silently — the judge runs with an empty (or wrong) variable and still returns a plausible-looking score.

  • 06 Experiments (Correctness, learner + instructor): restore the slice form $.messages[-1:].content for {{query}}. Clarify experiment evaluator variable mapping #15 rewrote the table from the slice form to a bare $.messages[-1].content while renaming the columns, and the evaluator's JsonPath engine (jsonpath-plus, eval: false) silently resolves bare negative indexes to nothing. Also adds warnings about leading whitespace in JsonPaths and a tip to debug via the langfuse-llm-as-a-judge execution traces.
  • 04 Monitoring (Out-of-Scope Request, learner + instructor): map {{last_user_message}} to $.messages[1].content. The evaluator targets the final generation of the tool-calling loop, whose input ends with tool results, so the [-1:] slice handed the judge the tool payload and obvious out-of-scope requests were scored as in-scope. The instructor notes already listed this pitfall; both docs now carry the fix.
  • 06 Experiments (keyword_overlap): note that the out-of-scope items' expectedKeywords test literal wording, so a good refusal phrased differently can score 0 on them.

Verification

Confirmed two ways:

  1. On Langfuse Cloud, by inspecting the compiled judge prompts in the langfuse-llm-as-a-judge execution traces before/after each mapping change (empty Query: / Ground truth: before; filled and scoring correctly after).
  2. Standalone, with jsonpath-plus@10.3.0 invoked the same way as packages/shared/src/features/evals/utilities.ts.

@CLAassistant

CLAassistant commented Jul 13, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Verified two ways: by inspecting the compiled judge prompts in the
langfuse-llm-as-a-judge execution traces on Langfuse Cloud, and by
reproducing locally with jsonpath-plus@10.3.0 invoked the same way as
packages/shared/src/features/evals/utilities.ts (eval: false).

- 06 Experiments (Correctness, learner + instructor): the mapping used
  $.messages[-1].content for {{query}}, but the evaluator's JsonPath
  engine does not resolve bare negative indexes - the variable silently
  compiles to empty and the judge returns a plausible-looking score with
  no query. Restore the slice form $.messages[-1:].content, which langfuse#15
  had rewritten to the bare form while renaming the table columns. Also
  warn about leading whitespace in JsonPaths (saved as-is, resolves to
  nothing) and add a tip to debug via the judge execution trace.

- 04 Monitoring (Out-of-Scope Request, learner + instructor):
  {{last_user_message}} was mapped to $.messages[-1:].content, but this
  evaluator targets the final generation of the tool-calling loop, whose
  input transcript ends with tool results ([system, user, assistant
  tool-calls, tool result]). The judge received the tool payload as the
  user message and mis-scored obvious out-of-scope requests as in-scope.
  Map to $.messages[1].content instead (the transcript always starts
  [system, user, ...]). The instructor notes already listed this pitfall
  under Watch for; both docs now carry the fix.

- 06 Experiments (keyword_overlap): note that the out-of-scope items'
  expectedKeywords test literal wording, so a good refusal phrased
  differently can score 0 - read the answer and compare with correctness.
@tkykenmt tkykenmt force-pushed the fix/evaluator-mapping-docs branch from 9e59501 to ab37a34 Compare July 14, 2026 03:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs: evaluator variable mappings in 04/06 silently break judge scores (bare negative index, tool-result as last_user_message)

2 participants