feat(studio): agent eval fileset descriptions, reusable eval configs#657
Merged
Conversation
nv-odrulea
force-pushed
the
reusable-eval-configs/odrulea
branch
from
ceea36c to
07fd566
Compare
Contributor
📝 WalkthroughWalkthroughChangesFileset picker configuration
Agent evaluation submission
Studio dataset display
Sequence Diagram(s)sequenceDiagram
participant User
participant SubmitEvaluationModal
participant EvaluationJobs
participant FilesetAPI
participant EvaluateAPI
User->>SubmitEvaluationModal: select agent and evaluation configuration
SubmitEvaluationModal->>EvaluationJobs: fetch prior evaluation jobs
EvaluationJobs-->>SubmitEvaluationModal: return existing configurations
SubmitEvaluationModal->>FilesetAPI: create or seed configuration fileset
SubmitEvaluationModal->>FilesetAPI: create output fileset
SubmitEvaluationModal->>EvaluateAPI: submit evaluation request
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
web/packages/common/src/components/UploadModal/DatasetUploader/Select.tsx (1)
61-97: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick winGuard against stale dataset fetches. If the user switches datasets before
filesListFilesetFilesreturns, the older response can still overwrite the newer selection’sSET_FILES/TOGGLE_FILE_SELECTIONstate. Track the active request or bail out when the selected dataset no longer matches.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@web/packages/common/src/components/UploadModal/DatasetUploader/Select.tsx` around lines 61 - 97, Update handleDatasetSelect to guard the asynchronous filesListFilesetFiles response against stale dataset selections: track the active request or verify that the current selected dataset still matches fileset before dispatching SET_FILES or TOGGLE_FILE_SELECTION. Apply the same guard to related success/error state updates so an older request cannot overwrite the newer dataset’s state.
🧹 Nitpick comments (2)
web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/SubmitEvaluationModal.tsx (2)
131-136: 🚀 Performance & Scalability | 🔵 TrivialFull job-history fetch just to populate a dropdown.
fetchAgentEvalJobspaginates the entire workspace's evaluation job history (unboundedwhileloop in api.ts). It's often cache-warm here since the query key matches the list route's, but any standalone usage of this modal (the docstring at lines 98-101 mentions anAgentPanelcontext) triggers this full fetch purely to build the "existing eval config" dropdown. Consider a lighter, agent-scoped endpoint/query as job volume grows.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/SubmitEvaluationModal.tsx` around lines 131 - 136, The SubmitEvaluationModal query currently loads all workspace evaluation jobs via fetchAgentEvalJobs solely for the existing-config dropdown. Replace this with a lightweight agent-scoped query or endpoint that retrieves only configurations relevant to the selected agent, while preserving the dropdown’s existing behavior and loading state.
164-178: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winSilent catch-all on output-fileset pre-create.
Any error (not just an expected 409) is swallowed with no logging. Since this is best-effort and won't block submission, at least logging the failure would preserve visibility into recurring pre-create issues.
🔍 Proposed tweak
} catch { - // Best-effort: if this fails, the job still auto-creates the output - // fileset (without a description). Don't block submission. + // Best-effort: if this fails, the job still auto-creates the output + // fileset (without a description). Don't block submission, but log + // it so recurring failures are visible. }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/SubmitEvaluationModal.tsx` around lines 164 - 178, Update the catch block around filesCreateFileset in the outputFileset pre-creation flow to capture the error and log it with relevant context, such as the fileset name, while preserving the best-effort behavior that allows submission to continue.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@web/packages/common/src/components/UploadModal/DatasetUploader/Select.tsx`:
- Around line 81-89: Update the auto-selection logic in the Select component’s
acceptable-file predicate to normalize acceptableFileTypes entries to lowercase
before comparing them with the already lowercased ext, matching
SimpleFilesTable’s allowedExtensions behavior.
In
`@web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/SubmitEvaluationModal.tsx`:
- Around line 57-80: Add format validation for the create-mode newName field in
submitEvaluationSchema before it is passed through buildSubmitSpec as
evalConfigFileset. Validate the platform-supported fileset naming rules inline,
while preserving the existing required-name check and applying the format
constraint only when evalConfig is CREATE_NEW.
In
`@web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/submitEvaluationSpec.ts`:
- Around line 65-71: Update the non-CREATE_NEW branch in submitEvaluationSpec to
validate the existingConfigs lookup result before constructing the submission.
If formData.evalConfig is missing, throw a clear error instead of defaulting
evalConfig to an empty string; preserve the existing return structure for
successful lookups.
- Around line 72-82: Update buildSubmitSpec’s MODE_FILESET branch to validate
datasetFile before passing it to parseFilesetLocation, then validate the parsed
result and its objectPath before accessing fields. Remove both non-null
assertions and throw a descriptive error when either validation fails, while
preserving the existing evalConfig and evalConfigFileset mapping for valid
fileset input.
---
Outside diff comments:
In `@web/packages/common/src/components/UploadModal/DatasetUploader/Select.tsx`:
- Around line 61-97: Update handleDatasetSelect to guard the asynchronous
filesListFilesetFiles response against stale dataset selections: track the
active request or verify that the current selected dataset still matches fileset
before dispatching SET_FILES or TOGGLE_FILE_SELECTION. Apply the same guard to
related success/error state updates so an older request cannot overwrite the
newer dataset’s state.
---
Nitpick comments:
In
`@web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/SubmitEvaluationModal.tsx`:
- Around line 131-136: The SubmitEvaluationModal query currently loads all
workspace evaluation jobs via fetchAgentEvalJobs solely for the existing-config
dropdown. Replace this with a lightweight agent-scoped query or endpoint that
retrieves only configurations relevant to the selected agent, while preserving
the dropdown’s existing behavior and loading state.
- Around line 164-178: Update the catch block around filesCreateFileset in the
outputFileset pre-creation flow to capture the error and log it with relevant
context, such as the fileset name, while preserving the best-effort behavior
that allows submission to continue.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 24ea5681-29d4-49b2-aeb3-eb2fb5cb8e2f
📒 Files selected for processing (17)
web/packages/common/src/components/DatasetFileSelect/ControlledDatasetFileSelect.tsxweb/packages/common/src/components/DatasetFileSelect/DatasetFileSelect.tsxweb/packages/common/src/components/UploadModal/Context/useUploadModalReducer.tsweb/packages/common/src/components/UploadModal/DatasetUploader/Select.test.tsxweb/packages/common/src/components/UploadModal/DatasetUploader/Select.tsxweb/packages/common/src/components/UploadModal/InlineUploadPicker.tsxweb/packages/common/src/components/UploadModal/SimpleFilesTable.tsxweb/packages/common/src/components/UploadModal/types.tsweb/packages/common/src/utils/generateDefaultName.tsweb/packages/studio/public/sample-agents/email-phishing-analyzer/eval.ymlweb/packages/studio/src/components/DatasetsTable/columns.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/AgentEvaluationsListRoute.test.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/AgentEvaluationsListRoute.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/SubmitEvaluationModal.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/submitEvaluationSpec.test.tsweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/submitEvaluationSpec.tsweb/packages/studio/src/routes/agents/AgentSuggestionsRoute/api.ts
Contributor
|
… for config Signed-off-by: Octavian Drulea <odrulea@nvidia.com>
Signed-off-by: Octavian Drulea <odrulea@nvidia.com>
nv-odrulea
force-pushed
the
reusable-eval-configs/odrulea
branch
from
fda9b55 to
c5721bb
Compare
Contributor
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/SubmitEvaluationModal.tsx (1)
295-301: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick winIsolate
buildSubmitSpecfrom the swallowing catch.
buildSubmitSpecruns synchronously inside the same try assubmitEvaluation. If it throws (e.g.parseFilesetLocation(...)!orgetSampleAgenton a bad key), the emptycatchswallows it silently —submitErrornever gets set (it's a mutation-only error), soerrorTextnever renders and the user gets no feedback at all.🐛 Proposed fix
const onSubmit: SubmitHandler<SubmitEvaluationFormData> = async (formData) => { + let spec: SubmitSpec; + try { + spec = buildSubmitSpec(formData, existingConfigs); + } catch (err) { + toast.error(err instanceof Error ? err.message : 'Failed to build evaluation spec'); + return; + } try { - await submitEvaluation(buildSubmitSpec(formData, existingConfigs)); + await submitEvaluation(spec); } catch { // Error rendered via errorText prop. } };Based on learnings, avoid "empty"
catch {}blocks inonSubmit/submit handlers that would swallow errors frombuildSubmitSpec— isolate it outside thetryso build/runtime failures don't cause a silent UX regression.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/SubmitEvaluationModal.tsx` around lines 295 - 301, Move the synchronous buildSubmitSpec(formData, existingConfigs) call outside the try block in onSubmit, then pass its result into submitEvaluation within the existing error-handling scope. Keep the catch for submitEvaluation failures and preserve the errorText behavior, while allowing buildSubmitSpec failures to propagate instead of being silently swallowed.Source: Learnings
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In
`@web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/SubmitEvaluationModal.tsx`:
- Around line 295-301: Move the synchronous buildSubmitSpec(formData,
existingConfigs) call outside the try block in onSubmit, then pass its result
into submitEvaluation within the existing error-handling scope. Keep the catch
for submitEvaluation failures and preserve the errorText behavior, while
allowing buildSubmitSpec failures to propagate instead of being silently
swallowed.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 896c2d17-5937-43d2-ba41-d69aca93cc55
📒 Files selected for processing (16)
web/packages/common/src/components/DatasetFileSelect/ControlledDatasetFileSelect.tsxweb/packages/common/src/components/DatasetFileSelect/DatasetFileSelect.tsxweb/packages/common/src/components/UploadModal/Context/useUploadModalReducer.tsweb/packages/common/src/components/UploadModal/DatasetUploader/Select.test.tsxweb/packages/common/src/components/UploadModal/DatasetUploader/Select.tsxweb/packages/common/src/components/UploadModal/InlineUploadPicker.tsxweb/packages/common/src/components/UploadModal/SimpleFilesTable.tsxweb/packages/common/src/components/UploadModal/types.tsweb/packages/studio/public/sample-agents/email-phishing-analyzer/eval.ymlweb/packages/studio/src/components/DatasetsTable/columns.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/AgentEvaluationsListRoute.test.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/AgentEvaluationsListRoute.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/SubmitEvaluationModal.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/submitEvaluationSpec.test.tsweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/submitEvaluationSpec.tsweb/packages/studio/src/routes/agents/AgentSuggestionsRoute/api.ts
🚧 Files skipped from review as they are similar to previous changes (14)
- web/packages/studio/src/routes/agents/AgentEvaluationsRoute/AgentEvaluationsListRoute.test.tsx
- web/packages/studio/public/sample-agents/email-phishing-analyzer/eval.yml
- web/packages/studio/src/routes/agents/AgentEvaluationsRoute/AgentEvaluationsListRoute.tsx
- web/packages/studio/src/routes/agents/AgentSuggestionsRoute/api.ts
- web/packages/common/src/components/DatasetFileSelect/ControlledDatasetFileSelect.tsx
- web/packages/studio/src/components/DatasetsTable/columns.tsx
- web/packages/common/src/components/UploadModal/types.ts
- web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/submitEvaluationSpec.test.ts
- web/packages/common/src/components/UploadModal/DatasetUploader/Select.test.tsx
- web/packages/common/src/components/UploadModal/InlineUploadPicker.tsx
- web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/submitEvaluationSpec.ts
- web/packages/common/src/components/UploadModal/SimpleFilesTable.tsx
- web/packages/common/src/components/UploadModal/Context/useUploadModalReducer.ts
- web/packages/common/src/components/UploadModal/DatasetUploader/Select.tsx
steramae-nvidia
approved these changes
Jul 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Agent Eval: create new config from example
Agent Eval: create new config from existing fileset
Agent Eval: reuse existing agent eval config
Agent Eval: the 3 Filesets that are created, 2 now have descriptions
Run Agent Evaluation modal (
SubmitEvaluationModal.tsx, newsubmitEvaluationSpec.ts)eval jobs) or Create new eval config. Preselects the latest config for the selected
agent.
opened from the agent side panel).
adjective-animalfileset name(
generateEvalConfigName), editable ("New Fileset Name").purpose=generic, labelled "Fileset",auto-selects the first root-level
.yml/.yamlso you don't have to drill in.buildSubmitSpec,evaluateRequestBody,generateOutputFilesetName,evalOutputDescription).Fileset descriptions (no plugin change)
—
Agent Evaluation output, agent: <agent>, config: <config-fileset>. The job'sfileset_auto_createno-ops on existence, so the description sticks. Best-effort (neverblocks submission).
Agent Evaluation Config(viaensureEvalConfigFileset, only when Studio actually creates it).Output layout (
sample-agents/email-phishing-analyzer/eval.yml)output_dir: eval/agent→output_dir: .so eval artifacts land at the output filesetroot instead of a nested
eval/agent/subfolder.Filesets list (
DatasetsTable/columns.tsx)Shared UploadModal / file picker (
packages/common/.../UploadModal,DatasetFileSelect)filesetPurpose,datasetLabel,autoSelectFirstAcceptable. All default to today's behavior (backward compatible).SimpleFilesTable: Name column now fills the row width while Size stays a fixed 120px(fixes a width-distribution issue in the virtualized table); the "only
.yml/.yamlcanbe selected" hint is suppressed once a valid file is selected. Still virtualized.
List route CTA (
AgentEvaluationsListRoute.tsx)Out of scope / not in this PR
plugins/nemo-agentschanges.job-fileset-*clutter in the Filesets list (tracked separately).Summary by CodeRabbit