Skip to content

Feature/evals dod#175

Merged
reidbaker merged 4 commits into
flutter:mainfrom
reidbaker:feature/evals-dod
Jul 14, 2026
Merged

Feature/evals dod#175
reidbaker merged 4 commits into
flutter:mainfrom
reidbaker:feature/evals-dod

Conversation

@reidbaker-agent

Copy link
Copy Markdown
Contributor

Add proof of concept evals to definition-of-done.

#174

Follow up work coming for a better way to ignore directories because that introduced the pattern of rules based custom arguments.

Pre-launch Checklist

  • I read the Contributor Guide and followed the process outlined there for submitting PRs.
  • I read the Tree Hygiene wiki page, which explains my responsibilities.
  • I read the Flutter Style Guide recently, and have followed its advice.
  • I signed the CLA.
  • I listed at least one issue that this PR fixes in the description above.
  • I updated/added relevant documentation (doc comments with ///).
  • I added new tests to check the change I am making, or this PR is test-exempt.
  • All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a rubric-based LLM-as-a-judge evaluation system for agent skills, adding documentation, ignore rules, and a test suite for the definition-of-done skill. Feedback highlights that the new .gitignore rules for evals directories are redundant or ineffective due to Git's directory traversal behavior. Additionally, the evaluation prompt path in evals.json should be updated to use the correct relative path from the repository root to prevent execution errors.

Comment on lines +9 to +12
# Keep evals inside un-ignored skill folders
!**/evals/
!**/evals/**

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

In Git, if a parent directory is ignored (due to the * rule at the top), Git will not traverse into it. Therefore, !**/evals/ and !**/evals/** will not work for any third-party/ignored skill folders because their parent directories are excluded.

For the explicitly unignored skill folders (like !definition-of-done/), all of their contents (including evals/) are already unignored by default, making these rules redundant for them.

Since these rules are either redundant or ineffective, they should be removed to keep the .gitignore clean.

Comment thread tool/dart_skills_lint/.agents/skills/definition-of-done/evals/evals.json Outdated
reidbaker and others added 2 commits July 13, 2026 19:58
…evals.json

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
@reidbaker reidbaker merged commit 7b3fb9e into flutter:main Jul 14, 2026
14 checks passed
@reidbaker reidbaker deleted the feature/evals-dod branch July 14, 2026 00:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants