Leaderboard: red-hat-data-services/agentic-starter-kits (63.7/100 - Silver)#448
Open
tarun-etikala wants to merge 1 commit into
Conversation
📝 WalkthroughWalkthroughThis PR adds an assessment JSON report for the Red Hat Data Services agentic starter kit repository. The report captures tool metadata, repository identification, overall scoring, and detailed findings across multiple evaluated attributes with evidence and remediation guidance. ChangesAssessment Report
Suggested labels
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
✨ Simplify code
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@submissions/red-hat-data-services/agentic-starter-kits/2026-05-19T18-40-31-assessment.json`:
- Around line 474-531: Update the concise_documentation finding so the pass/fail
logic is unambiguous: either expand the "threshold" field for attribute id
"concise_documentation" to enumerate the structural requirements (e.g., "<500
lines, 3-5 headings per 100 lines, <10 lines per paragraph") or change the
"status" and/or add a new "substatus" field that clarifies which criterion
failed (length vs. structure) and references the measured_value and evidence
entries; ensure any change updates the same object containing
"attribute.id":"concise_documentation", "status", "threshold", "measured_value",
and "evidence" so consumers can clearly determine why the item is "fail".
- Around line 533-553: The report entry for the inline_documentation attribute
is incorrect: the field "status" is set to "pass" while "measured_value" is
"69.1%" which is below the "threshold" of "≥80%"; update the JSON object for the
attribute with id "inline_documentation" to set "status" to "fail" (and
optionally adjust "remediation" to a suggested fix) so the status matches the
measured_value vs threshold comparison that is performed elsewhere.
- Around line 80-100: The JSON entry for the "type_annotations" attribute
currently marks "status": "pass" while its measured_value is 63.0% which is
below the threshold "≥80%"; change the "status" field for the attribute with
"id": "type_annotations" (and/or "name": "Type Annotations") from "pass" to
"fail" so the reported status matches the evidence (e.g., the object showing
"Typed functions: 354/562", "Coverage: 63.0%", "threshold": "≥80%").
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 94467c91-4586-4d20-9322-b48075b0262d
📒 Files selected for processing (1)
submissions/red-hat-data-services/agentic-starter-kits/2026-05-19T18-40-31-assessment.json
Comment on lines
+80
to
+100
| { | ||
| "attribute": { | ||
| "id": "type_annotations", | ||
| "name": "Type Annotations", | ||
| "category": "Code Quality", | ||
| "tier": 1, | ||
| "description": "Type hints in function signatures", | ||
| "criteria": ">80% of functions have type annotations", | ||
| "default_weight": 0.1 | ||
| }, | ||
| "status": "pass", | ||
| "score": 78.73665480427046, | ||
| "measured_value": "63.0%", | ||
| "threshold": "\u226580%", | ||
| "evidence": [ | ||
| "Typed functions: 354/562", | ||
| "Coverage: 63.0%" | ||
| ], | ||
| "remediation": null, | ||
| "error_message": null | ||
| }, |
There was a problem hiding this comment.
Critical: Status "pass" contradicts measured value.
The type_annotations finding is marked as "status": "pass" but the measured coverage is 63.0%, which is below the stated threshold of ≥80%. This is a data integrity error - the status should be "fail" to match the evidence.
🐛 Proposed fix
"status": "pass",
- "score": 78.73665480427046,
+ "score": 78.73,
"measured_value": "63.0%",
"threshold": "≥80%",Change line 90:
- "status": "pass",
+ "status": "fail",📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| { | |
| "attribute": { | |
| "id": "type_annotations", | |
| "name": "Type Annotations", | |
| "category": "Code Quality", | |
| "tier": 1, | |
| "description": "Type hints in function signatures", | |
| "criteria": ">80% of functions have type annotations", | |
| "default_weight": 0.1 | |
| }, | |
| "status": "pass", | |
| "score": 78.73665480427046, | |
| "measured_value": "63.0%", | |
| "threshold": "\u226580%", | |
| "evidence": [ | |
| "Typed functions: 354/562", | |
| "Coverage: 63.0%" | |
| ], | |
| "remediation": null, | |
| "error_message": null | |
| }, | |
| { | |
| "attribute": { | |
| "id": "type_annotations", | |
| "name": "Type Annotations", | |
| "category": "Code Quality", | |
| "tier": 1, | |
| "description": "Type hints in function signatures", | |
| "criteria": ">80% of functions have type annotations", | |
| "default_weight": 0.1 | |
| }, | |
| "status": "fail", | |
| "score": 78.73, | |
| "measured_value": "63.0%", | |
| "threshold": "\u226580%", | |
| "evidence": [ | |
| "Typed functions: 354/562", | |
| "Coverage: 63.0%" | |
| ], | |
| "remediation": null, | |
| "error_message": null | |
| }, |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@submissions/red-hat-data-services/agentic-starter-kits/2026-05-19T18-40-31-assessment.json`
around lines 80 - 100, The JSON entry for the "type_annotations" attribute
currently marks "status": "pass" while its measured_value is 63.0% which is
below the threshold "≥80%"; change the "status" field for the attribute with
"id": "type_annotations" (and/or "name": "Type Annotations") from "pass" to
"fail" so the reported status matches the evidence (e.g., the object showing
"Typed functions: 354/562", "Coverage: 63.0%", "threshold": "≥80%").
Comment on lines
+474
to
+531
| { | ||
| "attribute": { | ||
| "id": "concise_documentation", | ||
| "name": "Concise Documentation", | ||
| "category": "Documentation", | ||
| "tier": 2, | ||
| "description": "Documentation maximizes information density while minimizing token consumption", | ||
| "criteria": "README <500 lines with clear structure, bullet points over prose", | ||
| "default_weight": 0.03 | ||
| }, | ||
| "status": "fail", | ||
| "score": 72.0, | ||
| "measured_value": "173 lines, 14 headings, 18 bullets", | ||
| "threshold": "<500 lines, structured format", | ||
| "evidence": [ | ||
| "README length: 173 lines (excellent)", | ||
| "Heading density: 8.1 per 100 lines (target: 3-5)", | ||
| "1 paragraphs exceed 10 lines (walls of text)" | ||
| ], | ||
| "remediation": { | ||
| "summary": "Make documentation more concise and structured", | ||
| "steps": [ | ||
| "Break long README into multiple documents (docs/ directory)", | ||
| "Add clear Markdown headings (##, ###) for structure", | ||
| "Convert prose paragraphs to bullet points where possible", | ||
| "Add table of contents for documents >100 lines", | ||
| "Use code blocks instead of describing commands in prose", | ||
| "Move detailed content to wiki or docs/, keep README focused" | ||
| ], | ||
| "tools": [], | ||
| "commands": [ | ||
| "# Check README length", | ||
| "wc -l README.md", | ||
| "", | ||
| "# Count headings", | ||
| "grep -c '^#' README.md" | ||
| ], | ||
| "examples": [ | ||
| "# Good: Concise with structure\n\n## Quick Start\n```bash\npip install -e .\nagentready assess .\n```\n\n## Features\n- Fast repository scanning\n- HTML and Markdown reports\n- 25 agent-ready attributes\n\n## Documentation\nSee [docs/](docs/) for detailed guides.\n", | ||
| "# Bad: Verbose prose\n\nThis project is a tool that helps you assess your repository\nagainst best practices for AI-assisted development. It works by\nscanning your codebase and checking for various attributes that\nmake repositories more effective when working with AI coding\nassistants like Claude Code...\n\n[Many more paragraphs of prose...]\n" | ||
| ], | ||
| "citations": [ | ||
| { | ||
| "source": "ArXiv", | ||
| "title": "LongCodeBench: Evaluating Coding LLMs at 1M Context Windows", | ||
| "url": "https://arxiv.org/abs/2501.00343", | ||
| "relevance": "Research showing performance degradation with long contexts" | ||
| }, | ||
| { | ||
| "source": "Markdown Guide", | ||
| "title": "Basic Syntax", | ||
| "url": "https://www.markdownguide.org/basic-syntax/", | ||
| "relevance": "Best practices for Markdown formatting" | ||
| } | ||
| ] | ||
| }, | ||
| "error_message": null | ||
| }, |
There was a problem hiding this comment.
Major: Threshold description incomplete for pass/fail logic.
The concise_documentation finding shows "status": "fail" despite the README being 173 lines (well under the 500-line threshold). The evidence reveals the failure is due to heading density (8.1 vs target 3-5) and wall-of-text paragraphs, but the threshold field only mentions "<500 lines, structured format" without quantifying "structured format" requirements. This makes the pass/fail determination unclear from the primary fields.
Consider either:
- Making the threshold more explicit:
"<500 lines, 3-5 headings per 100 lines, <10 lines per paragraph" - Or adjusting the status to reflect that length criteria are met but structure criteria are not
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@submissions/red-hat-data-services/agentic-starter-kits/2026-05-19T18-40-31-assessment.json`
around lines 474 - 531, Update the concise_documentation finding so the
pass/fail logic is unambiguous: either expand the "threshold" field for
attribute id "concise_documentation" to enumerate the structural requirements
(e.g., "<500 lines, 3-5 headings per 100 lines, <10 lines per paragraph") or
change the "status" and/or add a new "substatus" field that clarifies which
criterion failed (length vs. structure) and references the measured_value and
evidence entries; ensure any change updates the same object containing
"attribute.id":"concise_documentation", "status", "threshold", "measured_value",
and "evidence" so consumers can clearly determine why the item is "fail".
Comment on lines
+533
to
+553
| "attribute": { | ||
| "id": "inline_documentation", | ||
| "name": "Inline Documentation", | ||
| "category": "Documentation", | ||
| "tier": 2, | ||
| "description": "Function, class, and module-level documentation using language-specific conventions", | ||
| "criteria": "\u226580% of public functions/classes have docstrings", | ||
| "default_weight": 0.03 | ||
| }, | ||
| "status": "pass", | ||
| "score": 86.36363636363636, | ||
| "measured_value": "69.1%", | ||
| "threshold": "\u226580%", | ||
| "evidence": [ | ||
| "Documented items: 494/715", | ||
| "Coverage: 69.1%", | ||
| "Moderate docstring coverage" | ||
| ], | ||
| "remediation": null, | ||
| "error_message": null | ||
| }, |
There was a problem hiding this comment.
Critical: Status "pass" contradicts measured value.
The inline_documentation finding is marked as "status": "pass" but the measured coverage is 69.1%, which is below the stated threshold of ≥80%. Like the type_annotations finding, this status should be "fail".
🐛 Proposed fix
- "status": "pass",
+ "status": "fail",
"score": 86.36363636363636,
"measured_value": "69.1%",
"threshold": "≥80%",📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| "attribute": { | |
| "id": "inline_documentation", | |
| "name": "Inline Documentation", | |
| "category": "Documentation", | |
| "tier": 2, | |
| "description": "Function, class, and module-level documentation using language-specific conventions", | |
| "criteria": "\u226580% of public functions/classes have docstrings", | |
| "default_weight": 0.03 | |
| }, | |
| "status": "pass", | |
| "score": 86.36363636363636, | |
| "measured_value": "69.1%", | |
| "threshold": "\u226580%", | |
| "evidence": [ | |
| "Documented items: 494/715", | |
| "Coverage: 69.1%", | |
| "Moderate docstring coverage" | |
| ], | |
| "remediation": null, | |
| "error_message": null | |
| }, | |
| "attribute": { | |
| "id": "inline_documentation", | |
| "name": "Inline Documentation", | |
| "category": "Documentation", | |
| "tier": 2, | |
| "description": "Function, class, and module-level documentation using language-specific conventions", | |
| "criteria": "\u226580% of public functions/classes have docstrings", | |
| "default_weight": 0.03 | |
| }, | |
| "status": "fail", | |
| "score": 86.36363636363636, | |
| "measured_value": "69.1%", | |
| "threshold": "\u226580%", | |
| "evidence": [ | |
| "Documented items: 494/715", | |
| "Coverage: 69.1%", | |
| "Moderate docstring coverage" | |
| ], | |
| "remediation": null, | |
| "error_message": null | |
| }, |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@submissions/red-hat-data-services/agentic-starter-kits/2026-05-19T18-40-31-assessment.json`
around lines 533 - 553, The report entry for the inline_documentation attribute
is incorrect: the field "status" is set to "pass" while "measured_value" is
"69.1%" which is below the "threshold" of "≥80%"; update the JSON object for the
attribute with id "inline_documentation" to set "status" to "fail" (and
optionally adjust "remediation" to a suggested fix) so the status matches the
measured_value vs threshold comparison that is performed elsewhere.
Contributor
📈 Test Coverage Report
Coverage calculated from unit tests only |
3 tasks
Contributor
|
CI is failing. This appears to be a defect in the CI. I will fix it now. |
jwm4
approved these changes
May 20, 2026
Contributor
jwm4
left a comment
jwm4
left a comment
There was a problem hiding this comment.
The submission looks good to me.
Author
Do you want me to rebase this PR? |
Contributor
|
@tarun-etikala , I guess that's worth a try. I am not sure why it is still failing. |
Score: 63.7/100 (Silver) Repository: https://github.com/red-hat-data-services/agentic-starter-kits
tarun-etikala
pushed a commit
to tarun-etikala/agentready
that referenced
this pull request
May 20, 2026
The collaborator API requires admin access on the target repo, which the Actions token never has for external repos. This caused all leaderboard submissions for org-owned repos to fail CI with a false "no access" error. Align the GitHub path with the non-GitHub fallback: warn maintainers to verify manually rather than blocking the PR. Fixes CI for ambient-code#448. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
tarun-etikala
force-pushed
the
leaderboard-red-hat-data-services-agentic-starter-kits-2026-05-19T18-40-31
branch
from
2ab8bad to
e256cba
Compare
Author
Done |
There was a problem hiding this comment.
♻️ Duplicate comments (3)
submissions/red-hat-data-services/agentic-starter-kits/2026-05-19T18-40-31-assessment.json (3)
474-531:⚠️ Potential issue | 🟠 Major | ⚡ Quick winMajor: Threshold description incomplete for pass/fail logic.
The
concise_documentationfinding shows"status": "fail"despite the README being 173 lines (well under the 500-line threshold). The evidence reveals the failure is due to heading density (8.1 vs target 3-5) and wall-of-text paragraphs, but thethresholdfield only mentions"<500 lines, structured format"without quantifying "structured format" requirements. This makes the pass/fail determination unclear from the primary fields.Consider either:
- Making the threshold more explicit:
"<500 lines, 3-5 headings per 100 lines, <10 lines per paragraph"- Or adjusting the status to reflect that length criteria are met but structure criteria are not
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@submissions/red-hat-data-services/agentic-starter-kits/2026-05-19T18-40-31-assessment.json` around lines 474 - 531, The threshold field for the "concise_documentation" attribute is too vague, causing a misleading "fail" status; update the attribute with explicit structured criteria (e.g., set "threshold" to "<500 lines, 3-5 headings per 100 lines, <10 lines per paragraph") and/or add a new structured_criteria subfield that lists the quantified checks, ensure measured_value stays aligned, and if you prefer to preserve the existing wording adjust "status" logic to reflect which specific sub-criterion failed (e.g., add "status_details" listing length:pass, heading_density:fail, paragraph_length:fail) so callers can unambiguously determine pass/fail for attribute "concise_documentation".
533-553:⚠️ Potential issue | 🔴 Critical | ⚡ Quick winCritical: Status "pass" contradicts measured value.
The
inline_documentationfinding is marked as"status": "pass"but the measured coverage is 69.1%, which is below the stated threshold of ≥80%. Like the type_annotations finding, this status should be"fail".🐛 Proposed fix
- "status": "pass", + "status": "fail", "score": 86.36363636363636, "measured_value": "69.1%", "threshold": "≥80%",🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@submissions/red-hat-data-services/agentic-starter-kits/2026-05-19T18-40-31-assessment.json` around lines 533 - 553, The entry for the finding with attribute.id "inline_documentation" currently has status "pass" despite measured_value "69.1%" being below threshold "≥80%"; update the JSON so that the "status" field is set to "fail" for that object (and ensure the same validation logic that sets status for "type_annotations" is used: compare measured_value to threshold and set status accordingly), touching the fields "attribute.id", "measured_value", "threshold", and "status" in the offending object.
80-100:⚠️ Potential issue | 🔴 Critical | ⚡ Quick winCritical: Status "pass" contradicts measured value.
The
type_annotationsfinding is marked as"status": "pass"but the measured coverage is 63.0%, which is below the stated threshold of ≥80%. This is a data integrity error - the status should be"fail"to match the evidence.🐛 Proposed fix
- "status": "pass", + "status": "fail", "score": 78.73665480427046, "measured_value": "63.0%", "threshold": "≥80%",🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@submissions/red-hat-data-services/agentic-starter-kits/2026-05-19T18-40-31-assessment.json` around lines 80 - 100, The JSON entry for the "type_annotations" finding has an inconsistent status: the object with attribute.id "type_annotations" currently sets "status": "pass" while "measured_value": "63.0%" falls below the "threshold": "≥80%"; update the "status" field from "pass" to "fail" in that object (the JSON block containing "attribute": { "id": "type_annotations", ... }) so the status matches the evidence and any downstream consumers interpret the finding correctly.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Duplicate comments:
In
`@submissions/red-hat-data-services/agentic-starter-kits/2026-05-19T18-40-31-assessment.json`:
- Around line 474-531: The threshold field for the "concise_documentation"
attribute is too vague, causing a misleading "fail" status; update the attribute
with explicit structured criteria (e.g., set "threshold" to "<500 lines, 3-5
headings per 100 lines, <10 lines per paragraph") and/or add a new
structured_criteria subfield that lists the quantified checks, ensure
measured_value stays aligned, and if you prefer to preserve the existing wording
adjust "status" logic to reflect which specific sub-criterion failed (e.g., add
"status_details" listing length:pass, heading_density:fail,
paragraph_length:fail) so callers can unambiguously determine pass/fail for
attribute "concise_documentation".
- Around line 533-553: The entry for the finding with attribute.id
"inline_documentation" currently has status "pass" despite measured_value
"69.1%" being below threshold "≥80%"; update the JSON so that the "status" field
is set to "fail" for that object (and ensure the same validation logic that sets
status for "type_annotations" is used: compare measured_value to threshold and
set status accordingly), touching the fields "attribute.id", "measured_value",
"threshold", and "status" in the offending object.
- Around line 80-100: The JSON entry for the "type_annotations" finding has an
inconsistent status: the object with attribute.id "type_annotations" currently
sets "status": "pass" while "measured_value": "63.0%" falls below the
"threshold": "≥80%"; update the "status" field from "pass" to "fail" in that
object (the JSON block containing "attribute": { "id": "type_annotations", ...
}) so the status matches the evidence and any downstream consumers interpret the
finding correctly.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: c9d232d9-5f5f-4888-9eda-6e61528acfd0
📒 Files selected for processing (1)
submissions/red-hat-data-services/agentic-starter-kits/2026-05-19T18-40-31-assessment.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Leaderboard Submission
Repository: red-hat-data-services/agentic-starter-kits
Score: 63.7/100
Tier: Silver
Submitted by: @tarun-etikala
Validation Checklist
Automated validation will run on this PR.
Submitted via
agentready submitcommand.Summary by CodeRabbit