Agent Evaluation Layer

What it does

This skill enables an agent to implement a rigorous, rubric-based evaluation system for any AI agent project. It transforms qualitative 'vibes' into quantitative metrics by establishing a scoring framework and a programmatic harness to execute it.

When to use it

Use this skill when you need to:

• Measure the accuracy or quality of an agent's outputs against a specific rubric.
• Set up an 'LLM-as-a-judge' pipeline to automate grading.
• Compare multiple models, frameworks, or prompt versions (A/B testing) using a consistent benchmark.
• Track judge leniency and bias through human-graded reference cases.

What's included

• Instructions: Detailed workflow for designing rubrics, generating test cases, creating judge prompts, and building a Python eval harness.
• Evaluation Artifacts: Guidance on creating main.yaml rubrics and seed.yaml test cases.
• Metadata Contract: A standardized 7-field output format (latency, tokens, tool calls, etc.) to ensure cross-project compatibility.
• Reporting: Instructions for generating both per-subject Markdown reports and comprehensive HTML dashboards with leaderboards and radar charts.

Compatible agents

Designed primarily for Claude Code, but framework-agnostic and compatible with any agent capable of writing Python and managing a local workspace (e.g., Cursor, Codex).