OpenJudge — LLM Evaluation Pipeline

What it does

OpenJudge provides an architecture and tooling to create repeatable evaluation pipelines for LLM applications. It standardises graders (LLM-based or function-based), orchestrates batch runs with a GradingRunner, aggregates scores, and provides analyzers for win‑rates and validation metrics.

When to use it

Use this skill when you need to compare model outputs, validate responses against references, auto‑generate graders from labeled data, or produce statistics for model selection and reward modelling. Ideal for research, alignment evaluations, and production QA.

What's included

• Scripts: no top-level scripts listed (has_scripts=false) but companion docs exist in the repo (pipeline.md, graders.md, analyzer.md, generator.md).
• References: additional docs in sibling files describe grader configuration and analysis patterns.
• Instructions: quickstart example demonstrating model config, grader instantiation, and running GradingRunner over a dataset, plus details on result handling and model configuration.

Compatible agents

Best for developer-facing agents and tooling that can run Python code and call model APIs (OpenAI‑compatible endpoints, local LLM endpoints). Works with agents that support programmatic evaluation workflows.