This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment.
Automatically evaluate and compare multiple AI models or agents without pre-existing test data. Generates test queries from a task description, collects responses from all target endpoints, auto-generates evaluation rubrics, runs pairwise comparisons via a judge model, and produces win-rate rankings with reports and charts. Supports checkpoint resume, incremental endpoint addition, and judge model hot-swap. Use when the user asks to compare, benchmark, or rank multiple models or agents on a custom task, or run an arena-style evaluation. --- # Auto Arena Skill End-to-end automated model comparison using the OpenJudge `AutoArenaPipeline`: 1. **Generate queries** — LLM creates diverse test queries from task description 2. **Collect responses** — query all target endpoints concurrently 3. **Generate rubrics** — LLM produces evaluation criteria from task + sample queries 4. **Pairwise evaluation** — judge model compares every model pair (with position-bias swap) 5. **Analyze & rank** — compute win rates, win matrix, and rankings 6. **Report & charts** — Markdown report + win-rate bar chart + optional matrix heatmap ## Prerequisites ```bash # Install OpenJudge pip install py-openjudge # Extra dependency for auto_arena (chart generation) pip install matplotlib ``` ## Gather from user before running | Info | Required? | Notes | |------|-----------|-------| | Task description | Yes | What the models/agents should do (set in config YAML) | | Target endpoints | Yes | At least 2 OpenAI-compatible endpoints to compare | | Judge endpoint | Yes | Strong model for pairwise evaluation (e.g. `gpt-4`, `qwen-max`) | | API keys | Yes | Env vars: `OPENAI_API_KEY`, `DASHSCOPE_API_KEY`, etc. | | Number of queries | No | Default: `20` | | Seed queries | No | Example queries to guide generation style | | System prompts | No | Per-endpoint system prompts | | Output directory | No | Default: `./evaluation_results` | | Report language | No | `"zh"` (default) or `"en"` | ## Quick start ### CLI `
Critically analyze content, claims, or arguments with rigorous evaluation.
Use this when you need to EVALUATE OR IMPROVE or OPTIMIZE an existing LLM agent's output quality - including improving tool selection accuracy, answer quality, reducing costs, or fixing issues where the agent gives wrong/incomplete responses. Evaluates agents systematically using MLflow evaluation with datasets, scorers, and tracing. IMPORTANT - Always also load the instrumenting-with-mlflow-tracing skill before starting any work. Covers end-to-end evaluation workflow or individual components (tracing setup, dataset creation, scorer definition, evaluation execution).
Run a full Build + Style + Move + Write evaluation on a page — score each framework, produce a combined report out of /200 with prioritized recommendations across all four.
Analyze Inspect AI evaluation logs, understand EvalLog structure, extract samples, events, and scoring data using dataframes
skill-sample/ ├─ SKILL.md ⭐ Required: skill entry doc (purpose / usage / examples / deps) ├─ manifest.sample.json ⭐ Recommended: machine-readable metadata (index / validation / autofill) ├─ LICENSE.sample ⭐ Recommended: license & scope (open source / restriction / commercial) ├─ scripts/ │ └─ example-run.py ✅ Runnable example script for quick verification ├─ assets/ │ ├─ example-formatting-guide.md 🧩 Output conventions: layout / structure / style │ └─ example-template.tex 🧩 Templates: quickly generate standardized output └─ references/ 🧩 Knowledge base: methods / guides / best practices ├─ example-ref-structure.md 🧩 Structure reference ├─ example-ref-analysis.md 🧩 Analysis reference └─ example-ref-visuals.md 🧩 Visual reference
More Agent Skills specs Anthropic docs: https://agentskills.io/home
├─ ⭐ Required: YAML Frontmatter (must be at top) │ ├─ ⭐ name : unique skill name, follow naming convention │ └─ ⭐ description : include trigger keywords for matching │ ├─ ✅ Optional: Frontmatter extension fields │ ├─ ✅ license : license identifier │ ├─ ✅ compatibility : runtime constraints when needed │ ├─ ✅ metadata : key-value fields (author/version/source_url...) │ └─ 🧩 allowed-tools : tool whitelist (experimental) │ └─ ✅ Recommended: Markdown body (progressive disclosure) ├─ ✅ Overview / Purpose ├─ ✅ When to use ├─ ✅ Step-by-step ├─ ✅ Inputs / Outputs ├─ ✅ Examples ├─ 🧩 Files & References ├─ 🧩 Edge cases ├─ 🧩 Troubleshooting └─ 🧩 Safety notes
Skill files are scattered across GitHub and communities, difficult to search, and hard to evaluate. SkillWink organizes open-source skills into a searchable, filterable library you can directly download and use.
We provide keyword search, version updates, multi-metric ranking (downloads / likes / comments / updates), and open SKILL.md standards. You can also discuss usage and improvements on skill detail pages.
Quick Start:
Import/download skills (.zip/.skill), then place locally:
~/.claude/skills/ (Claude Code)
~/.codex/skills/ (Codex CLI)
One SKILL.md can be reused across tools.
Everything you need to know: what skills are, how they work, how to find/import them, and how to contribute.
A skill is a reusable capability package, usually including SKILL.md (purpose/IO/how-to) and optional scripts/templates/examples.
Think of it as a plugin playbook + resource bundle for AI assistants/toolchains.
Skills use progressive disclosure: load brief metadata first, load full docs only when needed, then execute by guidance.
This keeps agents lightweight while preserving enough context for complex tasks.
Use these three together:
Note: file size for all methods should be within 10MB.
Typical paths (may vary by local setup):
One SKILL.md can usually be reused across tools.
Yes. Most skills are standardized docs + assets, so they can be reused where format is supported.
Example: retrieval + writing + automation scripts as one workflow.
Some skills come from public GitHub repositories and some are uploaded by SkillWink creators. Always review code before installing and own your security decisions.
Most common reasons:
We try to avoid that. Use ranking + comments to surface better skills:
