skill-quality-auditor

AI agent skills promise expert guidance — but how do you know they’re any good? skill-auditor scores SKILL.md files against a 9-dimension quality framework and produces concrete diagnostics to make them better. Think of it as a linter with opinions: it catches structural issues, gaps in guidance, missing evals, and anti-patterns before your users do.

Quick start

go build -o dist/skill-auditor .
./dist/skill-auditor evaluate <path-or-key>

Skill: skill-full
Grade: B+ (123/140)

Dimensions:
  Knowledge Delta              20/20
  Mindset + Procedures          7/15
  Anti-Pattern Quality          9/15
  Specification Compliance     16/15
  Progressive Disclosure       15/15
  Freedom Calibration          13/15
  Pattern Recognition          10/10
  Practical Usability          15/15
  Eval Validation              18/20

Warnings:
  [D2]  no precondition signals detected
  [D2]  no postcondition signals detected
  [D7]  no negative anchors in description — skill may over-trigger

Install

Method	Command
install.sh (Linux / macOS)	`curl -fsSL https://raw.githubusercontent.com/pantheon-org/skill-quality-auditor/main/scripts/install.sh
mise	`mise use github:pantheon-org/skill-quality-auditor`
Go	`go install github.com/pantheon-org/skill-quality-auditor@latest`

Commands

Command	Purpose
`evaluate`	Score a single skill (D1–D9, grade A+ to F)
`batch`	Score multiple skills, sorted by grade
`duplication`	Detect overlapping skills (Jaccard similarity)
`aggregate`	Generate consolidation plan for a skill family
`remediate`	Generate or validate a remediation plan
`trend`	Track score changes over time
`eval`	Run LLM-based eval scenarios against a skill
`analyze`	TF-IDF and pattern analysis for a single skill
`validate`	Check skill artifact conventions and context frontmatter schemas (at a given path)
`init`	Install the skill into agent harness directories
`update`	Self-update the binary from GitHub releases
`prune`	Remove old audit directories, keep N per skill
`version`	Print the CLI version and, for release builds, the release date

Architecture documentation

Architecture overview — high-level package layout and data flow
Evaluate flow — the core scoring pipeline
Batch flow — multi-skill evaluation
Duplication detection — Jaccard-based overlap analysis
Aggregation planning — skill family consolidation
Remediation flow — generating and validating plans
Trend tracking — score history and deltas
Eval runner — LLM-based scenario evaluation
Init, update, prune — lifecycle commands
Validate & analyze — artifact validation and static analysis
Scoring dimensions — D1–D9 reference
Development setup — prerequisites and workflow
Adding a scorer — how to extend the framework
Adding an agent — how to support a new harness
Skills and rules — local agent rules and skills

Key packages

Package	Path	Responsibility
`cmd`	`cmd/`	Cobra CLI commands, asset embedding
`scorer`	`scorer/`	D1–D9 scoring engine
`reporter`	`reporter/`	Formatting, persistence, plans
`duplication`	`duplication/`	Inventory, pairwise Jaccard detection
`analysis`	`analysis/`	TF-IDF keywords, rule-based pattern detection
`agents`	`agents/`	Agent registry for the `init` command
`internal/llmclient`	`internal/llmclient/`	Provider-agnostic LLM client
`internal/patternconfig`	`internal/patternconfig/`	Loads externalised D1/D6/analysis-quality pattern words from YAML
`internal/tokenize`	`internal/tokenize/`	Text normalization and tokenization

Last updated: