Pluto is an evaluation and qualification framework for models and agents. It runs repeatable capability checks, applies product requirements, compares candidates, and produces reports suitable for review or CI.

Install

go install github.com/looprig/pluto/cmd/[email protected]

Run Pluto

Define the target, packs, and profile in Pluto configuration, then run the command against that checked input. Provider credentials are resolved from the environment rather than copied into the manifest.

pluto run --manifest model.yaml --report report.json

Features

Capability packs

Versioned packs group checks for capabilities, operations, safety, structured output, and tool use. Teams can select the packs that match their product boundary.

Evaluation runs

Pluto executes deterministic or model-backed scenarios with bounded concurrency. Partial and skipped results remain visible instead of being silently treated as passes.

Qualification profiles

Profiles declare required capabilities and restrictions. Pluto evaluates a candidate against that policy and returns a disposition with the evidence for each decision.

Pricing and comparison

Pricing preflight identifies unknown rates before paid evaluation begins. Comparison aligns candidate and incumbent scorecards without dropping unmatched tables.

Reports

JSON reports use a versioned, bounded format that can be reviewed, stored, or passed to another system. Report errors are typed so automation can distinguish malformed, oversized, and unsupported data.

CI qualification

Run Pluto in CI to prevent a model or agent change from shipping without the required capability and safety evidence. Treat the report as a release artifact, not as session history or model context.

Repository

Read the framework, CLI, packs, and reports in the looprig/pluto repository.