Evals
Pricing

Simple pricing. No subscription.

Pay for what each eval uses, and see an estimate before it runs.

$0
Platform fee
No seats, no minimum
30%
on top of model token cost
The model under test and LLM grading
Compute
sandboxes for agentic benchmarks
Charged per task, for what it uses

Questions

Can I set a max spend on evals?

Yes. Set a max spend when you launch an eval. As soon as its charges reach that amount, the eval stops, including any tasks still running.

What happens if I run out of credits and my eval is running?

The eval pauses: no new tasks start, and tasks already running finish and are charged, so your balance can dip slightly below zero. We email your organisation’s owners and admins. Once you add enough credit to cover its next tasks, the eval carries on where it left off. If it stays paused for more than 24 hours it stops, and you can resume it from the unfinished tasks once you have enough credit. Turn on auto top-up in Billing to avoid pauses.

Can I use my own OpenRouter key?

Yes. Add it in Settings and OpenRouter bills you directly for the model under test, with our 30% margin charged as usual through credits.

Invite-only beta

Request an invite

We're letting teams in a few at a time. Leave your work email and we'll be in touch.

Already invited? Log in