MCP server

Run evals from your agent.

Connect the Ethos Evals MCP server to Claude Code, Claude, Cursor or Codex and ask in plain English. Your agent searches the catalogue, prices the run, launches it and reads back the results, with the same access you have in the web app.

Server URL

Invite-only beta: signing in needs an invited Ethos Evals account.

Connect your agent

Claude Code

Run this in your terminal.
In Claude Code, run /mcp, pick ethos-evals and choose Authenticate.
Sign in with your invited Ethos Evals account in the browser window that opens.

Claude (claude.ai and desktop)

Open Settings → Connectors and choose Add custom connector.
Name it Ethos Evals and paste the server URL.
Choose Connect and sign in with your invited Ethos Evals account.

Cursor

Add this to ~/.cursor/mcp.json (or your project’s .cursor/mcp.json).
Open Cursor Settings → MCP; ethos-evals asks you to sign in on first use.
Sign in with your invited Ethos Evals account.

Codex

Run this in your terminal.
Sign in with your invited Ethos Evals account when Codex asks.

The server signs you in with OAuth: the first time your agent connects, a browser window opens on the Ethos sign-in page. Sign in with the account your invite was sent to. There is no API key to create or paste.

Your agent then acts as you. It sees the evals, organisations and settings you can see in the web app, and every permission is the web app's own: billing tools, for example, work for organisation owners and admins only.

Not invited yet? . Already invited? Log in to the web app.

What to ask

Compare two models on a benchmark

“Run GPQA Diamond on these two models. Show me the cost first, then tell me which questions they disagree on.”

list_benchmarks
list_models
estimate_eval
run_benchmark
get_eval
get_eval_results
list_prompts
get_prompt

get_eval reports progress until the eval completes; list_prompts sorted by score difference finds where the models split.

Evaluate your own model

“Save our endpoint as a model, then score it against two frontier models on Terminal-Bench and show me where it fails.”

save_model
estimate_eval
run_benchmark
get_eval
get_eval_results
list_trials
get_trial_trace

save_model checks the endpoint once and stores its key; each run then names it as runs[].saved_model_id, next to hosted models.

Run an eval on your own data

“Preview this Hugging Face dataset, price an eval of three models on it, and run it once I approve the estimate.”

preview_huggingface_dataset
estimate_dataset_eval
run_eval_from_huggingface
get_eval
get_eval_results
export_eval

For a local file, run_eval_from_file takes the rows directly.

What it costs

Connecting and reading are free. Evals are charged to your organisation's credit, at the rates on the pricing page.

Price a run first: estimate_eval for a benchmark, estimate_dataset_eval for your own data or a re-run. Estimates create nothing and cost nothing.
Tools that start or retry evals (run_benchmark, run_eval_from_file, run_eval_from_huggingface, rerun_eval, retry_eval, retry_failed_trials) spend credit as trials finish. A good agent shows you the estimate and waits for your go-ahead before it calls one.
Owners and admins can check spend with get_credit_balance and get_usage, and buy credit through create_top_up_link: nothing is charged until a person completes the checkout.

Tools

68 tools, grouped by job. This list is read from the server itself.

Catalogue

Find benchmarks, models, judges and agent harnesses.

list_benchmarksSearch benchmarks: Search the benchmark catalogue.
get_benchmarkGet a benchmark: One benchmark's full catalogue entry, including its complete description.
get_benchmark_anatomyLook inside a benchmark: What a benchmark actually asks of a model, shown on two real sample tasks.
list_modelsSearch models: Models you can evaluate.
list_judge_modelsList judge models: Models that can grade answers (judge_model on the create tools), and the default.
list_harnessesList agent harnesses: Agent harnesses an agentic benchmark can run under (harness_id on run_benchmark).

Create and run

Price an eval, then start it on a benchmark, your own data or a copy of an earlier run.

estimate_evalEstimate an eval's cost: Price a benchmark eval before running it; nothing is created.
estimate_dataset_evalEstimate the cost of an eval on your data or a copy: Price an eval on your own data, a Hugging Face dataset or a copy of an eval before running it; nothing is created.
run_benchmarkRun a benchmark: Start an eval of one or more models on a benchmark.
run_eval_from_fileRun an eval on your own data: Start an eval on your own dataset: each row is one prompt, answered by every model and graded.
preview_huggingface_datasetPreview a Hugging Face dataset: Look at a Hugging Face dataset before running it: its configs, splits, columns (per config), row count and a few sample rows of the selected config.
run_eval_from_huggingfaceRun an eval on a Hugging Face dataset: Start an eval on a Hugging Face dataset; each row is one prompt.
rerun_evalRe-run an eval: Run a copy of an eval you own (or one in an organisation you administer), on the same prompts, as a new eval you own.
request_benchmarkRequest a new benchmark: Ask for a benchmark that is not in the library, and queue an eval of your models on it.

Manage

Follow, rename, cancel and retry your evals.

list_evalsList evals: Evals you can see, newest first: your own, ones shared with you and (as an organisation owner or admin) the rest of your organisations'.
get_evalGet an eval: Everything about one eval: status, progress counts, settings, per-model summaries, the comparative report, cost cap, why it stopped, and its credit charge.
rename_evalRename an eval: Rename an eval (its owner, or an owner/admin of its organisation).
cancel_evalCancel an eval: Stop an eval that hasn't finished: pending or running.
retry_evalRetry an eval: Run a completed, failed or cancelled eval again, keeping the trials that already finished.
retry_failed_trialsRetry failed trials mid-run: While an eval is still running, re-run the failed trials of models that have already finished.
generate_interim_reportGenerate an interim report: Write the comparative report now, from the results so far, while the eval is still running.

Results

Scores, per-prompt answers, agent trajectories, files and exports.

get_eval_resultsGet scores: Per-model scores: average and raw score, latency, output length, completed count, a per-prompt score list (omitted on very large evals) and winner_analysis.
compare_evalsCompare evals: Side-by-side scores for several evals (the UI's Compare page): each eval's name, status, benchmark and per-model average score, latency and completed count.
list_promptsSearch prompts: Search and sort an eval's prompts, e.g. to find where one model beats another.
list_prompt_tagsList prompt tags: Every tag and its values across an eval's prompts (for list_prompts tags).
get_promptGet a prompt's results: One prompt in full: its text, the expected answer or rubric, and every model's output, scores, judge explanation, tokens and latency.
list_trialsList agent trials: An agentic eval's trials and per-run statistics (success rate, pass@k, consensus, steps, time, tokens).
get_trial_traceGet a trial's trajectory: One agent trial: reward, final answer, timings, the grader's verdict, the files and screenshots it produced plus the grader's own output files (fetch any of them with get_trial_file), and a page of its steps.
get_trial_stepGet one trial step in full: One step of a trial in full: its message, tool calls and whole observation (text past 2 MB is cut).
get_trial_fileGet a trial's screenshot or file: A screenshot (returned as an image), a file the agent produced, or a grader output file such as verifier/reward-details.json (returned as text, or base64 for binary files up to 5 MB).
get_trial_grader_logsGet a trial's grader logs: The grader's stdout and stderr for one trial: the best clue when a trial has no trajectory or a surprising score.
get_trial_download_linkGet a trial download link: One trial's trajectory or a download link for its workspace tarball.
export_evalExport an eval: Export an eval's results.

Share an eval with colleagues or by a public link.

get_sharingSee who an eval is shared with: Who this eval is shared with: collaborators (people with an Ethos account; the full list for the eval's owner, the owners/admins of the eval's organisation and Ethos staff, only yourself otherwise) and its live public_links (null unless you own the eval).
get_share_linkGet a share link: The eval's share link for people with an Ethos account (owner only).
accept_share_linkAccept a share link: Accept a share link someone sent you, giving you read access to their eval.
remove_collaboratorRemove a collaborator: Remove a collaborator (owner only).
create_public_linkCreate a public link: Create a public link anyone can open without signing in: a read-only view of the eval's results and traces (owner only).
revoke_public_linkRevoke a public link: Turn off a public link immediately (owner only).
set_screenshot_sharingShare screenshots: Let collaborators and public-link viewers see trial screenshots (owner only).

Organisations

Teams, members and invites.

list_organizationsList your organisations: Your organisations and your role in each (owner, admin or member).
create_organizationCreate a team organisation: Create a team organisation, with you as its owner (when list_organizations says can_create_team).
rename_organizationRename an organisation: Rename a team organisation (owners and admins).
delete_organizationDelete an organisation: Delete a team organisation (owners only).
leave_organizationLeave an organisation: Leave a team organisation.
list_organization_membersList members and invites: A team organisation's members (with roles) and its pending and expired invites (owners and admins).
invite_to_organizationInvite someone: Email someone an invite to a team organisation (owners and admins; only an owner can invite an owner).
resend_inviteResend an invite: Send an expired invite again, as a new invite (owners and admins).
cancel_inviteCancel an invite: Cancel a pending invite so its link stops working (owners and admins).
accept_organization_inviteAccept an invite: Join an organisation from an invite you received.
change_member_roleChange a member's role: Change a member's role (owners and admins; only an owner grants or removes the owner role, and the last owner stays an owner).
remove_organization_memberRemove a member: Remove a member (owners and admins; only an owner removes an owner).

Billing

Credit, usage, invoices and top-ups (organisation owners and admins).

get_credit_balanceGet credit balance: An organisation's prepaid credit (owners and admins): balance, week-to-date spend, top-up packs, low-balance alert, auto top-up settings and saved card, and the newest ledger entries (10 unless you ask for up to 50).
get_usageGet usage: What each eval cost the organisation, newest first (owners and admins): estimate, amount charged so far, and state (estimated = not started, reserved = running and charging as trials finish, settled = finished, released or skipped = nothing charged).
export_usage_csvExport usage as CSV: The organisation's eval charges for a date range as CSV (owners and admins), one row per eval at customer price: every eval whose charge settled in the range or that has any billing entry in it (a running eval's charges, a later correction or credit-back on an earlier eval).
list_billing_documentsList invoices and receipts: The organisation's invoices and receipts (owners and admins), each with its total, status and links to view or download it.
update_billing_settingsChange billing settings: Change the low-balance email and automatic top-up (owners and admins).
create_top_up_linkGet a top-up link: A secure checkout link to buy credit (owners and admins).
add_payment_cardGet a link to save a card: A secure link to save a card for automatic top-up (owners and admins).

Your settings

Your own models, provider keys and email notifications.

list_saved_modelsList your saved models: Your saved models ("Your models": your own OpenAI-compatible endpoints and Hugging Face checkpoints), yours and your organisations'.
save_modelSave your own model: Save your own model so evals can call it.
test_saved_modelTest a saved model: Call a saved model once to check it still answers.
rename_saved_modelRename a saved model: Rename a saved model.
delete_saved_modelDelete a saved model: Delete a saved model and its stored key.
list_provider_keysList your provider keys: Your saved model-provider keys for bring-your-own-key runs: id, provider, label and the last four characters, never the key.
add_provider_keySave a provider key: Save a model-provider key for bring-your-own-key runs.
delete_provider_keyDelete a provider key: Delete a saved provider key and its stored secret.
set_email_notificationsEmail notification settings: Your eval email notifications.