# Ethos Evals

> Ethos Evals is a hosted platform for evaluating AI models: run 274 benchmarks (107 of them agentic, run end to end in a sandbox) on any model, including your own OpenAI-compatible endpoint, and launch evals from your coding agent through an MCP server. It is in invite-only beta.

Ethos runs the sandboxes, agent harnesses and graders, so there is no infrastructure to build. Results include per-model scores, every prompt and answer, full agent trajectories, and an AI-written report on where each model fails.

- Access: invite-only beta. Request an invite on the landing page; the web app and the MCP server both need an invited account.
- MCP server: https://pollen.prod.askethos.com/mcp/evals/ (streamable HTTP; OAuth sign-in with an invited Ethos Evals account, no API key). Claude Code: `claude mcp add --transport http ethos-evals https://pollen.prod.askethos.com/mcp/evals/`
- Your own model: save an OpenAI-compatible endpoint once with the `save_model` tool, then evaluate it as `runs[].saved_model_id` next to hosted models.
- Pricing: No subscription and no seats: no platform fee, plus 30% on top of model token cost (the model under test and LLM grading), plus sandbox compute for agentic benchmarks, charged per task. On your own OpenRouter key we still charge 30% of the model's token cost. An eval stops as soon as its charges reach its max spend, and pauses if your organisation runs out of credit. Prices exclude tax. You can estimate any run's cost before you start it.
- Costs: evals are charged to your organisation's credit. Price a run first with `estimate_eval` or `estimate_dataset_eval`; tools that start or retry evals spend credit.
- Frontier benchmarks include SWE-bench Verified, SWE-Bench Pro, Terminal-Bench 2.1, OSWorld, tau-bench, Humanity's Last Exam (text), TheAgentCompany, SWE-Lancer Diamond, DeepSWE.

## Docs

- [MCP server docs](https://evals.askethos.com/mcp): connect Claude Code, Claude, Cursor or Codex, sign in, example agent workflows, costs and the full tool list
- [Benchmark catalogue](https://evals.askethos.com/benchmarks): every benchmark you can run, with the price per task by model class
- [Pricing](https://evals.askethos.com/pricing): what an eval is charged for
- [Ethos Evals](https://evals.askethos.com/): what the product does, and the invite request

## MCP tools

- [list_benchmarks](https://evals.askethos.com/mcp#tool-list_benchmarks): Catalogue. Search the benchmark catalogue.
- [get_benchmark](https://evals.askethos.com/mcp#tool-get_benchmark): Catalogue. One benchmark's full catalogue entry, including its complete description.
- [get_benchmark_anatomy](https://evals.askethos.com/mcp#tool-get_benchmark_anatomy): Catalogue. What a benchmark actually asks of a model, shown on two real sample tasks.
- [list_models](https://evals.askethos.com/mcp#tool-list_models): Catalogue. Models you can evaluate.
- [list_judge_models](https://evals.askethos.com/mcp#tool-list_judge_models): Catalogue. Models that can grade answers (`judge_model` on the create tools), and the default.
- [list_harnesses](https://evals.askethos.com/mcp#tool-list_harnesses): Catalogue. Agent harnesses an agentic benchmark can run under (`harness_id` on run_benchmark).
- [estimate_eval](https://evals.askethos.com/mcp#tool-estimate_eval): Create and run. Price a benchmark eval before running it; nothing is created.
- [estimate_dataset_eval](https://evals.askethos.com/mcp#tool-estimate_dataset_eval): Create and run. Price an eval on your own data, a Hugging Face dataset or a copy of an eval before running it; nothing is created.
- [run_benchmark](https://evals.askethos.com/mcp#tool-run_benchmark): Create and run. Start an eval of one or more models on a benchmark.
- [run_eval_from_file](https://evals.askethos.com/mcp#tool-run_eval_from_file): Create and run. Start an eval on your own dataset: each row is one prompt, answered by every model and graded.
- [preview_huggingface_dataset](https://evals.askethos.com/mcp#tool-preview_huggingface_dataset): Create and run. Look at a Hugging Face dataset before running it: its configs, splits, columns (per config), row count and a few sample rows of the selected config.
- [run_eval_from_huggingface](https://evals.askethos.com/mcp#tool-run_eval_from_huggingface): Create and run. Start an eval on a Hugging Face dataset; each row is one prompt.
- [rerun_eval](https://evals.askethos.com/mcp#tool-rerun_eval): Create and run. Run a copy of an eval you own (or one in an organisation you administer), on the same prompts, as a new eval you own.
- [request_benchmark](https://evals.askethos.com/mcp#tool-request_benchmark): Create and run. Ask for a benchmark that is not in the library, and queue an eval of your models on it.
- [list_evals](https://evals.askethos.com/mcp#tool-list_evals): Manage. Evals you can see, newest first: your own, ones shared with you and (as an organisation owner or admin) the rest of your organisations'.
- [get_eval](https://evals.askethos.com/mcp#tool-get_eval): Manage. Everything about one eval: status, progress counts, settings, per-model summaries, the comparative report, cost cap, why it stopped, and its credit charge.
- [rename_eval](https://evals.askethos.com/mcp#tool-rename_eval): Manage. Rename an eval (its owner, or an owner/admin of its organisation).
- [cancel_eval](https://evals.askethos.com/mcp#tool-cancel_eval): Manage. Stop an eval that hasn't finished: pending or running.
- [retry_eval](https://evals.askethos.com/mcp#tool-retry_eval): Manage. Run a completed, failed or cancelled eval again, keeping the trials that already finished.
- [retry_failed_trials](https://evals.askethos.com/mcp#tool-retry_failed_trials): Manage. While an eval is still running, re-run the failed trials of models that have already finished.
- [generate_interim_report](https://evals.askethos.com/mcp#tool-generate_interim_report): Manage. Write the comparative report now, from the results so far, while the eval is still running.
- [get_eval_results](https://evals.askethos.com/mcp#tool-get_eval_results): Results. Per-model scores: average and raw score, latency, output length, completed count, a per-prompt score list (omitted on very large evals) and `winner_analysis`.
- [compare_evals](https://evals.askethos.com/mcp#tool-compare_evals): Results. Side-by-side scores for several evals (the UI's Compare page): each eval's name, status, benchmark and per-model average score, latency and completed count.
- [list_prompts](https://evals.askethos.com/mcp#tool-list_prompts): Results. Search and sort an eval's prompts, e.g. to find where one model beats another.
- [list_prompt_tags](https://evals.askethos.com/mcp#tool-list_prompt_tags): Results. Every tag and its values across an eval's prompts (for list_prompts `tags`).
- [get_prompt](https://evals.askethos.com/mcp#tool-get_prompt): Results. One prompt in full: its text, the expected answer or rubric, and every model's output, scores, judge explanation, tokens and latency.
- [list_trials](https://evals.askethos.com/mcp#tool-list_trials): Results. An agentic eval's trials and per-run statistics (success rate, pass@k, consensus, steps, time, tokens).
- [get_trial_trace](https://evals.askethos.com/mcp#tool-get_trial_trace): Results. One agent trial: reward, final answer, timings, the grader's verdict, the files and screenshots it produced plus the grader's own output files (fetch any of them with get_trial_file), and a page of its steps.
- [get_trial_step](https://evals.askethos.com/mcp#tool-get_trial_step): Results. One step of a trial in full: its message, tool calls and whole observation (text past 2 MB is cut).
- [get_trial_file](https://evals.askethos.com/mcp#tool-get_trial_file): Results. A screenshot (returned as an image), a file the agent produced, or a grader output file such as `verifier/reward-details.json` (returned as text, or base64 for binary files up to 5 MB).
- [get_trial_grader_logs](https://evals.askethos.com/mcp#tool-get_trial_grader_logs): Results. The grader's stdout and stderr for one trial: the best clue when a trial has no trajectory or a surprising score.
- [get_trial_download_link](https://evals.askethos.com/mcp#tool-get_trial_download_link): Results. One trial's trajectory or a download link for its workspace tarball.
- [export_eval](https://evals.askethos.com/mcp#tool-export_eval): Results. Export an eval's results.
- [get_sharing](https://evals.askethos.com/mcp#tool-get_sharing): Sharing. Who this eval is shared with: `collaborators` (people with an Ethos account; the full list for the eval's owner, the owners/admins of the eval's organisation and Ethos staff, only yourself otherwise) and its live `public_links` (null unless you own the eval).
- [get_share_link](https://evals.askethos.com/mcp#tool-get_share_link): Sharing. The eval's share link for people with an Ethos account (owner only).
- [accept_share_link](https://evals.askethos.com/mcp#tool-accept_share_link): Sharing. Accept a share link someone sent you, giving you read access to their eval.
- [remove_collaborator](https://evals.askethos.com/mcp#tool-remove_collaborator): Sharing. Remove a collaborator (owner only).
- [create_public_link](https://evals.askethos.com/mcp#tool-create_public_link): Sharing. Create a public link anyone can open without signing in: a read-only view of the eval's results and traces (owner only).
- [revoke_public_link](https://evals.askethos.com/mcp#tool-revoke_public_link): Sharing. Turn off a public link immediately (owner only).
- [set_screenshot_sharing](https://evals.askethos.com/mcp#tool-set_screenshot_sharing): Sharing. Let collaborators and public-link viewers see trial screenshots (owner only).
- [list_organizations](https://evals.askethos.com/mcp#tool-list_organizations): Organisations. Your organisations and your role in each (owner, admin or member).
- [create_organization](https://evals.askethos.com/mcp#tool-create_organization): Organisations. Create a team organisation, with you as its owner (when list_organizations says `can_create_team`).
- [rename_organization](https://evals.askethos.com/mcp#tool-rename_organization): Organisations. Rename a team organisation (owners and admins).
- [delete_organization](https://evals.askethos.com/mcp#tool-delete_organization): Organisations. Delete a team organisation (owners only).
- [leave_organization](https://evals.askethos.com/mcp#tool-leave_organization): Organisations. Leave a team organisation.
- [list_organization_members](https://evals.askethos.com/mcp#tool-list_organization_members): Organisations. A team organisation's `members` (with roles) and its pending and expired `invites` (owners and admins).
- [invite_to_organization](https://evals.askethos.com/mcp#tool-invite_to_organization): Organisations. Email someone an invite to a team organisation (owners and admins; only an owner can invite an owner).
- [resend_invite](https://evals.askethos.com/mcp#tool-resend_invite): Organisations. Send an expired invite again, as a new invite (owners and admins).
- [cancel_invite](https://evals.askethos.com/mcp#tool-cancel_invite): Organisations. Cancel a pending invite so its link stops working (owners and admins).
- [accept_organization_invite](https://evals.askethos.com/mcp#tool-accept_organization_invite): Organisations. Join an organisation from an invite you received.
- [change_member_role](https://evals.askethos.com/mcp#tool-change_member_role): Organisations. Change a member's role (owners and admins; only an owner grants or removes the owner role, and the last owner stays an owner).
- [remove_organization_member](https://evals.askethos.com/mcp#tool-remove_organization_member): Organisations. Remove a member (owners and admins; only an owner removes an owner).
- [get_credit_balance](https://evals.askethos.com/mcp#tool-get_credit_balance): Billing. An organisation's prepaid credit (owners and admins): balance, week-to-date spend, top-up packs, low-balance alert, auto top-up settings and saved card, and the newest ledger entries (10 unless you ask for up to 50).
- [get_usage](https://evals.askethos.com/mcp#tool-get_usage): Billing. What each eval cost the organisation, newest first (owners and admins): estimate, amount charged so far, and `state` (estimated = not started, reserved = running and charging as trials finish, settled = finished, released or skipped = nothing charged).
- [export_usage_csv](https://evals.askethos.com/mcp#tool-export_usage_csv): Billing. The organisation's eval charges for a date range as CSV (owners and admins), one row per eval at customer price: every eval whose charge settled in the range or that has any billing entry in it (a running eval's charges, a later correction or credit-back on an earlier eval).
- [list_billing_documents](https://evals.askethos.com/mcp#tool-list_billing_documents): Billing. The organisation's invoices and receipts (owners and admins), each with its total, status and links to view or download it.
- [update_billing_settings](https://evals.askethos.com/mcp#tool-update_billing_settings): Billing. Change the low-balance email and automatic top-up (owners and admins).
- [create_top_up_link](https://evals.askethos.com/mcp#tool-create_top_up_link): Billing. A secure checkout link to buy credit (owners and admins).
- [add_payment_card](https://evals.askethos.com/mcp#tool-add_payment_card): Billing. A secure link to save a card for automatic top-up (owners and admins).
- [list_saved_models](https://evals.askethos.com/mcp#tool-list_saved_models): Your settings. Your saved models ("Your models": your own OpenAI-compatible endpoints and Hugging Face checkpoints), yours and your organisations'.
- [save_model](https://evals.askethos.com/mcp#tool-save_model): Your settings. Save your own model so evals can call it.
- [test_saved_model](https://evals.askethos.com/mcp#tool-test_saved_model): Your settings. Call a saved model once to check it still answers.
- [rename_saved_model](https://evals.askethos.com/mcp#tool-rename_saved_model): Your settings. Rename a saved model.
- [delete_saved_model](https://evals.askethos.com/mcp#tool-delete_saved_model): Your settings. Delete a saved model and its stored key.
- [list_provider_keys](https://evals.askethos.com/mcp#tool-list_provider_keys): Your settings. Your saved model-provider keys for bring-your-own-key runs: id, provider, label and the last four characters, never the key.
- [add_provider_key](https://evals.askethos.com/mcp#tool-add_provider_key): Your settings. Save a model-provider key for bring-your-own-key runs.
- [delete_provider_key](https://evals.askethos.com/mcp#tool-delete_provider_key): Your settings. Delete a saved provider key and its stored secret.
- [set_email_notifications](https://evals.askethos.com/mcp#tool-set_email_notifications): Your settings. Your eval email notifications.

## Optional

- [Contact](mailto:support-evals@askethos.com?subject=Ethos%20Evals%20at%20scale): questions, and running evals at lab scale
- [Terms of Use](https://evals.askethos.com/terms)
- [Privacy Policy](https://evals.askethos.com/privacy)