Compare models

Founder-only model benchmark: compares Claude, xAI Grok, Groq, Google Gemini (through the Lovable gateway / direct Google, GEMINI_COMPARE_MODEL, default google/gemini-3.1-pro-preview) and Perplexity (PERPLEXITY_API_KEY, PERPLEXITY_COMPARE_MODEL, default sonar-pro, may web-search). Writes nothing to campaign data (only spend records). mode 'fixtures' (default) runs eval-fixtures/dataset-v1.json through the real QA judge rubric and reports accuracy / precision / recall on hard fails, mean scores, agreement with the first provider, latency and cost. mode 'judge' (campaign_id) runs the real qaReviewMessages on up to `limit` qualified speakers' stored, scored trios with each provider as the judge and reports per-slot scores, trio pass rate, agreement with the stored verdicts and with claude, rewrite rate, plus example disagreements. mode 'draft' (campaign_id) has each provider write M1 from the identical real skill_v1 prompt, concurrently per speaker, then scores every candidate with the hard-fail gate and (unless judge=false) the Claude judge; returns side-by-side samples with each provider's M1, gate codes, word count, latency and cost next to the stored M1, never persists a candidate. A provider that errors (e.g. a missing key, named in the error) is counted under its errors and the others still return. mode 'human' (campaign_id optional) scores each provider as the QA judge against your own decisions: drafts you rejected with a rewrite request or substantively hand-edited are BAD, your hand-edited versions and messages sent exactly as drafted are GOOD; reports good_pass_rate, bad_catch_rate, balanced_accuracy, auc, pair_win_rate (your edit vs the draft), mean scores, p50/p95 latency and cost per message. mode 'founder' scores each provider as the founder judge: it predicts the founder's own send / don't-send call (and his reason) on his calibration marks in eval-fixtures/founder-labels.json, held-out marks when there are any, otherwise leave-one-out; reports agreement, send_agree, dont_agree, rule_match, cost per message and passes_gate (at least 18 of 20 held-out marks), plus every disagreement in full. mode 'founder_draft' (campaign_id) is the blind writing test: per speaker it puts the stored draft ('current'), the recipe engine ('recipe', with this campaign's m1_recipe or an inline recipe for this campaign) and each writer model (writers, several models per provider allowed) with one identical short founder-style prompt side by side, plus any extra_candidates you pass, and judges every message with the founder judge (judge_provider / judge_model); returns each speaker's inputs, every message with word count, checks and the judge's call, and per-writer totals; never persists. Pass writers=[], include_baselines=false and judge=false to get the speakers' inputs only, with no model calls. mode 'signals' (campaign_id) asks Perplexity, Gemini (Google grounding), Grok (web + X search), Claude (web search), Firecrawl and Apify the same question (this speaker's, or with target='event' this conference's, recent dated signals) and scores every finding the same way: URL loads, cited by the source, dated within 90 days, judged relevant by Claude; reports usable findings per speaker, hit rate, latency and cost per source. mode 'x_signals' (campaign_id) asks xAI live X search for each qualified speaker's handle and recent posts and says whether that signal is new. Each provider stops once its estimated spend passes max_cost_usd. One call does a small page and stops taking new items after 40s: page with offset until next_offset is null; elapsed_ms says how long it took. Pass dry_run=true for the call count first; keys_configured shows which of the five providers have keys.

Request

curl -X POST "https://sayintel.com/api/public/v1/model-comparisons" \
  -H "Authorization: Bearer $SAYINTEL_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{}'

Try it out

Body

Response

{
  "ok": true,
  "tool": "compare_models",
  "kind": "write",
  "data": { ... },
  "error": null
}

Notes

• Scope required: write. How scopes work.
• Accepts Idempotency-Key. A retry with the same key replays the first result instead of repeating the work.
• Accepts dry_run: true and always returns cost_preview before spending anything.
• Same capability over MCP as the tool compare_models. MCP setup.