Benchmarks
Every number on this page is reproducible. Dataset is public, harness is MIT-licensed, latency is raw curl wall-clock against the live production endpoints. Methodology beats marketing — if you can't re-run it, it isn't a benchmark.
If you only read three sentences: AgentOracle's accuracy is measured on a public academic benchmark, not a number we made up. Anyone can re-run our results, the dataset is public, the harness is MIT-licensed. That's the difference between verification you can trust and verification you have to take on faith.
Accuracy — AVeriTeC 2024 Reproducible
Historical result: measured on 28 May 2026 using an earlier pipeline. That pipeline was replaced in September 2026. These figures do not measure the current service.
Public academic fact-checking benchmark from Schlichtkrull et al., EMNLP 2024. 500-claim dev set with 4-label space: Supported / Refuted / Not Enough Evidence / Conflicting Evidence. AgentOracle's /evaluate pipeline was run end-to-end against the live API. Calibration / held-out split is 250/250 by deterministic dataset index.
The benchmark is brutal. AgentOracle is roughly 2× the academic state of the art.
| System | Accuracy on dev |
|---|
| BERT-base classifier (paper) | ~25% |
| T5 (paper) | ~30–35% |
| Best paper-provided baseline | ~30% |
AgentOracle /evaluate | 57.6% |
Reported on the AVeriTeC 2024 dev set, with the verdict mapping selected by inspection on the calibration half. Retrieval recall is not reported and parametric-knowledge contamination is not controlled for. Treat as an internal baseline, not a leaderboard result.
AVeriTeC is multi-evidence fact-checking with a 4-label space. Published academic systems land near 30%. AgentOracle's pipeline scores roughly double the strongest paper-provided baseline on the same dev set. The score is high because the benchmark is hard, not because the bar is low.
Headline
| Split | N | Overall accuracy |
|---|
| Full dev set | 498 | 57.6% |
| Calibration half | 250 | 57.6% |
| Held-out half | 248 | 57.7% |
Reported on the AVeriTeC 2024 dev set, with the verdict mapping selected by inspection on the calibration half. Retrieval recall is not reported and parametric-knowledge contamination is not controlled for. Treat as an internal baseline, not a leaderboard result.
Held-out matches calibration → verdict mapping is not overfit. Selected via inspection on the calibration half; held-out is untouched during selection.
Per-category accuracy (held-out half)
| Claim category | Accuracy |
|---|
| Supported | 70.6% |
| Refuted | 61.6% |
| Not Enough Evidence | 27.3% |
| Conflicting Evidence / Cherrypicking | 13.6% |
Why the lower numbers on Conflicting / NEE are a safety property, not a model failure: when evidence genuinely conflicts, AgentOracle leans skeptical — most misclassified Conflicting claims received Refuted verdicts, not falsely-confident Supported ones. That is the right failure mode for regulated content. A verifier that fails toward caution is safer than a verifier that fails toward confident-wrong. We publish the absolute numbers as-is so a buyer can audit the failure shape themselves.
Reproduce
git clone https://github.com/TKCollective/agentoracle-eval-harness
cd agentoracle-eval-harness
curl -sL https://raw.githubusercontent.com/MichSchli/AVeriTeC/main/data/dev.json -o dev.json
python3 scripts/run_dev_eval.py
python3 scripts/score.py results/2026-05-28-dev/results.jsonl
Run completes in ~25 minutes at 3 concurrent workers against the live /evaluate endpoint. Submission registered at TKCollective/agentoracle-benchmark. MIT-licensed. Open submissions — see submission-format.md.
Footnote on inference path
The harness uses POST /evaluate, which is free and unmetered during beta. Pricing and payment details on this page are out of date. Current information: https://tanilo.io/pricing
Shipped — deterministic-first grounding. Claims that resolve by lookup rather than judgment route to
POST /v1/verify-facts, with no model in the trust chain. The 57.6% number above measures AgentOracle's
probabilistic pipeline on AVeriTeC's full claim mix (
spec) routes claims that resolve via structural lookup — field match, range check, comparison, set membership, citation existence — through a deterministic path that doesn't invoke an LLM at all and resolves at near-100%. The LLM tier becomes the small probabilistic slice. Every receipt will disclose
resolution_path: deterministic | probabilistic | hybrid so the buyer can see exactly how each verdict was reached. Not yet implemented — scoped, repo-visible, on the roadmap.
Why we publish this honestly: the value of a benchmark is that anyone can re-run it and check. If we picked numbers that flatter and hid the rest, the benchmark would be marketing. The point is the opposite — falsifiability is the moat. Run it yourself.
Latency — /v1 endpoints
Raw curl wall-clock from a single sandbox client against the live production hosts. Includes TCP + TLS + HTTP + handler + crypto + response. Sequential, single-connection-per-request. Measured 2026-06-25T22:14:00Z.
POST /v1/sign — single Ed25519 signature
Withdrawn. This endpoint signed bytes supplied by the caller under the production key with no authentication (the forgery-oracle defect, see the incident record). Since 28 August it refuses: it returns 503 and signs nothing. The figures below are historical.
Takes one canonical_bytes_b64u, returns one JWS signature entry. The primitive every composed envelope leg uses.
| Statistic | ms |
|---|
| min | 120.8 |
| p50 | 136.2 |
| p90 | 156.2 |
| p99 | 400.8 |
| max | 400.8 |
| mean | 146.1 |
n=50 sequential calls. Host: https://agentoracle.co (Vercel edge). Dominated by network round-trip and TLS, not crypto — the Ed25519 sign itself is sub-millisecond.
POST /v1/compose — full 2-signer composed envelope
Closed 28 Aug 2026. This agentoracle.co endpoint stopped issuing on 28 August 2026 (a8ce2a18d) and returns 503. It is a different route from the API gateway's /v1/compose, which was withdrawn on 24 September 2026.
Benchmark result withdrawn 2026-08-28. These timings measured a path whose AgentOracle leg returned a hard-coded verdict without evaluating the claim, so they are not comparable to any figure that includes verification work. The endpoint no longer issues while that path is in wire-up. Republished when the verdict path is connected and re-measured. The hard-coded verdict came from the AgentOracle stub, not AgentTrust. AgentTrust's leg signs its own bytes under its own key; its signature is not evidence that AgentOracle evaluated the claim.
Single HTTP call. Orchestrates AT /v1/compose → v_gate_skill, AO computes v_gate, builds canonical bytes once, AT /v1/sign for AT's signature, AO signs locally, assembles JWS general serialization. Returns full 2-signer envelope verifiable against both published JWKS.
| Statistic | ms |
|---|
| min | 380.2 |
| p50 | 401.8 |
| p90 | 417.6 |
| p99 | 745.0 |
| mean | 417.3 |
n=20 sequential calls. Includes one round-trip to AT /v1/compose + one to AT /v1/sign over the public internet from the AO origin region to the AT origin region. Co-location would cut this further; published as-is.
POST /v1/sign/batch — N canonical bytes, N signatures
Withdrawn. This endpoint signed bytes supplied by the caller under the production key with no authentication (the forgery-oracle defect, see the incident record). Since 28 August it refuses: it returns 503 and signs nothing. The figures below are historical.
Bulk signing primitive for high-frequency agent loops. Max 100 items per request. Returns one signature entry per input; all entries share the same kid (issuer is constant). It was offered for anchoring many envelopes in one network round-trip; it no longer signs.
Latency is dominated by HTTP round-trip, not crypto. Batched cost-per-item drops linearly with N for any N ≥ 2.
What this is and isn't: these are wall-clock numbers from one sandbox client against the live production endpoints. They include network and TLS overhead. They are not the verification accuracy (see AVeriTeC above), they are not isolated crypto benchmarks (Ed25519 sign is <1ms on any modern CPU), and they are not the latency you would see from a co-located client. These are archived measurements from 2026-06-25; the signing endpoints they timed no longer sign, so the commands are historical reference only, not something you can re-run.
Conformance — composed envelope 11/11 vectors
Parallel Node + Python verifiers, both byte-identical on Phase 1 (4 accept + 3 reject) and Phase 2 (3 accept + 1 reject). Independently verified end-to-end on 2026-06-24 by @giskard09 (IETF action-ref maintainer at argentum-core).
| Vector | What it covers | Result |
|---|
| comp-001…004 | 2-signer accept (both-act, AT-halt, AO-halt, trail-resolved) | PASS |
| comp-r01…r03 | 2-signer reject (tampered sig, null mycelium_trail_id, AND_PRESENT mismatch) | PASS |
| comp-005…007 | 3-signer accept (with Presidio screen_ref, comp-006 = PII_BLOCKED halts a payment both gates approved) | PASS |
| comp-r-screen | screen_ref recompute mismatch | PASS |
Full suite + verifiers at TKCollective/agentoracle-receipt-spec. Spec anchor: draft-krausz-verification-state-01 (IETF Internet-Draft).