OpenEvidence Accuracy Varies by Task, Studies Show No Single Score

Key Highlights

• A peer-reviewed Nature Medicine study reported 89.6% accuracy for OpenEvidence on 500 MedQA questions, alongside a 62.6-point HealthBench rubric score and a 3.24 out of 4 mean clinician rating.
• A separate non-peer-reviewed medRxiv pilot found 31% mean accuracy for quick search and 39.5% for Deep Consult across 100 complex reasoning scenarios, a difference that was not statistically significant.
• The studies used different tasks, samples and measures, so they do not support a single overall accuracy score or a universal ranking for medical AI.

Context and Core Dynamics

OpenEvidence is a medical AI search-and-answer service, and the published evidence does not collapse into one accuracy score for all uses. A peer-reviewed Nature Medicine study published June 12, 2026, evaluated the system across three distinct tasks, each with its own sample and measure. On 500 USMLE-style medical knowledge questions in MedQA, OpenEvidence answered 89.6% correctly.

The same study scored 500 HealthBench items against a clinician-alignment rubric, where OpenEvidence received 62.6 points on a 0-100 scale; this is a rubric score, not an accuracy percentage. In a third evaluation, 12 blinded US clinicians rated answers to 100 de-identified physician queries from NYU Langone Health on clinical correctness, completeness, safety and clarity using a 1-4 scale. OpenEvidence's aggregate mean was 3.24 across 99 answers that were not refusals. That clinician-rated test found no significant performance difference among OpenEvidence, UpToDate Expert AI and Google AI Overview. General-purpose models formed a higher-scoring tier, with mean ratings of 3.52-3.62. On the shared MedQA set, Gemini 3.1 Pro and GPT-5.2 significantly outperformed OpenEvidence in reported pairwise tests; the study did not report that finding for Claude Opus 4.6. These results describe the named tests, not every medical task.

A separate non-peer-reviewed medRxiv pilot tested 100 complex reasoning scenarios sampled from MedXpertQA. Overall mean accuracy was 31% for quick search and 39.5% for Deep Consult; the difference was not statistically significant (p=0.09). Two authors entered questions manually without prompt engineering. Quick search averaged 13 seconds and 5.2 references per answer, while Deep Consult averaged 259 seconds and 34.7 references. The pilot found no significant correlation between reference count and accuracy (r=0.068, p=0.177), so more references did not predict higher accuracy in this sample.

Caregiving Application and Social Impact

For the International AI Caregiving Association (IAICA), the practical value lies less in a headline score than in task-matched validation. A clinician-support tool used for medication questions, discharge planning or caregiver education should be assessed on the relevant query set, scoring rubric and safety criteria. The pilot's finding that additional references did not predict accuracy is especially relevant where caregivers may equate citation volume with reliability.

Ethics and Compliance Note

Ethical and compliance boundaries follow from the same evidence. Medical AI outputs should not be presented as a universal accuracy guarantee, and clinician ratings of correctness, completeness, safety and clarity do not substitute for local validation, privacy safeguards or accountability. Organisations should document the evaluated version, task and sample, and retain human review for high-stakes care decisions.