34 open-weight AI models tested across 50 health tasks

Ortet released its first health evaluation, testing 34 open-weight models on 50 tasks from 15 public benchmarks using its Orion GPU cluster. It also compared ten models accessed through APIs. Ortet says leading open models were competitive on basic medical knowledge, but both open and closed models struggled on the patient-prediction tasks it tested. More reasoning usually helped, it says, though the highest setting did not always yield the best observed score.