OpenEMR benchmark: compiled replay vs a computer-use agent
On the public OpenEMR demo, a compiled replay finished 19 of 20 runs of an 18-step charting task and made no model calls. A computer-use agent finished 10 of 10 and spent about $0.55 a run on model fees. The agent had the higher success rate. On the one compiled miss, the program clicked Save, saw that the screen didn’t change as recorded, and stopped instead of reporting success. Compiled replay was also 1.8x faster at the median. ...