Benchmark

Most automation decides that a save worked by looking for the green “Saved” banner. We tested how far that banner can be trusted. In our fault test, 72 of 90 runs left the record wrong. A check that trusted the banner passed 54 of 72 bad saves (75.0%). A check that read the saved record back by a separate route passed 9 of 72 bad saves (12.5%). A bad save that reports success does more harm than a crash, because nobody gets paged. The error turns up later with whoever owns the record, as a note in the wrong patient’s chart or a payment on the wrong loan. The automation’s check confirmed that something was saved, without asking whose record it was. ...

On the public OpenEMR demo, a compiled replay finished 19 of 20 runs of an 18-step charting task and made no model calls. A computer-use agent finished 10 of 10 and spent about $0.55 a run on model fees. The agent had the higher success rate. On the one compiled miss, the program clicked Save, saw that the screen didn’t change as recorded, and stopped instead of reporting success. Compiled replay was also 1.8x faster at the median. ...

A computer-use agent is an AI model that works from screenshots and chooses each click as it goes. It doesn’t need selectors, an application API, or a recording, so it can take on a task that nobody has automated yet. The cost shows up when the task repeats. The agent calls the model at each step of each run, even when the screens and the goal haven’t changed since yesterday. ...