Benchmark

A crashed bot is a support ticket. A bot that writes to the wrong record (or writes the wrong thing, or writes nothing at all) and then reports success is a different kind of problem, and almost nobody publishes a number for it. Nothing pages anyone. The dashboard is green. The error surfaces later, owned by whoever owns the record it landed in: a note in the wrong patient’s chart, a payment posted to the wrong loan. And the defining property is that the tool’s own verification passed. It confirmed that something was saved. It never checked whose record it was, or whether the database agrees with the banner. ...

Computer-use agents are good now. Point one at a screenshot and a goal and it’ll find the button, on software it has never seen, with no selectors and no API. That still feels a little like magic to me. But watch one do the same task for the fiftieth time. It re-reasons everything, every run. Same login screen, same referral queue, same eleven clicks. Each time, a frontier model stares at the pixels and thinks the whole thing through again, and you pay for that thinking in seconds and tokens. Occasionally you also pay in a creative wrong answer. ...

The obvious objection to the 500th run was “sure, it’s your demo app.” Fair. So on 2026-07-08 we ran the same head-to-head against the official OpenEMR public demo: a dense, frame-heavy, LAMP-era EMR that anyone can point software at, fake patients only. Quick recap of what’s being compared. openadapt-flow compiles a recorded demonstration (browser, desktop, Citrix) into a deterministic workflow. Every step carries redundant visual evidence (a template crop, an OCR label, geometry landmarks) plus postconditions derived from what your demonstration actually changed on screen. A healthy run makes zero model calls. When the UI drifts, a resolution ladder heals the step and writes the fix back as a reviewable diff. When verification fails, it halts. It doesn’t improvise against a patient chart. ...