Turn real-world failures into automatic tests
Rox's team had Devin turn records of their AI product's real failures into automatic tests; they report test coverage rising from 43% to 72%.
Done with Devin
What they did
Rox's AI team set up a repeating routine for Devin, a coding agent. Devin pulls records of times their AI product failed for real users. It recreates each failure in a safe practice copy of the product. Then it turns that failure into an automatic test the team can rerun.
How it went
The author says the share of problems their tests cover went from 43% to 72%, and is still growing.
Worth knowing
The post doesn't explain how the new tests are checked or what happens when a failure can't be recreated reliably. It is also partly a job advert.
Try it yourself with Devin
Pull the last [number] failed traces from [observability tool, e.g. Braintrust] for [project]. For each one, reproduce the failure in a preview environment of [repo URL]. Then write an eval case that captures the expected correct behaviour, using our existing eval format in [eval folder]. Group duplicates together. Show me the list of proposed evals with the trace each came from, and wait for my approval before opening a PR.
Swap the highlighted [parts] for your own details.
Source: X · @shriram_s · Oct 15, 2026
Written with AI assistance from the original post and checked against our editorial guidelines. Results are as reported by the source, not independently verified.
