Test a bookkeeping agent on made-up businesses
Ramp tested its bookkeeping agent on made-up businesses with messy books; the author reports cutting eight instruction sets to four gained 2.8 points and condensed memory added 5.7.
Done with Other
What they did
Ryan Stevens describes how Ramp tested a bookkeeping agent for Ramp Stack. The team built made-up businesses, called worlds, each with messy books and Excel files. One was a home goods brand selling straight to customers. Each test gave the agent a task, like checking fourth-quarter bills owed, and another AI graded the work against accountants' criteria. The team used these tests to remove and shorten the agent's instruction sets, and to try memory as each world moved forward a month.
How it went
The author reports that cutting eight instruction sets to four raised the average score by 2.8 points across six worlds. Shortening the spreadsheet instructions by 61% left quality essentially unchanged and saved about $30k a month. Condensed memory added 5.7 points, while automatically tuned prompts and telling the model to think harder did not help.
Worth knowing
Memory should keep how the business works but drop amounts tied to one period. One memory that carried over last month's specific recipe dropped a task from 75% to 25%.
Try it yourself
Help me build an end-to-end eval for my [type of] agent. First, list the main task categories in the job and break one task into the steps a human expert would take. Then design a synthetic test dataset with realistic messy data, not clean examples. For each eval case, define the task, the data the agent can access, and a rubric of [number] specific criteria that a [domain expert] would check. Then propose a plan to ablate the agent's skills one at a time against a no-skill baseline. Show me the design before building anything.
Swap the highlighted [parts] for your own details.
Source: DEV Community · Ryan Stevens · Oct 14, 2026
Written with AI assistance from the original post and checked against our editorial guidelines. Results are as reported by the source, not independently verified.
