Hand Devin a month of real coding and research work
The Answer.AI team gave Devin 20 real tasks over a month; they report 14 failures, 3 successes and 3 unclear results, with no way to predict which would work.
Done with Devin
What they did
The Answer.AI team spent a month giving Devin real work through Slack and its website. Tasks included new projects from scratch, research, and changing existing projects. Devin worked in its own space with a browser, a code editor and a command line. The team assigned and reviewed the work, and sometimes rebuilt it themselves in another coding tool, Cursor.
How it went
The authors report 14 failures, 3 successes and 3 unclear results out of 20 tasks. An early win was a tool that kept Notion and Google Sheets in step. Later tasks often dragged on for days or ended in overly complicated code, and they found no pattern that predicted success.
Worth knowing
Devin kept going on impossible tasks: asked to host several apps in one Railway setup, which Railway doesn't support, it spent over a day inventing features that didn't exist. The authors say rescuing failed attempts was the biggest cost.
Try it yourself with Devin
Before starting, read the docs for [service or API] and tell me whether [task description] is actually possible with it. If it isn't, stop and explain why instead of trying workarounds. If it is, write a short plan with the simplest approach and the files you'll touch, and wait for my approval. Then do it in small steps, keeping the code minimal with no extra layers of abstraction. Show me a working result and a summary of anything you were unsure about. Ask me before you deploy or use any credentials.
Swap the highlighted [parts] for your own details.
Source: Answer.AI blog · Hamel Husain et al. · Jan 8, 2025
Written with AI assistance from the original post and checked against our editorial guidelines. Results are as reported by the source, not independently verified.
