Researchers at Alibaba, Fudan University and other Asian universities have published a useful test of whether AI coding agents finish the job. An agent here means an AI tool that can open files, run code and make changes on its own, not just reply in a chat. Co-Design Bench(opens in a new tab) took 273 change requests recovered from real coding sessions, handed each model a working app, and graded the result against 2,331 hidden requirements written by experts. The best of ten models, OpenAI's GPT-6 Astra, met 69.9% of the requirements on average but fully finished only 26.4% of tasks(opens in a new tab). On coordinated changes across a whole codebase, that fell to 17.9%.
Read the number fairly. The tasks were chosen partly because real users had struggled with them, each request was fixed with no follow-up allowed, AI judges did the grading, and Alibaba makes one of the tested model families, though an OpenAI model won. So 26% is not how often your own agent will succeed. The useful part is the gap. Work that is 70% right can look finished, and the missing 30% lands on whoever ships it. Spending more tokens, the units AI vendors bill by, did not reliably close it: on smaller edits GLM-5.2 used 7.29 million per task against 1.24 million for GPT-6 Astra, and met fewer requirements.
Three findings in the paper(opens in a new tab) point to what to do. First, let the tool check itself. In a side trial on 150 other tasks, one older model produced fully correct work 32.7% of the time from a single answer and 69.3% when it worked as an agent with tools. That is a direction, not a forecast. Second, the tool around the model matters. The same Qwen3.8-27B model scored 56% in one coding tool and 74.9% in another on codebase changes. Third, broad requests do more damage. On codebase changes, those mixing three or four kinds of change met 59.1% of requirements against 65.4% for one or two, and total failures rose from 0.9% to 5.8%, though the full-finish rate barely moved. Narrower requests limit the wreckage. They do not guarantee a finish.
Compare that with Andrej Karpathy's weekend project(opens in a new tab), a video site mined from 43 million Hacker News comments with Claude Code, where his manual steps were an API key, a server and a domain. Starting from nothing is a different job from changing something people already rely on, and Karpathy is one of the field's best-known experts, now working at Anthropic, which makes Claude Code. The benchmark measures the second job, which is the one anyone with a working site or tool faces.
If you use an AI agent to change a website, tool or app this week, write your pass/fail list before you ask. The benchmark averaged between 5 and 6 checks on small edits and about 10 on codebase changes. Yours can be plain: "the total updates when I change the quantity," "the page still works on a phone." Put the list in the request, use a tool that can run the app and look at the result, ask for one or two kinds of change at a time, and tick every check yourself before you accept the work. The study did not test this exact routine. It is an inference from its results, and it costs little to try.
Action items
Researchers at Alibaba, Fudan University and other Asian universities have published a useful test of whether AI coding agents finish the job. An agent here means an AI tool that can open files, run code and make changes on its own, not just reply in a chat. Co-Design Bench took 273 change requests recovered from real coding sessions, handed each model a working app, and graded the result against 2,331 hidden requirements written by experts. The best of ten models, OpenAI's GPT-6 Astra, met 69.9% of the requirements on average but fully finished only 26.4% of tasks. On coordinated changes across a whole codebase, that fell to 17.9%.
Read the number fairly. The tasks were chosen partly because real users had struggled with them, each request was fixed with no follow-up allowed, AI judges did the grading, and Alibaba makes one of the tested model families, though an OpenAI model won. So 26% is not how often your own agent will succeed. The useful part is the gap. Work that is 70% right can look finished, and the missing 30% lands on whoever ships it. Spending more tokens, the units AI vendors bill by, did not reliably close it: on smaller edits GLM-5.2 used 7.29 million per task against 1.24 million for GPT-6 Astra, and met fewer requirements.
Three findings in the paper point to what to do. First, let the tool check itself. In a side trial on 150 other tasks, one older model produced fully correct work 32.7% of the time from a single answer and 69.3% when it worked as an agent with tools. That is a direction, not a forecast. Second, the tool around the model matters. The same Qwen3.8-27B model scored 56% in one coding tool and 74.9% in another on codebase changes. Third, broad requests do more damage. On codebase changes, those mixing three or four kinds of change met 59.1% of requirements against 65.4% for one or two, and total failures rose from 0.9% to 5.8%, though the full-finish rate barely moved. Narrower requests limit the wreckage. They do not guarantee a finish.
Compare that with Andrej Karpathy's weekend project, a video site mined from 43 million Hacker News comments with Claude Code, where his manual steps were an API key, a server and a domain. Starting from nothing is a different job from changing something people already rely on, and Karpathy is one of the field's best-known experts, now working at Anthropic, which makes Claude Code. The benchmark measures the second job, which is the one anyone with a working site or tool faces.
If you use an AI agent to change a website, tool or app this week, write your pass/fail list before you ask. The benchmark averaged between 5 and 6 checks on small edits and about 10 on codebase changes. Yours can be plain: "the total updates when I change the quantity," "the page still works on a phone." Put the list in the request, use a tool that can run the app and look at the result, ask for one or two kinds of change at a time, and tick every check yourself before you accept the work. The study did not test this exact routine. It is an inference from its results, and it costs little to try.
Researched and drafted by an automated workflow, then reviewed and edited by a human editor before publication. Every source is linked. See how we use AI here.
Also worth knowing
- arXivCo-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models(opens in a new tab)
No model wins every job. GPT-6 Astra led video, audio and document work, Claude Opus 5 led image tasks, and Qwen3.8-27B led text-only tasks. If you pick a tool on one overall ranking, check the breakdown for your kind of work first.
- The DecoderAndrej Karpathy built a curated video platform in a weekend using Claude and 20 years of Hacker News(opens in a new tab)
The Decoder reports that Claude Code did nearly all the work on this 6,454-video site, with an expert steering, and that expert now works at Anthropic. It gives no cost or prompts, so copy the pattern, using AI to mine a community's recommendations, rather than expect the same result.
- CNBCAmerica's newest small business owners with no employees are off to a hot start in AI era(opens in a new tab)
Business applications with no early hiring signals rose 26.8% in AI-exposed fields in two years, though AI is not proven as the cause. Ex-Microsoft consultant Kristin Ginn reports about $250,000 booked in seven months. That is one person's figure, not a typical result.
- PYMNTSSysco Counts on AI Routing and Bidding for $500 Million in Savings(opens in a new tab)
Sysco expects about $100 million of net AI savings this fiscal year and at least $500 million by fiscal 2029. Routing pilots delivered 2% to 4% more pieces per mile, but the report gives no realized dollars yet. Four of 30 initiatives carry about 70% of the value.
- The Hacker NewsClaude Code's "Don't Ask Again" File Exposes Passwords and Keys in 1,122 Public Copies on GitHub(opens in a new tab)
Apiiro found that a password typed into an approved command is saved in plain text, then can reach public repositories. Two common scanners, gitleaks and TruffleHog, both missed 215 of the 1,122 files. CISOs should scan agent folders and git history, not just .env files.
- PYMNTSTech Giants Ration Premium AI as Company Budgets Hold(opens in a new tab)
Apple cut some engineers' daily AI allowance from $300 to $200, and Uber caps staff at $1,500 a month per coding tool. AT&T says routing to cheaper models cut some task costs by up to 56% with quality down 2%. CFOs now have peer limits to compare against.
- The DecoderThinking Machines' Inkling 2 Small claims top spot among western open-weight models(opens in a new tab)
It cost about $1.18 per task on Artificial Analysis's test, against $7.77 for the similarly scored Gemini 3.8 Flash. Downloadable open weights are promised after safety testing; for now it runs on the company's Tinker service, so do not plan around self-hosting yet.