A study published Tuesday in PNAS Nexus tested a question most AI rollouts skip: which part of the job should the person keep? Kartik Hosanagar of the Wharton School and Daehwan Ahn of the University of Georgia(opens in a new tab) had 285 students write a 1,000-word story on their own, then write another with ChatGPT under one of three setups. In "Human Confirmation," the AI came up with the idea, outlined, drafted and edited, and the student only approved or rejected each step. In "Human Creativity," the student picked the idea and wrote the outline, and the AI drafted and edited. In "Copilot," students used the AI at whatever stage they liked.
Total writing time did not differ significantly between the three setups, so keeping the idea and the outline cost nothing in speed. Quality did differ. Four raters scored every story on a 1 to 7 scale, and the approve-only stories improved less than the Copilot stories, mostly because they were less interesting and less coherent. Students who had written the strongest stories on their own did worse under approve-only. That group also reported less pride in their stories and a drop in how much flexibility they felt they had, and their stories were the most alike of any group. The press release(opens in a new tab) sums it up: approve-only stories were rated below the students' unaided work, while the other two setups beat it.
That matters because many AI deployments are built exactly like the approve-only setup: the model produces, a person signs off, and the sign-off is called keeping a human in the loop. It looks like the cheapest design, since the person does the least. In this test it saved no time over the alternatives and produced the weakest work. Most AI business cases count hours saved. This study suggests the human's place in the work decides what those hours are worth. Uniform output is a cost as well, for anyone whose writing, pitches or product copy has to stand out.
The limits are real. The writers were students, the task was fiction, the model was ChatGPT 3.5, several generations old, and everyone wrote without AI first, so before and after comparisons mix in learning and fatigue. It does not prove the same holds for code, analysis or customer email. But it is a randomized test of how to divide the work, which is rarer and more useful than another survey of how many people use AI.
If you write with AI, or manage people who do, try this on one recurring piece of work this week: a proposal, a product description, a monthly report. Write the main point and a rough outline yourself, in your own words, before you open the AI tool, then let it draft and edit. If you run a team, find any workflow where a person's only job is to approve AI output, and move that person to the start of the work instead of the end.
Action items
A study in PNAS Nexus by Kartik Hosanagar of Wharton and Daehwan Ahn of the University of Georgia had 285 students write stories with ChatGPT under three setups. Those who only approved the AI's ideas, outline and draft produced the weakest stories, with no time saved over setups where the student kept the idea and the outline. The cheapest-looking way to keep a human in the loop, signing off at the end, may be the one that costs you quality.
If you write with AI, or manage people who do, try this on one recurring piece of work this week: a proposal, a product description, a monthly report. Write the main point and a rough outline yourself, in your own words, before you open the AI tool, then let it draft and edit. If you run a team, find any workflow where a person's only job is to approve AI output, and move that person to the start of the work instead of the end.
Researched and drafted by an automated workflow, then reviewed and edited by a human editor before publication. Every source is linked. See how we use AI here.
Also worth knowing
- Unite.AIOpenAI Study Finds AI Freelance Projects Carry Larger Budgets(opens in a new tab)
In OpenAI and Upwork's data, AI-related fixed-price jobs posted budgets 52.4% higher, but hourly rates were only 1.5% higher, and AI-related hourly contracts ran 40.6% more hours in their first 90 days. If you freelance, the bigger budget buys more work, not a better rate.
- Thurrott.comFirst Android XR Smart Glasses From Warby Parker, Gentle Monster Are Launching in December(opens in a new tab)
Gemini glasses start at $395 from Warby Parker and run $449 to $489 from Gentle Monster, shipping December 10. They add a 12MP camera, Gemini voice help and live translation in 70+ languages. Bystanders' faces are blurred before Gemini sees an image.
- Hugging FaceLLMON: an LLM-native markup language, now in Granite(opens in a new tab)
IBM's preview Granite models, downloadable now, mark which text is an instruction and which is data, to block prompt injection, where commands hidden in a document hijack an AI. IBM's own test on one benchmark shows attacks almost eliminated. IBM says it is not for production yet.
- U.S. Securities and Exchange CommissionNVIDIA Announces CEO Transition: Jensen Huang Resigns; Colette Kress Appointed Interim CEO(opens in a new tab)
Huang resigned with no reason given. CFO Colette Kress is interim CEO, and new board chair Harvey Jones says customer commitments and the roadmap continue. Results on November 17 are the next read for anyone buying GPU capacity.
- SemaforSoftBank loses billions after over-ordering energy equipment for AI buildout(opens in a new tab)
SoftBank took a $3.7 billion hit after canceling gas power equipment it had ordered for AI data centers and could not use. Overbuying for the AI buildout is starting to show up as losses, not just as capex.