🔍 Read the full analysis: ChatGPT Improved Consultants’ Data Science Work, But Not Their Retention on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
In a randomized trial at Boston Consulting Group, consultants given brief ChatGPT training scored higher on coding, statistics and prediction tasks than a group using web resources. After ChatGPT was removed, researchers found no evidence that the treated group performed better on technical questions, suggesting the gains measured in this study depended on access to the tool.
A randomized trial at Boston Consulting Group found that consultants using ChatGPT scored higher on data science tasks than colleagues using web resources, but the advantage did not appear on a technical test taken without the chatbot. Published October 26 in Nature Human Behaviour, the study suggests that, in this experiment, AI expanded what workers could do while they had access to it without showing that they could then perform the same work independently.
The study allocated 986 BCG consultants to a ChatGPT group or a control group that could use Google and sites including Stack Overflow and Khan Academy. Researchers report that 487 participants submitted work on both assigned tasks, forming the main analysis sample. The ChatGPT group received 15 to 20 minutes of instruction on prompting for technical problems and used GPT-4.
Participants completed two of three 90-minute assignments: Python data cleaning, statistical analysis, or a prediction exercise about international football matches. The tasks were designed with BCG data scientists and OpenAI’s Economic Impacts Research team so ChatGPT could help but could not independently provide correct answers to all questions. The researchers compared results with the average performance of 44 BCG data scientists who completed the tasks without AI.
Compared with the control group, consultants using ChatGPT scored 49 percentage points higher in coding, 20 points higher in statistics and 17 points higher in prediction. The ChatGPT group still averaged below the data scientists on all three tasks. Afterward, both groups answered technical questions without ChatGPT; the authors report no evidence that the ChatGPT group did better on that follow-up measure.
The trial in motion · BCG consultants · 2024
With ChatGPT: close to the experts. Without it: no measurable gain.
A randomized trial with a control group that used Google and Stack Overflow instead of ChatGPT.
- 1 · Training15–20 minutes on prompting ChatGPT
- 2 · TasksTwo 90-minute data science tasks, with ChatGPT
- 3 · ScoresGap to 44 data scientists closes, not fully
- 4 · Tool removedNo ChatGPT from here on
- 5 · TestFive technical questions
Coding task score
0 = data scientists’ average
ChatGPT on+49 pp on codingStill below the data scientists on average
Knowledge test without ChatGPT
ChatGPT group vs. control, five questions
ChatGPT off−7.7 to +4.8 ppNone significant · joint test p = 0.531
Source: Wiles, Krayer et al., “Generative AI and the temporary upskilling of knowledge workers”, Nature Human Behaviour (2026); authors’ manuscript, Sections 2.1 and 2.3, Tables B7 and B9. Bars scaled to the five published differences.
Graphic: Powered by Thorsten Meyer AI
Task Gains Did Not Equal Retention
The results address a distinction that matters to employers: completing a task with AI is not the same as learning to do it without AI. In the trial, ChatGPT helped consultants produce stronger work on assignments beyond many participants’ existing technical skills. The follow-up test, however, did not show a corresponding advantage once the tool was unavailable.
For companies, that difference affects how they interpret productivity and training. A worker may deliver a better immediate result with an AI assistant without acquiring skills that transfer to a later task. The study’s findings support treating AI-assisted performance and independent ability as separate outcomes when evaluating training, staffing or quality controls. They do not establish that ChatGPT can never teach skills; they show that this particular intervention did not produce a detectable improvement on the study’s tool-free test.
The short-term task results also came with a reported risk in judgment: the ChatGPT group became more confident that the system could solve problems it could not. The authors describe this part of the analysis as exploratory, so it should not be read as a definitive estimate of how often users overtrust AI. Still, it highlights why output review and clear limits on tool reliance can matter alongside productivity.
Products Worth Considering
Logitech MX Mechanical Wireless Illuminated Keyboard, Clicky Switches
Bose QuietComfort Wireless Noise-Cancelling Headphones, White Smoke
DJI Mini 5 Pro Fly More Combo with DJI RC 2
As an affiliate, we earn on qualifying purchases.
How the BCG Trial Was Designed
The experiment was preregistered on March 13, 2024, before data collection, and ran in late March and early April that year. Participants had at least a bachelor’s degree; the authors say most reported no coding experience. The control group received instruction on web research, making the comparison one between two forms of assistance rather than between AI and no resources.
The researchers used a benchmark of data scientists to show how close the consultants came to specialist performance. On coding, the ChatGPT group was an average of 14 percentage points below that benchmark, compared with 63 points for the control group. In statistics, the ChatGPT group was 12 points below the benchmark and the control group 32 points below. These comparisons describe task scores in this trial, not proof that consultants became equivalent to data scientists.
The paper also reports that the ChatGPT group completed coding and prediction tasks faster on average, by about nine and 14 minutes respectively; time on statistics did not differ significantly. Among participants the authors classified as having coding competence, the ChatGPT group was five points below the data scientists on coding and statistics. These subgroup results do not change the main finding that the overall group remained below the expert benchmark.
“The authors find no evidence that the ChatGPT group did better on technical questions answered without the tool.”
— The study authors, as summarized in the paper
Limits of the Retention Test
The follow-up consisted of five technical questions about probability, machine learning and coding, not a repeat of the original work tasks. The authors’ finding of no evidence of a group difference on that test does not prove that participants learned nothing, nor does it establish how performance might change after longer training or repeated AI-assisted work.
The study took place at one consulting firm, used specific assignments and examined GPT-4 after a brief training session. It is not clear from the reported results whether the same pattern would hold in other jobs, with different AI systems, more extensive instruction or longer periods of use. The results also do not establish whether the gains persist while the tool remains available over time, or how the reported confidence effect would play out in routine workplace decisions.
Testing Skills Beyond One Session
The paper reports results from the completed BCG experiment; the supplied study summary identifies no next trial or follow-up milestone. Further research would be needed to determine whether sustained training, repeated practice or different ways of integrating AI can produce skills that remain when a chatbot is unavailable.
For now, the evidence supports a narrow conclusion: ChatGPT improved performance on the study’s data science assignments, while the tool-free test showed no measurable advantage for the group that had used it. Employers interpreting such results can distinguish work completed with assistance from demonstrated independent skill, while waiting for evidence from broader settings and longer-term tests.
Key Questions
What did the BCG study find?
Consultants using ChatGPT scored higher than a control group on coding, statistics and prediction assignments. The study reports no evidence of better performance by the ChatGPT group on a later technical test taken without the chatbot.
How much higher were the task scores?
Compared with the control group, the ChatGPT group scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction. Those are differences between study groups, not evidence that the consultants outperformed the data scientists.
Did the consultants match the data scientists?
No. The ChatGPT group averaged below the 44 BCG data scientists on all three tasks, although the gap narrowed substantially on coding and statistics. The authors report especially close scores for a subgroup with coding experience, but that does not mean the full group matched experts.
Does the study prove ChatGPT cannot teach skills?
No. It tested a particular short training intervention, GPT-4, specific assignments and a five-question follow-up without AI. The result is that this experiment found no evidence of a tool-free advantage; it does not settle what longer or different training might achieve.
Source: ThorstenMeyerAI.com
