Watching an AI agent do data analysis is quite impressive. In a few minutes, it can handle relatively complex data, involving importing, merging, cleaning, and analysing it. The amount of code produced in a short amount of time is truly impressive. Now you can leave tools like Claude Code or OpenAI Codex to work for hours on your data. When they finish, you will see something that looks reasonable, but at the end of the day, they produce a black box that becomes increasingly complex.

In this post, I want to step back from the usual “how to” format. Instead, I will reflect on how tools like these are changing the way we do quantitative analysis.

Three ways of writing analysis code

If you have done quantitative research for a while and also tried to keep up with advances in coding tools, you have probably written code in three different ways.

The first is the traditional way where you write code yourself, line by line. It is slow and methodical. You need to make a plan and go step by step, from describing your data to analysis (see an example of a longitudinal workflow). While it is time-consuming, it also provides an opportunity to learn about the data and make important decisions, such as how to recode variables, whom to include or exclude in the analysis, and so on.

The second way to write code is the assisted way, using tools like GitHub Copilot. The tool watches what code you write and suggests the next line or block. This can be really helpful sometimes, though about half the time it doesn’t recommend exactly the code you want. You can reject the suggestions, but sometimes you might accept some code that is almost what you want just because it is faster. For example, it might recommend a more verbose way of coding than you normally use.

The third way to code is to use coding agents, such as Claude Code or Codex. Here you describe an outcome, such as “clean these five waves of data, merge them, and run a fixed effects model”. Then the AI agent for data analysis writes, runs, debugs and revises the code and creates a report. Now you have moved from analysis to commissioning.

What an AI agent for data analysis decides for you

Here is the problem with just asking the AI agent to “clean this dataset and run the analysis”. It sounds like one instruction, but in reality, it contains tens of decisions.

Anyone who works with data knows this intimately. Take something as routine as cleaning and merging multiple waves of data. Do you keep everyone who appears in at least one wave, or only complete cases? How do you treat someone who skipped wave 3 but returned in wave 4? What about a variable measured on a 5-point scale in early waves and a 7-point scale later? None of these has a “correct” answer.

Andrew Gelman called this the garden of forking paths. The same nominal analysis contains thousands of possible routes, yet researchers only ever see the one they walked.

When you write the code by hand, you walk the garden yourself. Slowly, sometimes badly, but you see (most of) the forks. In contrast, when an AI agent for data analysis does the work, the garden disappears from view and decisions are made for you. You get somewhere faster, but you’re not sure how you got there and if it’s really where you want to be.

Reasonable can still be wrong

Of course, you can ask the agent to explain its choices, and it will do so eloquently. But being told about a fork is not the same as having stood at one and made your own decision. A justification you are handed is something you check for plausibility. A decision you make yourself is weighed against everything you know about the data and the research question. These are different mental operations, and only one of them builds expertise.

The uncomfortable part is that the agent’s choices are, individually, almost always reasonable. If an AI agent for data analysis wrote bad code, we could potentially catch it. But as the code becomes more complex and the amount of AI-created code increases, this becomes more difficult to identify. Also, small decisions can compound, leading to an implausible result.

Taste: the skill an AI agent cannot give you

One essential skill when starting to automate statistical analyses is expertise, or “taste”. By taste, I mean the ability to recognise good-quality work as well as problems, based on countless hours of writing and reviewing code and results. This is an essential skill that enables you to identify key decisions and mistakes.

Taste is not intelligence, and it is not textbook knowledge. It is an intuitive sense of how analyses go wrong, built from past experiences and failures. The building blocks of taste are the hundreds of mistakes you have made over the long hours you have spent working with data.

What AI agents for data analysis mean for teaching and training

This brings me to the part I actually worry about: current students and new researchers.

Traditionally, part of doing quantitative analysis was spending time with data and code, finding errors and making hundreds of decisions. This is tedious work, but it is also where “taste” and expertise are developed.

Now the AI agent does the grunt work. As a result, new researchers can produce a polished analysis in an afternoon without confronting a single decision. They will be more productive than we were at their stage, and they will have learned less. Worse, their output gives no outward sign of the difference.

So the question is not whether students should use these tools. They will. The question is how to develop taste and expertise when the tools remove the experiences that build it.

So, while we figure out how best to use AI coding agents for doing quantitative analyses, we also need to consider how to build the skills needed for the next generation of researchers.

Three practical suggestions

Things are changing very fast, and I don’t have a complete answer, but I think there are two things we should consider given the current state of these models.

First, sequencing. Students need extended contact with raw, messy data before they automate anything. They should write the cleaning and preparation code themselves, hit the forks themselves, and sometimes choose wrong. Aviation is a useful comparison. Pilots learn the fundamentals through extensive manual practice before being expected to manage advanced automation.

Second, teach how to audit work produced by AI agents, as well as how to create guardrails. For example, we could give students a pipeline built by an AI agent for data analysis and have them interrogate it as a reviewer would. Where did observations disappear? What happened to the missing data? What are the important decisions, and how can we ensure we are in the loop?

We also need to keep experimenting with these new tools to develop best practices for working with AI in quantitative analyses. This will also help us understand what skills new researchers need.

Conclusions on AI agents for data analysis

An AI agent for data analysis can walk the garden of forking paths for you. But you still have to know the garden and what paths have been taken. That knowledge is built by hand, one small mistake at a time, ideally inside a reproducible workflow that makes your decisions visible. And while we figure out how best to integrate AI tools into our work, we also need to discover how best to develop “taste” and expertise in the next generation of researchers.