How the search for better AI data reshaped an organization
An AI project may start with a narrow goal, but the data needed to make it work can take an organization somewhere much bigger. Two researchers followed an AI hiring initiative at a multinational company for over three years and showed how datafication spread to unexpected corners of the organization.
It started with a hiring problem. A large multinational company received around 10,000 applications each year for just 100 places in its management trainee programs. Its HR department wanted a better way to identify candidates who were likely to succeed.
Artificial intelligence seemed like a good place to start. But the company soon discovered that getting the AI to work meant answering a more difficult question: what does a successful employee actually look like?
To find out, the company needed data about its existing employees. The problem was that much of this information did not yet exist in a form the AI could use.
Employees were asked to complete more than 50 gamified assessments designed to measure personality traits and skills such as leadership, memory, and risk-taking. Managers began turning performance into measurable indicators, while interviewers started recording their judgments digitally.
What began as an effort to improve candidate screening was gradually changing the way the company measured and understood its people.
When better data means changing the work
Researchers Elmira van den Broek, Assistant Professor at the House of Innovation at the Stockholm School of Economics, and Natalia Levina, Professor at New York University Stern School of Business and Warwick Business School, followed the process for three years.
They found that producing data for AI was far from a purely technical exercise. It required people across the organization to change how they worked.
Managers, for example, had to reconsider how they evaluated employee performance. Existing assessments often relied on professional judgment, making them difficult to translate into the supposedly objective data the AI system needed.
Employees also questioned whether personality scores could fully capture their abilities, while others worried about how private assessments might affect future career decisions.
The search for better data was creating new ways of understanding employees, but it also raised questions about what those measurements might be leaving out.
“The important lesson for managers is that an AI project is rarely just about the technology. The search for better data can start changing parts of the organization that were never included in the original project,” says van den Broek.
The data started moving
Then came another shift. Once the company had collected large amounts of new data, it began exploring what else that information could be used for.
Personality data collected for hiring started appearing in career-planning discussions. Performance indicators developed partly for the AI initiative became relevant to employee evaluations more broadly. Digital interview records even made it possible to assess the managers conducting the interviews.
Suddenly, data created for one purpose was finding uses across the organization. The researchers describe this process as “epistemic enchantment”: a collective attraction to the promise that AI can produce better knowledge from increasingly large and sophisticated datasets.
That promise can become self-reinforcing. More data might improve predictions. Better predictions might support better decisions. And once new data exists, organizations may find more reasons to collect and use it.
But the expansion comes at a price. Collecting data takes time and resources, and turning complex human abilities into numerical indicators can make some aspects of work more visible while pushing others into the background.
A manager may know something important about an employee that cannot easily be expressed as a number. A discussion between interviewers may reveal context that disappears when individual judgments become scores.
In other words, the more work becomes measurable, the more organizations may begin organizing around what can be measured.
What should organizations consider before collecting more data?
The findings suggest that organizations need to look beyond whether more data will make an AI system perform better. They should also consider what people must do differently to produce that data, what information might be lost in the process, and how the data could eventually be used.
For managers, this means weighing the potential benefits of better predictions against the time, resources, privacy concerns, and changes to everyday work required to produce them.
Sometimes collecting more data may be worth the cost. In other cases, accepting imperfect data, finding another technical solution, or deciding not to use AI for a particular task may be the better option.
The study does not argue against AI. Instead, it highlights something organizations can easily overlook: the consequences of an AI project may begin long before the technology produces its first prediction.
About the study
The study, In Pursuit of Ideal Data: Epistemic Enchantment and the Unintended Consequences of Datafying for Artificial Intelligence, was published in Administrative Science Quarterly. The researchers conducted a three-year ethnographic study of an HR department at a large multinational company, drawing on observations, interviews, and company documents.
Read the full study: In Pursuit of Ideal Data: Epistemic Enchantment and the Unintended Consequences of Datafying for Artificial Intelligence.
Contact
Elmira van den Broek
Assistant Professor, House of Innovation
Stockholm School of Economics
elmira.van.den.broek@hhs.se
