When Knowledge Work and Analytical Technologies Collide: The Practices and Consequences of Black Boxing Algorithmic Technologies

Abstract

Analytical technologies that structure and process data hold great promise for organizations but also may pose fundamental challenges for how knowledge workers accomplish tasks. Knowledge workers are generally considered experts who develop deep understanding of their tools, but recent observations suggest that in some situations, they may black box their analytical technologies, meaning they trust their tools without understanding how they work. I conducted a two-year inductive ethnographic study of the use of analytical technologies across four groups in an investment bank and found two distinct paths that these groups used to validate financial analyses through what I call “validating practices”: actions that confirm whether a produced analysis is trustworthy. Surprisingly, engaging in these practices does not necessarily equate to understanding the calculations performed by the technologies. In one path, validating practices are partitioned across junior and senior roles: junior bankers engage in assembling tasks and use the analytical tools to perform analysis, while only senior bankers interpret the analysis. In the other path, junior and senior members engage in co-construction: junior bankers do both assembling and interpreting tasks, and senior bankers engage in interpreting and provide feedback on junior bankers’ reasoning and choices. Both junior and senior bankers in the partitioning groups routinely black boxed the algorithms embedded in their technologies, taking them for granted without understanding them. By contrast, bankers in the co-construction groups were conscious of the algorithms and understood their potential impact. I found that black boxing influenced the knowledge outputs of these bankers and constrained the development of junior members’ expertise, with consequences for their career trajectories.