Ned Cooper, ANU School of Cybernetics

Snehal Prabhudesai, University of Michigan

Abstract

Decision-support systems (DSS) based on Artificial Intelligence (AI) provide situationspecific forecasts and predictions to human decision-makers, with a goal to reduce overall errors in complex decision-making scenarios. While DSS may reduce the cognitive burden of decision-makers, they limit freedom of choice and autonomy over decision making [1]. Growing legal, regulatory and ethical concerns have led to a rise of “human-in-the-loop” strategies that provide overall authority to human decisionmakers. For example, algorithm-initiated hand-offs enable DSS to hand over control to domain experts in case of unforeseen or potentially risky situations. However, strategies initiated by DSS maintain machine autonomy by design, and require domain experts to provide additional labour by identifying and correcting DSS errors. Such handoffs not only increase the affective demands on human decision-makers in the short term, but also expose the overall socio-technical system to the ironies of automation over the long term [2]. In this talk, we will argue that algorithm-initiated strategies, as currently construed, lead to further alienation of decision-makers rather than giving them autonomy. Using medical decision-making as a case study, we encourage a human-centered approach to reorient control in high-stakes decision-making scenarios. We call for re-examining socio-technical workflows and eliciting human decisionmakers strategies when things go wrong, to guide the design of systems that balance autonomy and control in human-machine collaboration within healthcare. Using these strategies, we argue that interaction between expert decision makers and DSS needs to be reconstructed and redesigned so that decision-makers are brought in from the periphery to re-negotiate their relationship with AI.