Amir Asadi, ANU School of Cybernetics

Abstract

Artificial intelligence has been advancing rapidly over the past few decades, leading to the widespread adoption of AI-enabled systems and machines across many industries and fields. This is resulting in the emergence of new models of work. One such model is human-machine collaboration (HMC), in which humans and machines work together towards one or more common goals. HMC has been suggested as a more effective approach than having humans or machines working alone – especially for tasks that need to be carried out in uncertain conditions. An example of such an uncertain environment is healthcare. Additionally, HMC could be a potential solution for the challenges that healthcare systems are facing, such as physician shortages and high levels of workload.

Despite the benefits of HMC, there is a lack of understanding about how HMCs are evaluated and how their effectiveness should be measured. Performance evaluation can help organizations to identify when and how these systems are not performing safely, responsibly, and sustainably and make necessary adjustments. Additionally, evaluating the performance of human-machine collaborations can help to ensure that humans and machines are collaborating effectively and efficiently to achieve organizational goals. There is no one size fits all answer to this problem, as the best way to evaluate the success of human-machine teams will vary depending on the specific application or domain as well as the relationships and processes that HMC entails in different contexts. A systems view of HMC, however, can provide a useful framework for thinking about the evaluation of HMCs, as it helps to identify the various factors that are necessary for creating effective and efficient HMCs.

In this work, we propose to provide an overview of performance evaluation methods and approaches for human-machine collaborations in healthcare. This includes understanding how performance is defined, how it is measured, and how it is evaluated. We aim to discuss the strengths and limitations of the current methods and present our perspective for taking a systems view of the problem.