Workshop Details

Drawing on ethnographic examples from Arts and Agents, an ongoing research partnership between the ANU School of Cybernetics and the Australasian Dance Collective, I ask us to consider the work involved in designing and realising how we relate to and move with robotic systems. With successful collaboration, a world in which people and machines meaningful relate snaps into being – seemingly naturally, with grace, and easily understood by others present. For the Australasian Dance Collective’s Lucie in the Sky project, successful collaboration entails small quadcopters and dancers relating to and moving with one another intimately on stage, expressing individuality, relationships and emotional connection.  Yet designing and enacting such people/cobot relationships requires a plurality of intelligences and skills, with success emerging through both hard yards and malleable motions. Using this example, I will consider more broadly how negotiation, feedback, and incremental experimentation result in the creation of worlds in which cobots and people become with one another.

While scholarship to date on human-machine collaboration has focused on characterizing human-in-the-loop decision making [1] for high-stakes professions such as the judiciary [2], it has dedicated less attention to creative industries—the exception being scholars studying journalists working with algorithms [3, 4] and musicians working with metrics [5]. This paper examines how the practice of fashion design is transforming with the extensive integration of algorithms and big data in the design process. The fashion industry provides a rich field for addressing questions concerning the division of human and machine judgment in creative work, as a site in which the use of algorithms is guiding and making design decisions at an accelerating pace. Furthermore, decisions are poised to shape material output in an industry already under scrutiny for its egregious levels of carbon emissions and broadly unsustainable production practices. 

In this paper, we explore human-algorithm collaboration in ultra-fast fashion (UFF) firms—a new cohort of companies distinctive for the extent that they draw inspiration directly from social media platforms, such as TikTok, and for the speed at which they generate new product offerings. These firms increasingly rely on algorithmic systems to drive decisions in the design process and as a result the role of the “auteur” fashion designer is shifting. While traditional firms make use of in-house designers, UFF firms eschew design expertise altogether, using audience A/B testing to determine their designs. Finesse, an “AI driven” fashion company, delegates design to product development executives, who work alongside the firm’s proprietary algorithms. Shein similarly de-prioritizes the role of in-house design expertise, instead selling garments chosen by the company’s proprietary algorithms and analytics system. 

In this paper, we interview ten fashion designers at UFF firms. We find that while algorithmic systems and big data are increasingly prevalent in the design process, the industry still relies on human work: creative professionals drawing from a diverse array of inputs, reflecting their judgment, lived experience, and taste. Our findings respond to the field’s need for empirical work which engages with logics and practices of professions, adding a rich account to the oversimplified discourse around AI and automation. 

Humans gather information about their environment through their senses. The human brain interprets and organises these data from the world. This understanding helps humans predict and interact with the world around them, and also to be creative and imaginative about alternative outcomes. Closer human machine collaboration offers the promise of new ways of augmenting human performance and human thinking. I will discuss how these collaborations could evolve as we develop new forms of machines, and also discuss how ethical considerations must be an important part of this development.

While much attention is given to the collaboration between humans and AI systems, with regard to the creation of artistic expressions, still too little attention is given to the collaboration between humans and blockchain-based systems with regard to the creation of artistic works. Focusing on the case of the Plantoid, this talk will investigate the new opportunities for innovative artistic practices arising not only from the instrumentalisation of blockchain-based systems by humans (e.g. NFTs) but also from the instrumentalisation of humans by autonomous and self-sufficient agents operating on top of blockchain-based systems (e.g. DAOs).

Artworks:
James Auger – Real Prediction Machines
Fabien Zocco – Spider and I
Raphaëlle Kerbrat – Bug Antenna
Melanie Lane and Damith Herath – Judy

Much thought is given to the potential of human-robot collaboration, including in our research group, the Human-Robot Collaboration and Companionship group at Cornell University. In many cases, though, robot priorities can conflict with human priorities and needs. In this talk I will discuss instances of human-robot conflict as they arise in our research studies, from robots that directly compete with humans for monetary rewards to robots that try to help humans make decisions, but might not have the correct skill for this type of cognitive activity.

A growing body of research on human-agent teaming [1], [2] and human-robot collaboration [3], [4] shows that machines are increasingly acting as team members to humans. The agents and robots in these studies often employ different forms of Machine Learning to enable them to adapt to their environment (e.g. [5], [6]). Given that humans are naturally adaptive, it is certain that partners will mutually adapt their actions. Over longer periods of time, this adaptation transforms into co-evolution [7].  

In our research, we investigate how to support human-machine team partners in identifying co-adaptive behavior and in sharing successful Collaboration Patterns (reflective communication [8]). Such communication enables the human-machine team to achieve shared awareness of these Patterns, and helps them to successfully co-learn. 

We have previously run experiments in a virtual Urban-Search-and-Rescue environment [9]. Human participants collaborated with a Reinforcement Learning agent in saving an earthquake victim from underneath a pile of rocks. Both the human and the agent had to learn to use their unique capabilities to jointly complete the task (interdependence [10]). Teams developed a great diversity of Collaboration Patterns, that were often continued for several rounds of the task. However, due to spontaneous deviations by both the human and the machine partner, mistakes sometimes occurred.  

To enable communication and the development of shared awareness of successful Collaboration Patterns, the human-machine team needs a common language. We have developed an ontology that provides a knowledge structure for Collaboration Patterns: a conceptual framework that functions as a basis for communication. An accompanying Graphical User Interface (GUI) enables team partners to formalize and refine Collaboration Patterns through communication. The ontology and GUI were evaluated using video recordings of human-machine teams at work. Results showed that they supported humans in recognizing and defining Collaboration Patterns in the videos successfully. We are currently preparing an evaluation of the use of the ontology and GUI by a human-machine team during task execution.  

Collaborative learning is essential for human-machine teams to be successful. Our research contributes to the formalization, implementation and validation of a framework that supports identification, reflection and agreement of successful human-machine team behaviors. 

Explanations are often framed as an essential pathway towards improving fairness in human-AI decision-making. Empirical evidence on explanations’ ability to enhance distributive fairness is, however, inconclusive [1]. Prior work has found that humans’ perceptions towards an AI system are influenced by the features that a system is considering in its decision-making process [2,3,4]. For instance, if explanations were to highlight the importance of sensitive features (e.g., gender or race), it is likely that humans will perceive such a system as unfair. However, researchers have challenged the assumption that “unawareness” of an AI with regard to sensitive information will generally lead to fairer outcomes [5,6,7]. Moreover, the relationship between humans’ perceptions and their ability to override wrong AI recommendations and adhere to correct ones—i.e., to appropriately rely on the AI—is not well understood. 

In our work we examine the interplay of explanations, perceptions, and appropriate reliance on AI recommendations; and we argue that claims regarding explanations’ ability to improve distributive fairness should, first and foremost, be evaluated against their ability to foster appropriate reliance—i.e., enable humans to override wrong AI recommendations and adhere to correct ones. To empirically support our conceptual arguments, we conducted a user study for the task of occupation prediction from short bios. In our experiment, we assess differences in perceptions and reliance behavior when humans see and do not see explanations, and when these explanations indicate the use of sensitive features in predictions vs. when they indicate the use of task-relevant features. Ultimately, we test for differences in perceptions and reliance behavior across conditions and infer implications for the appropriate characterization of explanations’ role in human-AI decision-making. 

Our findings show that explanations influence humans’ fairness perceptions, which, in turn, affect reliance on AI recommendations. However, we observe that low procedural fairness perceptions lead to more overrides of AI recommendations, regardless of whether they are correct or wrong—a phenomenon sometimes referred to as “algorithm aversion”. This (i) raises doubts about the usefulness of common explanation techniques for enhancing distributive fairness, and, more generally, (ii) emphasizes that fairness perceptions must not be conflated with distributive fairness. 

Understanding the effects of providing greater control to intended users of algorithmic tools is central to advancing the responsible development and deployment of AI technologies in human-in-the-loop systems. While there is now an increasing emphasis on the use of participatory design methods for AI development, algorithms mostly continue to be designed by third-party researchers and organizations that may not fully understand users’ needs and values. This can lead to algorithm aversion, wherein human decision-makers are reluctant to use algorithms even when those algorithms outperform expert human judgment [1–3]. Studies have found that users are more willing to use algorithms as long as they have some control over the outcomes [4], and are more likely to perceive the algorithms as fair in those settings [6]. This ability to appeal or modify the outcome of a decision once it has been made is
termed “outcome control” [5]. Outcome control can be contrasted with “process control”, which entails control over the processes that lead to the algorithmic tool (e.g., data curation, the training procedure, etc.) The effect of process control on algorithm aversion is presently under-explored. We ask: Does process control mitigate algorithm aversion? Does providing both process control and outcome
control more greatly mitigate algorithm aversion than either form of control on its own? We conduct a replication study of outcome control [4], and test novel process control study conditions on Amazon Mechanical Turk (MTurk) and Prolific by allowing users to customize what input factors or model family (e.g., linear regression, trees, etc.) are used in the training process. Our results (mostly) confirm prior findings on the mitigating effects of outcome control. We find that process control in the form of choosing the training algorithm mitigates algorithm aversion, but changing inputs does not. Choosing the training algorithm also mitigates algorithm aversion to the same extent as does outcome control. Lastly, giving users both outcome and process control does not reduce algorithm aversion more than outcome or process control alone. Our study contributes to design considerations around mitigating algorithm aversion and reflects on the challenges of replication for crowdworker studies of human-AI interaction.

The very first robots, in Karel ÄŒapek’s play R.U.R., acted collectively to rebel against unfair working conditions.  The first real robots, developed during WWII to handle radioactive materials, moved their mechanical arms under the close supervision of human ‘tele-operators’ who used levers behind shielded walls.

Since then, roboticists have assumed that robots must be self-contained and carry their own power supply, memory, and computing circuitry. However, over the past decade robots have started to collaborate again, with each other and with humans using advances in networking and cloud computing. Collaborative robotics has become a fast-growing sector of the market. All major robot companies FANUC, KUKA/Midea, ABB, and Omron Adept have introduced collaborative robots, as have new robot companies Universal Robots, Fetch, Franka Emika and Kinova.

At Amazon, Google, and other leading companies, fleets of robots contact remote human teleoperators when they are at risk or unable to make progress.  Fleet Learning is a new approach to human-robot collaboration that treats robots as novice learners and humans as their expert supervisors, where each human can supervise multiple robots. Input from the remote pool of humans can also be used to improve the robot fleet control policy over time. A central question is how to effectively allocate limited human attention, when multiple robots interactively query and learn from multiple human supervisors.

We’ll summarize very new results with an IFL metric and algorithms evaluated on an open-source benchmark suite of environments built on NVIDIA Isaac Gym with a fleet of 100 robots and physical experiments with 4 ABB YuMi robot arms and 2 remote humans. Experiments suggest that the allocation of humans to robots can significantly affect the performance of the fleet, and that the new algorithm can achieves up to 8.8x higher return on human effort than baselines.

Robotic surgery has become the standard of care in a growing number of procedures. Surgical robots on the market are currently very sophisticated tools, operated under the nearly complete control of a surgeon, with whom they can interact in a number of ways. Although some autonomous capabilities are already within technical reach, they are in fact not yet deployed. The limiting factors are regulatory and legal uncertainty, and the lack of precise computational correlates for the notions of responsibility and liability.

Whereas the regulatory intent behind requirements of trustworthiness, correctness, and fairness are clear, systematically unpacking the meaning of basic terms is needed before one can move from principle to practice in cyberphysical human systems. This work presents a perspective on the concepts necessary to navigate dependencies between decisions jointly made by man and machine, as they cooperate via a range of interaction modalities known as the levels of autonomy. We examine the conceptual landscape (e.g., supervision, joint control, ..) under the lens of liability and in relationship to the requirements outlined in the European Union’s proposed regulation for AI systems, the 2021 AI Act [1].

The surgical domain has unique challenges, which can illuminate the terms of the general debate on human-machine collaboration. The notion of safety as the avoidance of harm cannot be directly applied, because surgery is in fact about causing controlled (local) harm in pursuit of a system-level benefit. Safety emerges as tightly tied to decisional correctness. Regulations that require human supervision and mandate that humans may intervene at any place in the decisional process improve the chances of catching machine errors—wherever humans still have the upper hand. Paradoxically, they also enormously widen the error surface where system-level errors could originate.

Grounded in examples from the surgical field, we explore ways in which humans can bias machines through their sensory limitations and incomplete knowledge, endeavoring to distinguish the errors we must strive to avoid through careful and ethical design from the errors we must learn to accept.

f a collaborator is unsure about a task that you are working on together, you expect them to communicate their uncertainty. We argue this practice should be followed when developing and deploying human-machine teams: if any team member is uncertain, efforts should be made to communicate and resolve or compensate for such uncertainty. Efforts have been made in the machine learning community to design frameworks which encourage a model to better incorporate uncertainty in its outputs [1], for instance, generating calibrated predictions [2, 3] or producing a set of plausible responses rather than a single estimate when unsure [4, 5]. However, while human probabilistic reasoning has been studied extensively within the cognitive science, psychology, and crowdsourcing communities [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16], human uncertainty representation specifically in machine learning has been limited, and if considered, is often captured through a single scalar measure [17, 18, 19, 20, 21, 22]. We argue for further elicitation and incorporation of human uncertainty, not just model uncertainty.

Our research program aims to address this gap. In Collins et al. [23], we took a step in this direction by
eliciting soft labels over multinomial label distributions representing annotators’ uncertainty in challenging image classification, finding that training models with richer labels can improve model generalization, robustness, and calibration. But this work has so far only focused on improving the machine learning system performance itself; human uncertainty has the potential to support the design of more effective and reliable collaborative systems. Already, recent works have begun to show that humans who are uncertain are more likely to side with a model, even if the model is wrong [24]. How then can we guard against propagating human biases in decision-making, while still designing systems which complement humans [25] accounting for their uncertainty to empower safer decision making under ambiguity? This is crucial in high-stakes settings, such as forming a treatment plan for a patient with comorbidities, or deciding from a set of policies to enact in response to geopolitical or climate instability. We believe the next generation of human-machine collaborative systems will benefit from a careful treatment of model and human uncertainty to adapt to an ever-uncertain world, and do so in ways that engender trust through appropriate transparency [26, 27].

In 2022, a number of companies have released Artificial Intelligence (AI) enabled tools that generate digital images via text prompts. For example: OpenAI announced DALL-E 2 and released an API with limited access; Google announced Imagen, though did not release an API; Midjourney announced a tool of the same name and released an API; and Stability.ai announced and open sourced Stable Diffusion. These models, also known as text-to-image diffusion models, are effectively automated media-generators (AMGs).

There is great interest in AMGs for creative and commercial pursuits, but they pose challenges to safe and responsible human-machine collaboration. In particular, the Stable Diffusion tool allows users to generate manipulated images of real people, governed only by a CreativeML OpenRAIL-M license, which requires users to self-regulate.

Self-regulation of AMGs contrasts with recent developments in individual protections against automated decision-making. For example, the General Data Protection Regulation affords individuals in the European Union and the European Economic Area (the EU) the right not to be the subject of a decision based solely on automated processing [1] and the Australian Human Rights Commission has recommended a similar right be established in Australia [2]. No explicit protections exist for individuals in relation to AMGs. However, the EU is currently considering legislation commonly known as the AI Act [3]. In its current form, the AI Act requires users to disclose when media has been generated or manipulated using an AI system.

In this talk, we will consider whether the protections proposed in the AI Act, are sufficient to ensure people’s safety in relation to AMGs, or whether more expansive protections are necessary, such as a right not to be the subject of an AMG. With a view to demonstrating the breadth of considerations relevant to the safety and responsibility of human-machine collaboration for AMGs, the talk will also consider:

• How can legislators keep up with advances in automated processing and their implications for human-machine collaboration?
• What is the role of education in strengthening the community’s capacity to engage with AMGs responsibly, such that those systems can operate safely at scale, in a changing world?

Artificial intelligence has been advancing rapidly over the past few decades, leading to the widespread adoption of AI-enabled systems and machines across many industries and fields. This is resulting in the emergence of new models of work. One such model is human-machine collaboration (HMC), in which humans and machines work together towards one or more common goals. HMC has been suggested as a more effective approach than having humans or machines working alone – especially for tasks that need to be carried out in uncertain conditions. An example of such an uncertain environment is healthcare. Additionally, HMC could be a potential solution for the challenges that healthcare systems are facing, such as physician shortages and high levels of workload.

Despite the benefits of HMC, there is a lack of understanding about how HMCs are evaluated and how their effectiveness should be measured. Performance evaluation can help organizations to identify when and how these systems are not performing safely, responsibly, and sustainably and make necessary adjustments. Additionally, evaluating the performance of human-machine collaborations can help to ensure that humans and machines are collaborating effectively and efficiently to achieve organizational goals. There is no one size fits all answer to this problem, as the best way to evaluate the success of human-machine teams will vary depending on the specific application or domain as well as the relationships and processes that HMC entails in different contexts. A systems view of HMC, however, can provide a useful framework for thinking about the evaluation of HMCs, as it helps to identify the various factors that are necessary for creating effective and efficient HMCs.

In this work, we propose to provide an overview of performance evaluation methods and approaches for human-machine collaborations in healthcare. This includes understanding how performance is defined, how it is measured, and how it is evaluated. We aim to discuss the strengths and limitations of the current methods and present our perspective for taking a systems view of the problem.

Decision-support systems (DSS) based on Artificial Intelligence (AI) provide situationspecific forecasts and predictions to human decision-makers, with a goal to reduce overall errors in complex decision-making scenarios. While DSS may reduce the cognitive burden of decision-makers, they limit freedom of choice and autonomy over decision making [1]. Growing legal, regulatory and ethical concerns have led to a rise of “human-in-the-loop” strategies that provide overall authority to human decisionmakers. For example, algorithm-initiated hand-offs enable DSS to hand over control to domain experts in case of unforeseen or potentially risky situations. However, strategies initiated by DSS maintain machine autonomy by design, and require domain experts to provide additional labour by identifying and correcting DSS errors. Such handoffs not only increase the affective demands on human decision-makers in the short term, but also expose the overall socio-technical system to the ironies of automation over the long term [2]. In this talk, we will argue that algorithm-initiated strategies, as currently construed, lead to further alienation of decision-makers rather than giving them autonomy. Using medical decision-making as a case study, we encourage a human-centered approach to reorient control in high-stakes decision-making scenarios. We call for re-examining socio-technical workflows and eliciting human decisionmakers strategies when things go wrong, to guide the design of systems that balance autonomy and control in human-machine collaboration within healthcare. Using these strategies, we argue that interaction between expert decision makers and DSS needs to be reconstructed and redesigned so that decision-makers are brought in from the periphery to re-negotiate their relationship with AI.

Background and purpose: There is a significant body of literature that has explored clinician’s perceptions of the use of artificial intelligence (AI) in health care (Laï et al. 2020; Sarwar et al.2019). In recent years, AI has been increasingly used in processes related to aspects of health care such as interpreting x-rays and mammograms and it’s adoption is often based on its capacity to reduce cost and improve health care outcomes (Shaheen, 2021). We explore the potential role of AI in mammography from the perspective of health care professionals currently involved in a population-based breast screening program in Melbourne, Australia.  

Method: We recruited health professionals working in BreastScreen Victoria and conducted 7 focus groups and interviews exploring their views of the potential role of AI in reading mammograms. Thematic analysis was employed to analyse data to determine the broad themes that emerged in the discussions. 

Results: 27 health care professionals participated in the focus groups and interviews. The analysis revealed overall support and enthusiasm about involving AI in general. Their concerns stemmed from limitations of AI programs that they are currently using and the risk it poses to their professional standards and safety and responsibility towards their patients. They raised ethical and medico-legal concerns with the involvement of AI such as accountability of error. Radiologists acknowledged the potential benefit of AI in freeing up time to engage in more patient centred communication processes.  

Conclusion: BreastScreen health professionals ranged from enthusiastic to hesitant on the use of AI for mammography. There was concern as to whether it could be implemented while also maintaining their duty of care to patients, professional responsibility, and high-quality work standards. This project demonstrates the importance of acknowledging the divide between opportunity and their lived experience of the use of AI in radiology and highlights the importance of continuous conversation and collaboration between radiology community about AI implementation in mammography. 

Artworks:
Patrick Tresset – RNP-S Sketching study
Sarah Fdili Alaoui and Léa Paymal: Physicalizing Loops Scores
Hugo Scurto: The Co-Explorer
Yosra Mojtahedi: L’érosarbénus and Sexus Fleurus

Ensuring that human-machine collaborations (HMC) do not dehumanise is an emerging focus of debate in the HMC literature, particularly when it comes to the application of HMC in medical contexts (Formosa et al. 2022). Taking a cybernetics lens, HMCs can be seen as purposive systems (Von Foerster, White, Peterson, & Russel 1968) shaped by human values. This interactional stance (Friedman & Hendry 2019) – where technology is shaped by humans and is concurrently shaping humans – is of particular relevance to the responsible design of machines that will be used by humans in collaborative ways. However, much of the human-machine collaboration literature focuses on how human-machine systems allocate resources in optimal ways (Hu & Chen 2017; Liu & Zhao 2021), structure human and autonomous teammates’ roles (Scholtz 2003), measure performance (Ma, Ijtsma, Feigh, & Pritchett 2022) and interact safely (Heinzmann & Zelinsky 1999; Ma & Wang 2022); relatively little attention is given to the importance of the earliest design phases where values are explicitly or implicitly chosen and begin to be embedded in design decisions, shaping HMCs. This talk posits that a focus on dignity as a value guiding these early stages (and then revisited throughout the design and implementation process) could provide a fruitful avenue of exploration for the future of responsible HMCs. It shares an interdisciplinary review on what dignity can look and feel like, pondering its meaning from a cybernetic perspective which considers HMCs as systems comprising technological, human and environmental factors. In doing so, this talk highlights  a plurality of meanings of the concept of dignity and its potential relevance to HMC, including human rights-based discourse on the meaning of dignity (Mattson & Clark 2011), concepts of environmental dignity (Manaster 1976) and non-Western perspectives of dignity as communal responsibility (Ikuenobe 2016). It also shares some preliminary learnings from working in intervention research with early-stage entrepreneurs, highlighting how a focus on dignity may influence the initial phases of the design of recommender algorithms. We hypothesise that learnings from the recommender algorithm design context may also apply to the field of HMC. In doing so, it hopes to provoke conversation around the use of dignity as a value for the future of collaborative machines. This talk will be of interest to those who are intrigued about how we might practically ensure that what it means to be human is preserved and enabled in human-machine collaborations. 

The increasing use of machine learning algorithms to support human decision-making has brought about the popular notion of “trustworthy AI”. Accuracy and explainability, among other things, are deemed to be two key elements in the trustworthiness of machine learning systems. They have become not only essential terms for formulating ethical AI guidelines but also important goals for computer science research efforts. The underlying assumption is that, if the output of AI systems is more “accurate” and “explainable,” then they become more trustworthy and trusted by users. Drawing on extensive participant observation and 36 semi-structured interviews with radiologists in China, this paper problematizes such assumptions and proposes an alternative framework centered on “human-machine alignment” to understand the issue of trustworthiness. I argue that radiologists develop their trust based on the degree of alignment between their own judgment and the algorithmic output, including what I call “direct alignment” and “adjusted alignment.” Regardless of the claimed performance indicated by the statistical parameters such as sensitivity and specificity, radiologists are still prompted to judge if the algorithmic decisions directly align with their own for each case. Such direct alignment practices are motivated by two factors. First, the probabilistic nature of the evaluation metrics of the algorithm’s performance cannot guarantee its correctness in the specific case in question, especially with the unavailability of a ready “ground truth” in real-world clinical practices. Second, under current legal and regulatory regimes, radiologists are held accountable for the medical imaging reports and are therefore motivated to doublecheck AI’s recommendations. Yet, even if the direct alignment is low, radiologists may still develop trust in and make use of the algorithmic output if they can observe certain patterns of, and thus explain away, the misaligned algorithmic output. This leads to an “adjusted alignment” based on the radiologist’s own interpretations. In conclusion, the paper suggests that notions of accuracy and explainability, rooted in algorithmic testing and designing, are misplaced in conceptualizing user’s trust in AI in real-world applications; instead, the trustworthiness of AI is a result of human-machine alignment and could not be reduced to some intrinsic features of the algorithms.

When machine-learning algorithms are deployed in high-stakes decisions, we want to ensure that their deployment leads to fair and equitable outcomes. This concern has motivated a fast-growing literature that focuses on diagnosing and addressing disparities in machine predictions. However, many machine predictions are deployed to assist in decisions where a human decision-maker retains the ultimate decision authority. In this article, we therefore consider how properties of machine predictions affect the resulting human decisions. We show in a formal model that the inclusion of a biased human decision-maker can revert common relationships between the structure of the algorithm and the qualities of resulting decisions. Specifically, we document that excluding information about protected groups from the prediction may fail to reduce, and may even increase, ultimate disparities. While our concrete results rely on specific assumptions about the data, algorithm, and decision-maker, they show more broadly that any study of critical properties of complex decision systems, such as the fairness of machineassisted human decisions, should go beyond focusing on the underlying algorithmic predictions in isolation.

Many of the world’s leading banks, including Australian banks, are at the forefront of using Advanced Analytics and artificial intelligence (AI) technology through their smartphone banking application. AI simultaneously brings value to the business, opportunities, and challenges that should be holistically viewed and adequately addressed to all stakeholders. In financial institutions specifically, AI has driven a dramatic shift in how to attract and retain active customers (World Economic Forum, 2018). Organisations like banks need to leverage analytics and AI, including machine learning and deep learning to compete effectively and drive customer value, deliver a more personalised customer experience and develop innovative new propositions (Deloitte, 2017). 

Smartphone has transformed the way we do banking from traditional physical transaction to digital online transaction. In this article, I will explore human-machine collaboration in the smartphone banking application with the focus on the sustainability aspect. I will use Cybernetics system thinking to provide a new way of reimagining the future of this collaboration with essential information on the implications of technology and machines, humans’ responsibilities, and sustainability for our environment. Cybernetics gives us a way of looking back and considering the technological developments in the banking industry. Particularly the one using Artificial Intelligence (AI) in terms of their patterns and systems, making sense of the relationships and feedback loops between those systems, as part of understanding and improving how those systems functioned holistically.  

Many banks are now beginning to invest in long-term sustainable financing commitments based on the data they have on their customer’s spending habits. They can target suitable customers to offer and invest in more sustainable products to create a better community support, increase customer loyalty and their financial well-being. The important role of customers not only as a user but more importantly as the essential collaborator for the future of sustainable banking.  Customers use their smartphone banking apps for almost everything in their everyday life. From the simple financial transactions such as paying for online grocery shopping to more advanced transactions such as applying for more sustainable “green” home loan or using AI in the recommendation systems for more personalised services.  

Bounded rationality is typically understood with respect to three limitations: time, information, and processing capacity (Simon, [1947] 1997; 1983; Cf. Kahneman, 2011). In response to these external constraints on human thinking, approaches to artificial intelligence have been developed in order to compensate for these limitations through the genesis of design thinking (Simon, 1981). Design thinking favors simplicity on the principle of near decomposability, which allows the breaking of complex systems down into part-whole relationships while still retaining the sense of their overall hierarchical organization (Simon, [1968] 1996; 1995). However, since the 1960s, design thinking has undergone a period of rationalization in which this sense of near decomposability has been reduced to total decomposability (Boden, 2016). This is a shift in which the design of heuristic problem-solving has become subjugated to the design of algorithms where heuristics are thought of as a sub-class of problem-solving methods. As a matter of means-ends relationships, this is a shift from satisficing a solution in a manner compatible with human problem-solving toward the optimization of a solution in a manner that is potentially unintelligible to human problem-solvers. This raises the question of the degree to which human-machine collaborations can be meaningful at the level of human participation rather than reducing human contributions to models of machine problem-solving. In that respect, this paper proposes a framework to evaluate the degree to which artificial intelligence achieves artificial intelligibility. This framework of artificial intelligibility operationalizes five criteria to describe the adaptive problem-solving capacity of a machine to meaningfully participate in the constitutive socially situated character of practical ritualistic activity, typically undertaken by human problem-solvers in relation to artifacts of design and discourse (Goffman, 1967; 1974; Brown, 2014; Raphael, 2017; Cf. Turner, 2018). Drawing on literature in the field of cognitive sociology, the paper details how the constitutive socially situated character of practical ritualistic activity describes criteria by which a machine can use, respond to, and invite the articulation of language, abstractions, and concepts in which its intelligence has to take into account the oscillation of the conditions of meaningfulness in an ongoing course of activity. Using these criteria, we argue and conclude that the study of human-computer interaction and its evaluation of the prospects for collaboration require (re)focusing on the socially situated character of discourse.

Despite high-profile successes in the field of Artificial intelligence (AI) [1-4], machine-driven solutions still suffer important limitations, particularly for complex tasks where creativity, common sense, intuition, or learning from limited data is required [5-8]. Both the promises and challenges of AI have motivated work exploring frameworks for human-machine collaboration [9-13]. The hope is that we can eventually develop hybrid systems that bring together human intuition and machine rationality to tackle today’s grand challenges effectively and efficiently.  

In this talk, we will overview ongoing research to develop and test hybrid prediction markets for crowd+AI collaboration. This builds on our o