Trust completely defines how we interact, and it does not always respond to evidence. This occurs in relationships between humans, but also when we interact with any tool—whether or not it features artificial intelligence. However, in this case, highly significant circumstances arise that require special attention and new evaluation methodologies.
Differences Between Trust in AI and Non-Intelligent Systems
- Standard System FailuresSystems that do not make use of artificial intelligence can fail due to programming flaws, hardware issues, improper use, or lack of maintenance. This improper use is typically caused by usability problems, negligent actions, or a lack of training.
- The AI Paradigm ShiftConversely, systems using AI can make mistakes without this necessarily implying a system failure. This fact represents a major paradigm shift where humans must understand that the tool they are using can be wrong, and there will be no clear signs that it is broken—because it does not have to be. It could simply be mistaken, much like a human colleague or oneself could be.
- Mutual ObservationSimilarly, that tool might be observing and learning from the user. While observing, the artificial intelligence could prevent the user from committing a negligent act or, conversely, learn inappropriate actions as correct.
- Critical ImplicationsIn this context, how trust develops between a human and a tool takes a quantum leap, affecting multiple fronts such as legal, ethical, technical, and emotional areas. In use cases affecting critical sectors, like aerospace, this matter becomes particularly relevant.We are talking about scenarios where accepting or rejecting an AI’s recommendations can lead to high-impact consequences, often compounded by the human’s lack of time to deeply analyze the provided recommendations.
Have You Considered These Questions About Trust in AI?
- Is it right for an AI to act when it considers that a human is about to make a catastrophic error, actively preventing them from performing an action, without considering that the human insists on doing it? Would this answer apply to some cases or all cases?
- What happens if an AI provides a recommendation, the human follows it, and a catastrophic failure occurs? What if the catastrophic failure happens because they did not follow it?
- How can we evaluate whether an AI system is failing—which would be a manufacturing defect—or if it simply made a mistake?
- If an AI learns from a human, can the human be held responsible for its mistakes? Could it serve as a way to evaluate their performance? Won’t this make the human feel self-conscious?
These are elementary questions that everyone considers or will consider when working with AI. Some countries have already provided a legal response to these questions, but the debate is far from closed. Furthermore, many people are not even aware that legislation on this matter exists.
Why Is It Necessary to Evaluate Trust in Artificial Intelligence?
- User CategorizationAny study of usability, user experience, or interaction will consider as one of its primary factors who the users are going to be. A fundamental issue in categorizing users is their knowledge and experience regarding the product being used.
- The Fallacy of Mixing DataFor example, in a benchmarking usability test (measuring effectiveness and efficiency), it makes no sense to mix results from experienced users with those who are not. Honestly, it wouldn’t make sense in practically any type of usability test, but for the sake of clarity for those unfamiliar with user testing, I leave this case as the most evident one.
- Trust as a Behavioral FactorSimilarly, one cannot overlook the differences in usage that will exist between users who trust a tool and those who distrust it. Experience and trust are key factors that identify the user and must be taken into account when analyzing the interaction.
- Evolutionary DynamicsAnother similarity is that both experience and trust can evolve. In principle, experience should improve with usage time, but in the case of trust, it can evolve either positively or negatively.
- Impact on AdoptionResearch in the fields of human-computer interaction and human-robot interaction conducted to date clearly shows that user trust in artificial intelligence is one of the crucial factors influencing the adoption and use of any system employing AI. Therefore, there is no doubt that evaluating trust in AI is essential, and this evaluation must be carried out periodically, as it can evolve and drastically influence its use.
How to Evaluate Trust in Artificial Intelligence?
Many academic and corporate researchers have pointed out the need to develop qualitative and quantitative methods to evaluate the degree of trust in interaction studies that use artificial intelligence-based tools.
- Perceived Trust vs. Real ReliabilityIn this regard, we must begin by highlighting the need to distinguish between the trust perceived by a user and the “actual reliability” of a system using artificial intelligence, which is tied to statistical data showing effectiveness and efficiency. This necessity cannot be ignored because evidence demonstrates that what a user perceives can differ significantly from reality, which heavily influences how the user interacts with an AI.
- The POTDAI ToolAlthough this circumstance also occurs in systems that do not employ artificial intelligence, the context is profoundly different. In the article “POTDAI: A Tool to Evaluate the Perceived Operational Trust Degree in Artificial Intelligence Systems”, published in the journal IEEE Access, I present a method to evaluate trust in systems that employ AI, which is especially useful for cases where following or ignoring an AI’s indications can have serious consequences.
- Methodology and FrameworkThe research work focused on the development of trustworthy AI models for use in police interventions, within the framework of the “HUMANE-AI” project, funded by the European Commission. Thanks to the work carried out, we have developed a brief questionnaire that is quick and easy to apply, inspired by the original version of the Technology Acceptance Model (TAM) with six Likert-type items.
- Expanding Existing ModelsIn this way, we also address the need pointed out by several authors to expand one of the most widespread technology acceptance evaluation models to address issues related to user perception in systems with an AI component. This questionnaire can be used alone or in combination with the TAM to obtain additional information regarding its usefulness and ease of use.