Download PDF Spring Bridge on AI: Promises and Risks April 15, 2025 Volume 55 Issue 1 This issue of The Bridge features fresh perspectives on artificial intelligence’s promises and risks from thought leaders across industry and academia. Alignment of AI Systems' Risk Attitudes, and Four Real-Life Examples Friday, April 11, 2025 Author: Elisabeth Paté-Cornell AI systems’ risk preferences should be consistent with those of humans, particularly in critical domains like healthcare and national security. Artificial intelligence (AI) systems perform two kinds of functions: gathering and processing information, and, in some cases, suggesting or directly implementing decisions based on that information. With respect to preferences, information should be neutral; decisions are not. The focus in this article is on decisions. The question of AI alignment is how to design and use AI to make or support risk management decisions under uncertainty so that, in the end, the system’s recommendations fit the preferences of the decision-maker if they did not in the first place (Paté-Cornell 2024). AI Preferences and Decision-Making Preferences have several dimensions, including trade-offs among decision outcome attributes as well as a risk attitude. Imagine, for instance, a medical situation in which an AI system recommends a test that the patient may not want, or a military attack in which drones are guided by an AI system without human intervention. The AI system may have different approaches to its decisions under uncertainty, but they reflect a risk attitude one way or the other. That risk attitude has to match that of the decision-maker. It is assumed here that the AI algorithm provides rational decisions based on a decision analysis framework and the von Neumann axioms, often supported by a utility function embedded by the decision analysts within the algorithm (Russel and Norvig 2021). Furthermore, it is assumed that the outcomes x of all scenarios are described by a single unit, like a monetary currency, and a single utility function. In the embedded utility function U(x), the risk attitude at any given level of potential loss x is the negative of the ratio of the second to the first derivative—i.e., -U”(x)/U’(x). In that framework, the optimal option for the decision-maker is that which maximizes his or her expected utility (Abbas and Howard 2016). If the outcomes involve several attributes, the risk attitude may not be the same for all of them. One simple solution adopted in this article is to measure all attributes with the same unit (e.g., US dollars) to obtain a single input in the considered utility function. Biases can occur in the selection of training data and the processing of information. Note that the facts involved in the AI-generated information are assumed to be free of preferences, even though, in reality, many types of biases may affect both the information as gathered and processed and the decisions that it supports. Biases can occur, for instance, in the selection of the training data, their processing by the algorithm, and their interpretation by the decision-maker. The information in the AI system could thus be right or wrong, but the facts—if not the judgment of the facts—are generally verifiable. Yet, it is clear that information biases may distort the decision process, and step one is to check that the information is right. Biases can occur, for instance, in the selection of the training data, their processing by the algorithm, and their interpretation by the decision-maker. In that respect, the case of decisions is different because there is no “right” risk attitude, and each decision-maker may feel differently. That is why the relevant risk attitude has to be determined as a function of specific decisions and of who makes them (Hassan et al. 2024). Differences in risk attitudes between the human decision-makers and the AI system that gives them recommendations prevent relevant AI advice. The AI system, however, may not involve the relevant risk attitude for several reasons. The analyst may not know who is or will be the decision-maker, the system may be defined to advise a group of people who may not have the same utility, or the AI prescription may reflect crowd decisions that have been made in the past in a different environment. In that case, the problem is to align the AI risk attitude with that of the anticipated decision-maker(s) or affected organization. This requires encoding a common risk attitude, then, if appropriate, accessing the risk attitude factor of the AI system and aligning it to fit the decision-makers’. Addressing the alignment problem thus requires at least two things: accessing the risk attitude encoded in the AI system (Russel and Norvig 2021) and knowing that of the decision-maker(s), with the understanding that preferences may vary over time. For a decision-maker to adopt the decisions of the AI algorithm, these decisions need to be guided by his or her risk attitude. One may thus have to adjust that of the AI algorithm. To do so, one needs to have access to that factor as encoded by the decision analyst and the ability to modify it. Differences in risk attitudes between the human decision-makers and the AI system that gives them recommendations prevent relevant AI advice. The problem of AI systems alignment for different users has been considered in the last few years from different perspectives (Caburao 2025; Gent 2023; Iason 2020; Ransbotham et al. 2022; Wixom et al. 2020). In particular, the alignment problem has been described as being at the interface of risk analysis and decision analysis (Abbas and Howard 2016). Maarten Sap, for instance, tests the social intelligence and the capacity for AI systems to interact with society at large (Sap 2024). This article focuses mostly on individual actors, although it encompasses the interaction between AI recommendations and society, considered as a mega decision-maker. The alignment of the AI system is relevant to a variety of real-life situations. One way to describe the alignment problem and its importance in real life is to consider actual situations in which decisions under uncertainty are guided by AI systems. In what follows, four types of situations are described that involve both human and engineering elements: 1) A patient’s decision of whether or not to take a medical test; 2) A national security situation involving attack drones; 3) Sports decisions with a specific application to sailing races; and 4) The design of autonomous vehicles, considering their response to different conditions and actors on the road. In each of these cases, what matters most are the circumstances of the decision, the options, the uncertainties, and the risks involved, including their possible consequences. In each case, a decision analysis can reveal the effects of the risk attitudes and of potential discrepancies, which justify the alignment of the AI system. Case #1: A Medical Test The AI system can assess the information but not make the decision to take the test given the risks. Assume that you are a witness in a doctor’s office. A patient had agreed to a mammography, and the result was essentially negative, except for a minor problem with the imaging technology. She agreed to a needle biopsy, which came back negative, but there was still a little fuzziness in the image. An AI system was programmed to provide a recommendation (AlSamhori et al. 2024), which, in this case, was to proceed to further testing via a surgical test. That AI advice was based on the image and its mild uncertainty, as well as the age of the patient. The system was risk averse to protect patients and apparently made its decision based on a risk to the general population. The patient realized that she was less risk averse than the AI system and that she had additional information about herself. She declined further testing based on her assessment of her own health condition and on a second medical opinion. Twenty-five years later, she is still free of cancer, and her decision turned out to be the right one. In this case, the patient’s decision relied on several factors. She was convinced that although the AI system generally knows better about the risks and the benefits involved, she had a clearer knowledge of her own case: her health, her medical history, and the medical history of her family—and thus of the chances a priori that she had breast cancer. Also, she was sensitive to the risks of an invasive surgical test, the possibility of infection, and the pains of recovery. Note also that some AI recommendations regarding cancer treatments, such as those of the IBM Watson system, have been wrong in the past (Lohr 2021). The patient was not in the best position to assess the chances of a false negative, but she had the benefit of the opinion of an expert who could access the image. Based on that information, the patient was willing to face later the remote possibility of a cancer that could have been avoided by early detection. These uncertainties, the patient’s valuation of the various outcomes and of the trade-offs involved, her time value (discounting), and her risk attitude were thus at the core of her decision. The patient needs to know how the AI recommendation was made. Although the patient had the benefit of an expert opinion, to seriously consider the AI system’s surgery recommendation, the patient needed to know the basis of the AI-generated information related to mammography (Branco et al. 2024), as well as the preferences that guided the recommendation (Fryer 2024). But that information is seldom available in the medical field, where AI and large language models are increasingly used, both for diagnosis and prescription (Khalifa et al. 2024; Magrabi et al. 2019; Rajpurkar et al. 2022). Most people may not understand the difference between the facts that guide the AI reasoning and the (arbitrary) preferences encoded in the prescription system. Therefore, patients may simply adopt whatever the algorithm recommends, based on the fact that it generally knows more than they do, but they seldom consider the possibility that the system may have a risk attitude different from theirs. Increasing patients’ access to the model parameters and patient education would result in better alignment of the AI system and strengthen the system’s role in patients’ decision-making. Case #2: The Use of Autonomous Drones in Combat The United States does not allow the use of autonomous drones in combat, but some of its adversaries do. Most drones are used in the US armed forces for reconnaissance and for attacking targets under human judgment. There is thus a human in the loop, both in the attack and in the response that may follow. By contrast, autonomous drones and other lethal autonomous weapons systems are programmed to make decisions without direct human involvement when anticipating threats, identifying targets, and attacking enemy positions (Garamone 2023; Springer 2013). Yet, the distinction between controlled and autonomous drones is, in fact, somewhat ambiguous. The US Department of Defense (DoD), although it has developed autonomous drones, requires human judgment in combat. China has also developed unmanned combat aerial vehicles that can perform both reconnaissance missions and precision strikes. Such systems could make attack decisions, which could become critical in Chinese operations in the South China Sea. By contrast, other countries, such as Turkey, Russia, and Iran, have actually used fully autonomous drones in combat. Should the United States change its rules of operations? From the US perspective, the question for the moment is how to deal with a classic attack situation using controlled drones as opposed to fully autonomous ones, when some US enemies do use fully autonomous drones, which would allow them to react instantaneously to the detriment of the United States. In the future, the United States may thus face a critical decision about using fully autonomous forces to match the attack of an enemy using such drones. When using autonomous drones, some adversaries have a critical advantage. Should the United States do so? Two critical questions will thus have to be addressed by the United States regarding AI-guided military drones: 1) How should one use conventional drones (e.g., when should one delay a counterattack)?, and 2) Should there be a human in the loop, or should the United States allow the use of fully autonomous drones? The advantages of autonomous drones in combat may be clear, but they will have to be balanced against their potential risks, especially in front of an adversary that is also using them. The autonomous drone might simply be programmed to destroy everything it can, and given the uncertainties, it could destroy an innocent target or, on the contrary, it could lose a target for being too cautious. And like all other autonomous systems, it might simply make a mistake and, for instance, attack its own forces. Consider, first, the current possibility of an enemy using autonomous drones when the US military is not allowed to do so. After an attack, a US commander has to decide how and when to respond in a situation where attacks can be extremely fast, and speed is critical. That commander may want to launch an immediate counterattack since the conventional forces may be devastated otherwise. Assume that the commander is at a level of authority where it is necessary to consider the national effects of his or her decision, in addition to the immediate tactical ones. The commander may want to take some time to make a strategic decision, considering the long-term consequences of an immediate reaction to try to decrease the risk of escalation. Since an unchallenged enemy attack can be devastating to the United States, the commander may thus face a trade-off between a minor delay that could allow diplomacy to lead to a truce and an immediate response. In that situation, a key decision is thus that of the commander to respond right away with the available forces or to voluntarily delay that response to allow for diplomatic negotiation or another military option. This assumes that the enemy attack does not reflect a software problem in their own drones, which the commander may not be able to suspect. In the future, the DoD may thus have to make a fundamental strategic decision of whether or not to use fully autonomous drones and let an AI system guide immediate automatic responses to an attack. On the one hand, by doing so, the US military might match—or surpass—the enemy forces. On the other hand, it may lose the opportunity of a strategic delay for rational thought in pursuit of combat. The information that will guide these decisions is obviously not available to the public, or even perhaps to the DoD at this time, but the fundamental formulation of the problem is clear. Under the current circumstances, a commander may or may not want to follow or even access an AI system that could yield useful advice but might not reflect his or her risk attitude and preferences in combat operations. If an AI system is to guide attacks by autonomous drones without direct human intervention, it must be devised so that these attacks can be observed and stopped by a decision-maker according to his or her preferences and risk attitude. What is the effect of an AI-controlled drone system on deterrence? Whether fully autonomous drones enhance or decrease attack deterrence is not obvious. On the one hand, they may give incentives to the adversary to slow down, considering the losses that it may incur from an immediate, well-targeted US response. On the other hand, autonomous drones may trigger attacks that could be avoided or delayed for a better chance of an expeditious move towards a truce. There is also the possibility of the enemy hacking the United States’ AI and rendering it useless, but it is difficult to “hack” humans, making classic systems more reliable in that respect. In any case, the commanders must be able to stop automatic AI combat systems, which have to be aligned to their preferences to permit choices that may limit destruction on both sides. But one must recognize that this ability may render the United States’ AI system vulnerable and ineffective. Case #3: AI in Sailing Races There is recent evidence of the value of superior AI systems in regattas. The victory of the Emirates Team New Zealand over its adversaries in the America’s Cup of October 2024 (EmiratesTeamNZ 2024; Goodhart 2021) is a recent demonstration of the value of a superior AI system in a major regatta. There were actually two distinct applications of AI, one in the design of the boat and the other as an operational advisor to the skipper. The boat was an AC75 foiling monohull yacht that included a futuristic hydrofoil designed with the help of AI, which allowed the boat to fly above the water and travel at over 100 km/h (Gladwell 2021). The AI system on board was autonomous and adapted to the boat but the skipper still had some critical decisions to make. The skipper gets information from that system but often still faces uncertainties about possible variations of the actual weather, as well as the position, speed, and strategy of the competitors. Therefore, he or she may have to make decisions under uncertainties that vary constantly, given his or her main objective about what rank to aim for in the race and what risk to take. Should the skipper follow the advice of the AI system or account for his or her immediate instincts? The skipper can consider two options: to trust the AI system and follow its recommendations to shift or not shift the course of the boat, or to tighten or relax the sails (Alves et al. 2017; Anderson et al. 2023; Tagliaferri et al. 2016; Tagliaferri and Viola 2017). The alternative is to keep the boat on its course as planned. In this decision, the skipper faces several sources of uncertainty: the accuracy of the AI sensors that provide information and guidance but may be failing or still learning; changes in the environmental conditions, such as wind and waves, that the AI system may not have anticipated; the moves and the strategy of the competitors; and, finally, the risk of collision with other boats in the race, for instance, around a buoy, which the skipper needs to manage carefully when choosing a course and setting the sails. The skipper may have the option to specify his or her risk attitude at the time of the design of the AI system or before using it, but that risk attitude may change during the race—for instance, with unexpected performances of the competitors. Yet, a good alignment based on experience can be set at the onset of the race. In the case of New Zealand’s victory in the 2024 America’s Cup, the owner, the skipper, and, to some degree, the crew were involved from the beginning in the design of the AI system, including its role in the design of the boat and the choice of maneuvering options. All uncertainties were not resolved, but there was a good alignment of preferences in a top-class AI system. Will AI kill the legitimate competition of sailing races? All boats, however, may not be guided by an AI algorithm of that power, even if they have AI support (Scuttlebutt Sailing News 2023; van Aartrijk 2002). One may wonder at what point the sport aspect of regattas will be dominated by technology rather than the skills of the crew. Meanwhile, in addition to AI guidance, the boats will still have to be able to follow the instincts and the skills of the skipper, given good but imperfect AI-generated information, the skipper’s assessment of the key factors in the race, and his or her risk attitude. The relevance of the AI system, as in previous cases, thus depends on its alignment with the general risk attitude of the skipper. The capabilities of the top existing AI systems are confidential and carefully protected. Therefore, one does not know exactly what dynamic adjustments will be feasible, but improvements of their alignment and possible changes of preferences can give these boats even greater power than sophisticated algorithms with fixed preference parameters. Case #4: Autonomous Vehicles The AI system for AVs should be designed to accommodate a variety of human drivers and customers. Fully autonomous vehicles (AVs)—as opposed to automatic ones—are totally guided by artificial intelligence; hence, there is a risk of AI malfunctions that may affect passengers (Ajenaghughrure 2020; Thomas 2024; Tong et al. 2023; van der Smagt 2021). The AI system makes decisions that would otherwise be made by a human driver when the AV drives among conventional vehicles (CVs). Comparing risk attitudes is thus complex because of the diversity of drivers, their behaviors, and their risk-taking, especially in situations that may lead to a rear-ending accident. When the AI system works as it should, the AVs follow the rules of the road, whereas some of the CV drivers may not. The AV risk management decisions thus have to be anticipated by the AI analyst when setting the rules of driving in a number of situations, including cars driven as expected or at different speeds and other drivers obeying or not obeying the rules of the road. Similar problems may also occur when AVs, obeying all the rules, do not drive as expected by other drivers. Managing the risk of an AV accident is complex. The designer of the AI system thus has to manage the risk of accidents involving AVs. First, technical accidents can be caused by a malfunction of the sensors, actuators, and vision capabilities of the AV (Wang 2022). For example, in 2023, an AV nearly drove into a trench at a construction site in San Francisco (The San Francisco Standard 2023). More generally, AVs can be challenged by poor visibility and other vehicles. These recognition issues are engineering problems that can be addressed as such. But most accidents of AVs in mixed traffic are rear-ends caused by CVs (Cunneen 2023). Again, this may have to do with differences in speed, especially in areas where the speed is limited. The risk attitude of the AV can perhaps be automatically modified if it perceives an immediate danger, but that flexibility may create another risk if the system malfunctions. In many cases, educating common drivers so that they are familiar with the functioning of AVs could help reduce risk (Dragomir et al. 2024). Similarly, the passenger who decides to ride in an AV is the one who takes the risk inherent to the vehicle, and many people are still uncertain about taking that risk (Naiseh et al. 2024). One can perhaps imagine a system in which the passenger has access to the guidance system through a knob on the board that would allow him or her to post a risk aversion level and to align it to that of the AI system. However, there might be liability issues if the AV got into an accident when the knob was set at a high-risk tolerance. Conclusion When using an AI system in risk management, the challenge is to ensure its alignment with the risk attitude of the user whenever feasible. The decision-maker, supported by an AI system, still plays a major role in the choice of management options. A patient advised by an AI system should know the source of the information to be able to make a personal decision. A commander who operates automatic combat drones has to be able to override the AI decisions based on the knowledge he or she gained from wargaming. The skipper who runs in a race can be trained on a simulator and acquire the skills that allow him or her to adapt the AI system’s decisions to the circumstances. And the designer of an autonomous vehicle should account for the risk attitude of the expected customers when designing the safety system. When using an AI system in risk management, the challenge is to ensure its alignment with the risk attitude of the user whenever feasible. One of the analytical challenges is to consider the dynamics of the situation and the variations of risk attitudes over time. To do so requires that the analyst know the decision-maker’s preferences when designing the algorithm. If the situation requires it, that alignment must be adjusted at decision time, which will require transparency and flexibility of the AI system. This will imply allowing for disclosure and possibly adjustment of its risk attitude for a specific decision-maker or a homogeneous group of people affected. Acknowledgment The author is most grateful for the advice of Professor Peter Glynn and of Admiral James O. Ellis, both members of the NAE. References Abbas A, Howard R. 2016. Foundations of Decision Analysis. Pearson. Ajenaghughrure IB, da Costa Sousa SC, Lamas D. 2020. Risk and trust in artificial intelligence technologies: A case study of autonomous vehicles. In: 13th International Conference on Human System Interaction, 118–123. IEEE Computer Society. AlSamhori JF, AlSamhori ARF, Duncan LA, Qalajo A, Alshahwan HF, Al-abbadi M, Al Soudi M, Zakraoui R, AlSamhori AF, Alryalat SA, and 1 other. 2024. Artificial intelligence for breast cancer: Implications for diagnosis and management. Journal of Medicine, Surgery and Public Health 3:1–10. Alves B, Veloso B, Malheiro B. 2017. An Agent-Based Platform for Autonomous Sailing Research and Competition. In: Robotic Sailing 2017: Proceedings of the 10th International Robotic Sailing Conference, 31–38. Øvergård KI, ed. Springer. Anderson J, Sithungu S, Ehlers E. 2023. Route optimization for sailing vessels using artificial intelligence techniques. In: Proceedings of the International Conference of Computational Intelligence and Intelligent Systems, 60–66. Association for Computing Machinery. Branco PESC, Franco AHS, de Oliveira AP, Carneiro IMC, de Carvalho LMC, de Souza JIN, Leandro DR, Cândido EB. 2024. Artificial intelligence in mammography: A systematic review of the external validation. Rev Bras Ginecol Obstet. Online at doi 10.61622/rbgo/2024rbgo71. Caburao E. 2025. Leveraging AI in risk management for effective implementation. Safety Culture, Feb 12. Cunneen M. 2023. Autonomous vehicles, artificial intelligence, risk and colliding narratives. In: Connected and Automated Vehicles: Integrating Engineering and Ethics. Studies in Applied Philosophy, Epistemology and Rational Ethics, vol 67, 175–95. Fossa F, Cheli F, eds. Springer. Dragomir D, Popișter F, Kabak KE. 2024. Using AI tools to enhance the risk management process in the automotive industry. In: Advances in Manufacturing IV, volume 2 (Lecture Notes in Mechanical Engineering), 189–98. Trojanowska J, Kujawińska A, Pavlenko I, Husar J, eds. Springer. EmiratesTeamNZ. 2024. The Emirates Team New Zealand Story. Youtube, Oct. Fryr N, Groß D, Lipprandt M. The ethical requirement of explainability for AI-DSS in healthcare: A systematic review of reasons. BMC Medical Ethics 25: article 104. Garamone J. 2023. DOD Updates Autonomy in Weapons System Directive. DoD News, US Department of Defense, Jan. Gent E. 2023. What is the AI alignment problem and how can it be solved? New Scientist, May 10. Gladwell R. 2021 America’s Cup: Emirates team New Zealand use artificial intelligence to find the fastest way. Sail-World, March 4. Goodhart B. 2021. How team New Zealand did the unexpected with its America’s Cup boat. GQ, March 10. Hassan N, Slight R, Bimpong K, Bates D, Weiand D, Vellinga A, Morgan G, Slight S. 2024. Systematic review to understand users’ perspectives on AI-enabled decision aids to inform shared decision making. npj Digital Medicine 7: article number 332. Iason G. 2020. Artificial intelligence, values and alignment. Minds and Machines 30:411–37. Khalifa M, Albadawy M, Iqbal U. 2024. Advancing clinical decision support: The role of artificial intelligence across six domains. Computer Methods and Programs in Biomedicine 5: article number 100142. Magrabi F, Ammenwerth E, McNair JB, Dde Keizer NF, Hyppönen H, Nykänen P, Rigby M, Scott PJ, Vehko VT, Wong S-Y, Georgiou A. 2019. Artificial intelligence in clinical decision support: Challenges for evaluating AI and practical implications. Yearbook of Medical Informatics 28(1):128–34. Naiseh M, Clark J, Akarsu T, Hanoch Y, Brito M, Wald M, Webster T, Shukla P. 2024. Trust, risk perception, and intention to use autonomous vehicles: An interdisciplinary bibliometric review. AI & Society. Online at https://doi.org/10.1007/s00146-024-01895-2. Paté-Cornell E. 2024. Preferences in AI algorithms: The need for relevant risk attitudes in automated decisions under uncertainties. Risk Analysis44(10):1–7. Online at https://doi.org/10.1111/risa.14268. Rajpurkar PE, Chen E, Banerjee O, Topol EJ. 2022. AI in health and medicine. Nature Medicine 28:31–8. Ransbotham S, Kiron D, Candelon F, Khodabandeh S, Chu M. 2022. Achieving Individual- and Organizational-Value with AI, Findings from the 2022 Artificial Intelligence and Business Strategy Global Executive Study and Research Project. MIT Sloan Management Review. Russel S, Norvig P. 2021. Artificial Intelligence: A Modern Approach, 4th edition, Pearson Series in Artificial Intelligence. Pearson. Sap M. 2024. Artificial social intelligence? On the challenges of socially aware and ethically informed large language models. The Bridge 54(4):20–23. Springer PJ. 2013. Military Robots and Drones: A Reference Handbook. ABC-CLIO. Tagliaferri F, Hayes B, Viola IM, Djokic S. 2016. Wind modelling with nested Markov chains. Journal of Wind Engineering and Industrial Aerodynamics 157:118–24. Tagliaferri F, Viola IM. 2017. A real-time strategy-decision program for sailing yacht races. Ocean Engineering 134:129–39. Thomas J. 2024. Integrating machine learning and AI in automotive safety. International Journal of Innovative Science and Research Technology 9(1). Online at https://doi.org/10.5281/zenodo.10670478. Tong K, Guo F, Solmaz S, Steinberger M, Horn M. 2023. Risk monitoring and mitigation for automated vehicles: A model predictive control perspective. In: 2023 IEEE International Automated Vehicle Validation Conference (IAVVC), 1–7. IEEE. van Aartrijk ML, Tagliola CP, Adriaans PW. 2002. AI on the ocean: The RoboSail Project. In: Proceedings of the 15th European Conference on Artificial Intelligence, 653–57. IOS Press. van der Smagt P. 2021. Artificial intelligence in the automotive industry. Joint Aida-Tran Hearing on AI and Transportation, EU Transport Policies. How to prepare for AI while minimizing risk. Zigoris J. 2023. Driverless Waymo self-driving car almost digs itself into hole–literally. The San Francisco Standard, Jan 15. Wang D, Fu W, Song Q, Zhou J. 2022. Potential risk assessment for safe driving of autonomous vehicles under occluded vision. Sci Rep 12: article number 4981. Wixom B, Someh I, Gregory R. 2020. AI alignment: A new management paradigm. Research Briefing. MIT Center for Information Systems Research. About the Author:Elisabeth Paté-Cornell (NAE) is a professor of engineering at Stanford University.