People are wary of AI giving moral advice — especially when it's utilitarian
Do people think that an AI giving moral advice would be as trustworthy as a human, and does this depend on the kind of moral judgments that an AI might give? In this work from the project led by Simon Myers along with P.I. Jim A. C. Everett, we examined these questions and found people trust so-called artificial moral advisors (AMAs) less than humans particularly when AI systems base their advice on utilitarian principles.
We conducted four pre-registered studies involving over 2,600 participants and found that people showed clear algorithm aversion in the moral domain, trusting human moral advisors more than AI counterparts, even when both gave the same moral advice. Moreover, we found that this depended on the kind of moral judgments the advisor made. When advisors made utilitarian decisions about harm to some for the greater good (“instrumental harm”) or decisions about prioritising the welfare of a greater number of strangers over people closer to home (“impartial beneficence”), they were trusted less. And it was not just that people trusted AI that made characteristically utilitarian decisions less, but they also thought that AI would be more likely to make these decisions: people both expect AI to be more utilitarian, but also trust it less for doing so.
Describing the results, Jim said:
“Our findings show that trust in AI isn’t just about whether a system is accurate, it’s about whether people feel comfortable with the kind of moral reasoning the AI uses. If we want AI to play a role in helping people navigate difficult ethical decisions – and that is a big if – we need to understand these intuitions that make people wary of machine-generated moral advice.”
Read More:
Myers, S., & Everett, J. A. (2025). People expect artificial moral advisors to be more utilitarian and distrust utilitarian moral advisors. Cognition, 256, 106028.
As machines powered by artificial intelligence increase in their technological capacities, there is a growing interest in the theoretical and practical idea of artificial moral advisors (AMAs): systems powered by artificial intelligence that are explicitly designed to assist humans in making ethical decisions. Across four pre-registered studies (total N =2604) we investigated how people perceive and trust artificial moral advisors compared to human advisors. Extending previous work on algorithmic aversion, we show that people have a significant aversion to AMAs (vs humans) giving moral advice, while also showing that this is particularly the case when advisors – human and AI alike – gave advice based on utilitarian principles. We find that participants expect AI to make utilitarian decisions, and that even when participants agreed with a decision made by an AMA, they still expected to disagree with an AMA more than a human in future. Our findings suggest challenges in the adoption of artificial moral advisors, and particularly those who draw on and endorse utilitarian principles – however normatively justifiable.
https://www.sciencedirect.com/science/article/pii/S0010027724003147