Moral disagreement and artificial intelligence

Abstract

AbstractArtificially intelligent systems will be used to make increasingly important decisions about us. Many of these decisions will have to be made without universal agreement about the relevant moral facts. For other kinds of disagreement, it is at least usually obvious what kind of solution is called for. What makes moral disagreement especially challenging is that there are three different ways of handling it.Moral solutionsapply a moral theory or related principles and largely ignore the details of the disagreement.Compromise solutionsapply a method of finding a compromise and taking information about the disagreement as input.Epistemic solutionsapply an evidential rule that treats the details of the disagreement as evidence of moral truth. Proposals for all three kinds of solutions can be found in the AI ethics and value alignment literature, but little has been said to justify choosing one over the other. I argue that the choice is best framed in terms ofmoral risk.