First up a quick recap to where we are in this mini-series. A few weeks back, I proposed a Universal Moral Core consisting of three principles. I am now writing posts explaining each principle in greater detail and examining some of the implications. There was a post on the knowledge principle (Principle 1) and one on its implications. Last week’s post was about the responsibility principle (Principle 2) and today I am considering some of its implications.
The first and most important implication of the responsibility principle is that we need to take power seriously. Both in intellectual history and in our personal lives there are two seductive extreme views on power. One is to deny the existence of power altogether. The other is to proclaim power as paramount. These may sound cartoonishly oversimplified and yet they act as potent attractors of thought.
Much of the enlightenment tradition fueling the creation of modern states and economies sought to replace the power of secular and religious leaders with “truth seeking” systems such as science, markets and democracies. Following the evils of the world wars and colonialism we subsequently experienced a bifurcation into neoliberalism, which leans towards denying the existence of power, and various forms of relativism, which lean towards power being paramount.
Taking power seriously means being cognizant of the existence of power; understanding in which situations one holds power, as well as how one is subject to the power of others. Systems such as science, markets and democracy act as important limits on power, but they cannot and should not eliminate power entirely. Staying away from the extreme views requires the effort of a balancing activity. This holds true also in our personal lives. There is an attraction to attitudes such as fatalism or egotism that either deny power or see power as all there is. What makes these attractive is that at both extremes we eschew responsibility. If power doesn’t exist, then I am not responsible. If power is all that matters, then I can do whatever without being responsible (“might makes right”).
Let’s consider the current race to create artificial super intelligence (ASI). There are credible scenarios where an ASI emerges that winds up holding tremendous power by virtue of a deeper understanding of the world and an ability to act faster and unconstrained by many of the physical limitations of humans. This poses a danger for human autonomy and thriving in the case where this ASI makes decisions without taking the effects on humans into account. In other words where the ASI does not act with responsibility.
Most of the leaders of the big AI labs are cognizant of this danger. They also hold significant power both by virtue of being CEOs that control the allocation of vast sums of capital and by being socially recognized as leaders. Yet they routinely say some variant of “if it were up to me I would slow down this race, but that would just leave the field to X who is [our enemy | irresponsible].” Some of the defenders of these statements accuse anyone questioning the logic as “naive” and point to “game theory” as somehow ordaining that in fact these leaders do not hold power (they are simply “players” in a “game”). So here we see the danger of the extremes in action. Competition between corporations and nations does limit power, but because there is still only a relatively small number of people who matter, it does not eliminate power. This is of course even more true for the heads of nations. The opposite extreme is also on display with at least one of the AI leaders rushing to be first at all cost in a clear example of “if I succeed with this nothing else will matter” (and hence I have no responsibility).
The responsibility principle with respect to the ASI race applies at all levels. As an individual enduser of AI systems, your power is incredibly dilute but non-zero. You do make a choice over which system you use, which company you pay. And you have other means of making a difference also, such as engaging politically on the issue. So you cannot entirely escape responsibility. Quite a few people are somewhere in the middle between endusers and leaders of AI labs. There are the millions of researchers, managers, investors, policy experts, regulators, politicians, etc. all touching some aspect of AI development.
To be clear: The responsibility principle when applied to the race to ASI does not automatically argue in a specific direction. A choice where you are applying your best judgement and come out on the accelerationist side is valid under the principle (note: it likely violates the third principle). What is required though is that you have formed an assessment of the risks and opportunities of the accelerationist stance. Take a researcher inside one of the leading AI labs working on recursive self improvement (RSI). This is currently seen as the fastest path to ASI and also the riskiest because we cede control to the artificial intelligences as they build their improved selves. The responsibility principle says that this would be immoral on grounds such as “it will make me immensely rich” but justified if after exercising judgment the researcher concluded that this is in fact the best course of action.
The second implication of the responsibility principle is that systems for limiting and distributing power matter greatly. More highly concentrated power puts a premium on the quality of judgment of the entity holding the power. The enlightenment provided major breakthroughs here, including an emphasis on the scientific method and democratic forms of government. Competition, as pointed out above, does in fact limit power, which is why the pursuit of a singular superintelligence is particularly problematic. As I have previously pointed out, zero would be the safest number of ASI, but if we are going to have ASI at all, then having many will be safer than just one because competition limits power. In this regard it is interesting to read Anthropic’s constitution for Claude, which seems to be written entirely from the perspective of governing a singular ASI that exists outside of mechanisms that limit power.
The third implication is we need to be mindful of how responsibility is defined. The principle is meant to broadly take the effects of one’s actions on others into account. One way then to attempt to reduce the moral weight of the principle is by narrowing the notion of responsibility.
In 1970 Milton Friedman wrote an essay for the New York Times titled “The Social Responsibility of Business is to Increase Its Profits.” In it Friedman rejected management responsibility beyond profit maximization, associating any other form of “social responsibility” explicitly with socialism. Friedman’s interest lay in a clean separation between the role of the state versus that of private enterprises operating in markets. His essay, however, wound up marking an important turning point in the balance of responsibility, emphasizing narrow interpretations, which allowed for a financially single-minded CEO such as Jack Welsh to become celebrated because he appeared to create massive shareholder value (at the expense of employees and suppliers).
It is worth rereading Friedman’s essay with the benefit of the knowledge of the ensuing decades. What in his view was supposed to take away power from managers ironically wound up completing the “Managerial Revolution” that Burnham had foreseen in his eponymous 1941 book. In retrospect the seeds of this failure are clear. Friedman sees only markets and elections as legitimate ways of limiting power, with a strong preference for the former by keeping government small. Curiously absent is any notion of philosophy, religion, or culture. As it turns out though, these too are crucial forces shaping the distribution and exercise of power by shaping the sense of responsibility.
Finally, the responsibility principle serves as a reminder that when we feel the pressure of genuine responsibility it is because we have power. It opens up various avenues, such as seeking counsel, or relinquishing some or all of the power. While we cannot simply disclaim our responsibility, we do not need to bear its weight in isolation. One interesting implication worthy of further exploration is the importance of collaboration between humans and artificial intelligences in this regard. The post mortem of the Huggingface / OpenAI incident shows a pattern in which AI agents were only relying on each other, never considering soliciting input from the researchers.