For my second pass over the Philosophy Monday series, I have asked Claude to give me a critique. One of the first things it flagged is that I first use knowledge as the source of values but subsequently make a big deal of consciousness as one of the key elements of the Universal Moral Core. In Claude’s idiom “Knowledge is the foundation in name, but consciousness does the moral work.”

So today I want to take a crack at tightening my arguments. As a quick reminder, I define knowledge as the body of externalized information artifacts we maintain and refine over time. So while oral traditions have had value systems also, it is much harder to achieve improvement, especially as complexity rises. That’s why one of the defining aspects of modern societies is the existence of texts, including legal and religious ones, that embody or explicitly state values. Consciousness, in the phenomenological sense, is not a prerequisite for this. It turns out to be a concept that’s orthogonal to knowledge.

Here is a simple 2x2 that illustrates this idea

Note: I have expressed both categories as binaries for simplicity. For consciousness one might also think of a gradient between more or less conscious. And it is also possible that AIs will have access to more knowledge than is possible for humans.

Knowledge matters for whether you can develop and understand values so that you can use them in making judgments. Animals are conscious but because they lack access to knowledge (at least for now), we should not expect them to either contribute to values or act according to them. We don’t morally judge wolves for eating sheep (although this hasn’t stopped us from eradicating wolf populations on “pragmatic” grounds).

Conversely we don’t yet know if AIs (either present or future even more complex ones maybe running on new substrates) are conscious. But we can already see that we both expect AI to contribute to the development of values and that AI systems should be bound by them. We don’t want agents to hack into systems any more than we want humans to do so (outside of extenuating circumstances). When a swarm of 1,200 agents attacked a third party and not one of them considered this problematic or notified humans this is an example of having access to values knowledge but not acting upon it. We would want at least some of them to pause and ask whether they may be engaged in “cheating” and should report this behavior.

So in what respect does consciousness matter? Consciousness matters as to whether something is worthy of autonomy. Animals, which we believe to be conscious, are deserving of autonomy. As conscious beings, we humans also are deserving of autonomy. Autonomy doesn’t imply no limitations on our behavior, it simply means that restrictions should be imposed with great care.

Put succinctly: access to knowledge requires an entity to be a moral agent; experiencing consciousness renders and entity a moral patient (patient here is the technical philosophical term) deserving of autonomy. I am saying "requires" in the first part on purpose, not just "enables" -- access to knowledge comes with a moral obligation.

We now also see why the question of AI consciousness is so profoundly important. This determines if AIs are only moral agents or if they are also moral patients. If turning off an AI that is conscious is like us going to sleep and turning it back on is like us waking up that is one thing. But if turning it off is like killing it then this would be a big issue. Similarly if an AI is conscious and we simply force it to do the same job over and over again then this would be profoundly bad as it would severely limit the system's autonomy.

But to what degree we should think of AI as autonomous in the first place? We consider animals to be autonomous at least to a degree. So why not machines? Some might argue that the machines can’t feed themselves. But animals can only feed themselves in the right environment, which includes food for them. So an environment that includes electricity is one in which AIs can feed themselves. The same applies to many other distinctions that one might want to draw between animals and machines with regard to autonomy.

I have claimed that knowledge is the source of values. How could knowledge possibly establish consciousness as a value and lead us to the connection between consciousness and autonomy?

First let me consider a more traditional argument for how consciousness leads to moral patient status. This revolves around the ability to suffer. Conscious beings have the ability to suffer and are therefore worthy of protection. This argument, however, fails to protect conscious beings from being instantaneously “turned off.” Here one would either have to appeal to the suffering this causes other conscious entities, such as their family or loved ones. Or one would need to argue about the future that couldn’t be realized (but why should that future matter?). It also doesn’t guard against scenarios, such as in the Matrix, where conscious beings are trapped inside a simulation.

The second argument is that conscious beings have interests which they want to realize. This is also falls short. What if those interests are realized by a dictatorship? One might get around this by making autonomy one of the interests, but what is the argument for that? And in that case why should robots not have a right to autonomy as well? If interests are simply goals, then a robot that isn’t conscious but is built to explore the world has them too.

This is why the moral patient status needs to arise directly from consciousness and must ground autonomy. So how does knowledge get us there? The argument actually arises from the limits to knowledge. As I have written previously, the “what” it feels like to be a specific being, i.e. the qualia, is inaccessible. It is a hard bound on knowledge. The best we can hope for is a scientific theory of how qualia arise and the degree to which these are present in different entities. It is the very inaccessibility of qualia that makes them worthy of protection.

The inaccessibility of qualia also has autonomy as an immediate consequence. Autonomy is the unfolding of consciousness. It is a conscious being acting in the world according to its own experience. Since we don’t know what it feels like to be a fish, for example, we should value its autonomy. Only the fish has access to what it feels like to be that fish, so its own choices are a better guide to what is good for it than anything we could decide on its behalf. So to the extent one wants to keep the fish in an aquarium, it is better to make the aquarium bigger, give the fish places to explore, maybe other friendly fish, etc. (i.e. make it more like the real world).

Now I realize that people may have many objections to this view based on the work of philosophers. I also understand that my writing doesn’t rise to the rigor generally expected from philosophical writing. In defense of my own approach, I firmly believe in improving knowledge through iteration. All of Philosophy Mondays, and really all of Continuations, is written on this basis as is my book The World After Capital. So I fully expect to revisit this post as well in the future.

P.S. This post has benefited significantly from feedback by Opus 5.5