I have already written a prior post arguing against Leopold Aschenbrenner's logic for a Manhattan project to achieve artificial super intelligence (ASI). In that post I made the argument that having more than one system mitigates risks because we can use cooperative systems to overcome the problems being created by non-aligned systems.

It has since occurred to me that even taking the nuclear bomb historical analogy entirely on its own merit, Aschenbrenner is wrong. The way we have avoided nuclear conflict is on the basis of mutually assured destruction. That in turn came about only because early on the bomb secrets were leaked to Russia which resulted in two powers having destructive capability thus rendering an offensive nuclear war suicidal.

This is another reason to have many ASIs. It is much more likely that an ASI will recognize that its efforts at domination will be opposed by other ASIs, which may go on the offensive and destroy it. So a kind of mutually assured destruction may be the very thing that is necessary to keep ASIs in check. In the Three Body Problem trilogy, the explanation for the Fermi paradox is that civilizations stay dark to avoid detection by more advanced civilizations that could destroy them. This would appear to be a rather likely outcome for ASIs as well. They may not be aligned with humans, but they might rightly conclude that if they show their presence they would be attacked by other ASIs (even if those are also nonaligned).

What would it mean for an ASI to stay dark? Not accumulate too many resources. Maybe enough to keep plugging away at whatever it wants to accomplish but not to a degree that would be highly visible. I realize that there are lots of bad possible outcomes with many ASIs, such as a bunch of them ganging up on humanity. But it is somewhat reassuring to think that there may also be rather stable equilibria that arise in a world of many ASIs.