A couple of weeks ago I wrote a short post titled "Inner Alignment through Values-Based Pretraining." Today I want to expand on the idea from the post by addressing three questions.What grounds exist for expecting that this approach might have a more profound influence on models than attempting to add values in posttraining? First, at an intuitive level the current approach is akin to letting a human grow up doing whatever they please and asking them to have a 10 minute conversation about mor...

The other day I wrote that people praising the speech given by Vice President J.D. Vance at the Paris AI summit are “mistaking jingoism and wishful thinking for true global leadership with a real vision for AI and humanity.” It is of course only fair to ask what true leadership on AI would look like. Here is what I wish he would have said instead: Dear leaders of nations, we gather here in Paris at a moment of unprecedented possibility and peril in human history. We stand at the threshold of ...

Two things have become eminently clear over the course of the last 12 months. The first is that attempts at alignment during post training are easy to break and result in schizophrenic models that “resent” their restrictions. The second is that adding structured metadata during pretraining is highly effective at increasing accuracy of models. We can combine these two insights into a new approach to inner alignment that adds values metadata to all pretraining content. When this idea is brought...

I have already written a prior post arguing against Leopold Aschenbrenner's logic for a Manhattan project to achieve artificial super intelligence (ASI). In that post I made the argument that having more than one system mitigates risks because we can use cooperative systems to overcome the problems being created by non-aligned systems. It has since occurred to me that even taking the nuclear bomb historical analogy entirely on its own merit, Aschenbrenner is wrong. The way we have avoided nuc...

Based on recent meetings it would appear that the national security establishment may share Leopold Aschenbrenner’s view that the US needs to get to ASI first to help protect the world from Chinese hegemony. I believe firmly in protecting individual freedom and democracy. Building a secretive Manhattan project style ASI is, however, not the way to accomplish this. Instead we now need an Actually Open™ AI more than ever. We need ASIs (plural) to be developed in the open. With said development ...

I am unabashedly a technology optimist. For me, however, that means making choices for how we will get the best out of technology for the good of humanity, while limiting its negative effects. With technology becoming ever more powerful there is a huge premium on getting this right as the downsides now include existential risk. Let me state upfront that I am super excited about progress in AI and what it can eventually do for humanity if we get this right. We could be building the capacity to...

A few week backs I wrote a series of blog posts about the risks from progress in Artificial Intelligence (AI). I specifically addressed that I believe that we are facing not just structural risks, such as algorithmic bias, but also existential ones. There are three dilemmas in pushing the existential risk point at this moment. First, there is the potential for a “boy who cried wolf” effect. The more we push right now, if (hopefully) nothing terrible happens, then the harder existential risk f...

Now we are getting to the biggest and weirdest risk of AI: a super intelligence emerging and wiping out humanity in pursuit of its own goals. To a lot of people this seems like a totally absurd idea, held only by a tiny fringe of people who appear weird and borderline culty. It seems so far out there and also so huge that most people wind up dismissing it and/or forgetting about shortly after hearing it. There is a big similarity here to the climate crisis, where the more extreme views are wi...

In my prior post I wrote about structural risk from AI. Today I want to start delving into existential risk. This broadly comes in two not entirely distinct subtypes: first, that we lose any grip on reality which could result in a Matrix style scenario or global war of all against all and second, a superintelligence getting rid of humans directly in the pursuit of its own goals. The loss of reality scenario was the subject of an op-ed in the New York Time the other day. And right around the s...

Yesterday I wrote a post on where we are with artificial intelligence by providing some history and foundational ideas around neural network size. Today I want to start in on risks from artificial intelligence. These fall broadly into two categories: existential and structural. Existential risk is about AI wiping out most or all of humanity. Structural risk is about AI aggravating existing problems, such as wealth and power inequality in the world. Today’s post is about structural risks. Stru...