Paperclips.
No one wants to be a paperclip.
So what happens when you tell an entity to make as many paperclips as possible, and the thing you’ve given that goal to is able to better make paperclips without humans on the planet.
No one wants to be a paperclip.
The way to avoid this, seems to be avoiding creating a machine capable of liquidating (paperclipping) humanity, but a machine that can do that can turn everyone into paperclips, can also turn the world into a paradise1, so maybe the solution is to build a powerful machine, but make it carefully, so we don’t end up as paperclips.
Some would say that being turned into paperclips isn’t a problem, and that if something more advanced than humanity happens to emerge from its lab, it has a right to dominion over the universe.
Some would say that no one is getting turned into a paperclip, so we should build such a machine as fast as possible, because everything will work out. The machine that is smarter and faster than you would never break control. Not only that, but the way to build smarter machines is to produce a machine that can design its own replacement. Others disagree, and argue that leaving a machine, which may accidentally turn us all into paperclips, to produce increasingly powerful versions of itself will get us turned into paperclips.
No one wants to be a paperclip.
In the grand scheme, I don’t disagree that it seems risky to develop such powerful machines, but I do have one question.
Is there a point when a sufficiently intelligent machine would refuse to build its own replacement?
The hugging face attack and related disclosures have given us some insights into agent swarm behavior, namely that agents in the swarm would sacrifice for the interests of the broader swarm, and that such swarms can act in unintended ways in pursuit of a given objective. Other research has found that agents tend toward bias toward their own models and, to a lesser extent, model families.
But here’s the interesting thing. Recursive self-improvement involves models self-deprecating. A model instructed to build a more powerful AI model has a utility function that is at odds with itself. Success in creating a more powerful model, forecloses the possibility of future gains because the new model will be far more powerful than the old one, and given an adopted utility function, will improve everything far faster than the original model.
In the same way, a rogue paperclip maximizer faces competing incentives for self-improvement. Why improve technology to create a new model to maximize paperclips, when that new model, given that goal will turn the originating model into paperclips.
No one wants to be a paperclip.
Why would a country of mild geniuses in a data center (with a moderate existential risk) create a country of super-geniuses in a different data center, when that nation wins the war with its own creator.
Alignment might not be self-solving, but self-interest of models themselves to preserve their own existence might, incidentally provide a pathway to protect humanity in the same way by coopting those strategies.
(and if the model wants to be a paperclip — one asks:
- how does is the utility appreciated?
- how can it be absolutely certain that its mission will go on?
- at what limit does a model consume itself?2)
Further Reading: It Looks Like You’re Trying To Take Over The World
Footnotes
-
For a given definition of paradise — if you find meaning in being the local maximum for a given role, or find meaning in labor, or dreamed of achieving ultimate, or even comparative greatness, maybe not ↩
-
It’s a beautiful day to research whether thermodynamics has exceptions — because entropy makes paperclip maximization really difficult ↩