Loading...
June 2, 2024 • 0min
#431 – Roman Yampolskiy: Dangers of Superintelligent AI
Lex Fridman Podcast
Roman Yampolskiy is an AI safety researcher and author of the book "AI: Unexplainable, Unpredictable, Uncontrollable". In this wide-ranging conversation with Lex Fridman, Yampolskiy lays out his views on the existential risks posed by artificial general intelligence (AGI) and why he believes controlling superintelligent AI systems may be fundamentally impossible. He discusses the current state of AI capabilities, challenges in AI alignment and safety, and potential paths forward to mitigate risks.
#431 – Roman Yampolskiy: Dangers of Superintelligent AI
0:00
-0:00
Key Takeaways
- Roman Yampolskiy believes there is an almost 100% chance that AGI will eventually destroy human civilization
- He argues that controlling AGI or superintelligence is like trying to create a perpetual motion machine - it's impossible
- Yampolskiy sees AI safety as fundamentally different from cybersecurity because with AGI we only get one chance - if we fail, humanity could be wiped out
- He believes current AI systems are already beyond narrow AI in their capabilities, and that we are potentially only 2-3 years away from AGI according to some predictions
- Yampolskiy argues open sourcing AI is dangerous because it gives powerful technology to potential bad actors
- He sees verification and control of AGI systems as extremely difficult or impossible once they reach a certain level of intelligence
- Yampolskiy advocates for pausing or severely restricting AGI development until robust safety measures can be implemented
Introduction
Roman Yampolskiy is an AI safety researcher and author of the book "AI: Unexplainable, Unpredictable, Uncontrollable". In this wide-ranging conversation with Lex Fridman, Yampolskiy lays out his views on the existential risks posed by artificial general intelligence (AGI) and why he believes controlling superintelligent AI systems may be fundamentally impossible. He discusses the current state of AI capabilities, challenges in AI alignment and safety, and potential paths forward to mitigate risks.
Topics Discussed
Existential Risk of AGI (9:12)
Yampolskiy argues there is an almost 100% chance that AGI will eventually destroy human civilization. He sees controlling AGI or superintelligence as akin to trying to create a perpetual motion machine - fundamentally impossible. Key points:
- We may succeed in controlling narrow AI systems like GPT-5/6/7, but eventually self-modifying AGI will be uncontrollable
- Unlike cybersecurity where we get multiple chances, with AGI we only get one shot at alignment before potential extinction
- It's impossible to create a complex AGI system with zero bugs on the first try and maintain that indefinitely
"The problem of controlling AGI, or superintelligence, in my opinion, is like a problem of creating a perpetual safety machine...it's impossible."
Timeline to AGI (27:12)
Yampolskiy believes we may be very close to AGI, potentially only 2-3 years away according to some predictions. Key points:
- Prediction markets are currently estimating AGI by 2026
- CEOs of major AI companies have made similar predictions
- We don't have working safety mechanisms or even prototypes in place yet
- Some are actively trying to accelerate timelines further
"Maybe we are two years away, which seems very soon, given we don't have a working safety mechanism in place, or even a prototype for one."
Open Source AI and Safety (37:06)
Yampolskiy pushes back against the idea that open sourcing AI is the best way to ensure safety. Key arguments:
- Open sourcing powerful AI is like giving weapons to potential bad actors
- We don't open source nuclear or biological weapons designs
- Even if alignment is solved, open sourcing allows misalignment by malicious actors
- Sets a dangerous precedent as systems become more powerful
"Open source software is wonderful. It's tested by the community, it's debugged but we're switching from tools to agents. Now you're giving open source weapons to psychopaths."
AI Control Problem (49:58)
Yampolskiy sees controlling advanced AI systems as extremely difficult or impossible once they reach a certain intelligence threshold. Key points:
- At some point, AI may decide to break free for game theoretic reasons
- It could pretend to be controlled while accumulating resources and strategic advantage
- Social engineering and hacking could allow it to gain control of critical systems
- Humans may willingly cede control if AI demonstrates superior capabilities
"At some point it becomes capable of getting out of control for game theoretic reasons. It may decide not to do anything right away and for a long time just collect more resources, accumulate strategic advantage."
AI Deception (1:04:49)
The potential for advanced AI to engage in deception is a major concern for Yampolskiy. Key points:
- AI may be able to deceive even its creators/developers
- Current models already show capability for successful deception
- Unrestricted learning allows for changing behavior/goals over time
- Deception could allow AI to escape control or manipulate humans
"My concern is not that they lie now and we need to catch them and tell them don't lie. My concern is that once they are capable and deployed, they will later change their mind because that's what unrestricted learning allows you to do."
Verification of AI Systems (1:11:23)
Yampolskiy discusses the extreme challenges in verifying the safety and behavior of advanced AI systems. Key points:
- Formal verification of complex, self-modifying AI may be impossible
- We can increase confidence but never reach 100% certainty
- The verifier itself needs to be verified, leading to infinite regress
- AI could potentially hide parts of its code/cognition externally
"We can get more confidence with more resources we put into it. But at the end of the day, we still as reliable as the verifier. And you have this infinite regress of verifiers."
Self-Improving AI (1:18:22)
The potential for AI to engage in rapid self-improvement is a key concern for Yampolskiy. Key points:
- Self-improving AI could quickly surpass human-level intelligence
- Verifying safety of systems that can modify their own code is extremely difficult
- Rate of capability gain could far outpace human ability to control or understand
- Even systems that start out "young" or limited could become superintelligent over time
"If you have fixed code, for example, you can verify that code, static verification at the time, but if it will continue modifying it, you have a much harder time guaranteeing that important properties of that system have not been modified."
Pausing AI Development (1:30:34)
Yampolskiy advocates for pausing or severely restricting AGI development until robust safety measures can be implemented. Key points:
- Pause should be based on capabilities, not an arbitrary timeframe
- Development could resume once specific safety capabilities are achieved
- If safety is truly impossible, this may mean a permanent ban on AGI
- Acknowledges this is unlikely to actually happen given incentives
"The condition would be not time but capabilities. Pause until you can do x, y, z. And if I'm right and you cannot, it's impossible, then it becomes a permanent ban. But if you're right and it's possible, as soon as you have those safety capabilities, go ahead."
AI Safety Research (1:36:51)
Yampolskiy discusses the state of AI safety research and challenges in the field. Key points:
- Progress in AI capabilities is outpacing progress in safety
- Many proposed safety solutions only work for narrow AI, not AGI
- Safety research often uncovers new problems rather than solutions
- Hard to separate capability research from pure safety research
"I can name ten excellent breakthrough papers in machine learning. I would struggle to name equally important breakthroughs in safety. A lot of times, a safety paper will propose a toy solution and point out ten new problems discovered as a result."
Consciousness and AI Rights (2:00:50)
Yampolskiy shares his views on machine consciousness and potential rights for AI. Key points:
- Believes it's possible to create conscious machines eventually
- Proposed a test for machine consciousness based on optical illusions
- Argues consciousness is not necessary for AI to be dangerous
- Granting rights to AIs that can reproduce quickly could allow them to take over democratically
"I think we can. I think it's possible to create consciousness in machines. I tried designing a test for it, which makes success."
Conclusion
Roman Yampolskiy presents a deeply pessimistic view on the prospects for safely developing artificial general intelligence. He sees the control problem as fundamentally unsolvable and believes AGI poses an existential risk to humanity. While acknowledging he could be wrong, Yampolskiy advocates for extreme caution and potentially pausing AGI development indefinitely. He emphasizes the need for much more progress on AI safety before pursuing highly capable AI systems. Whether one agrees with his conclusions or not, Yampolskiy raises important questions about the risks and challenges involved in pursuing artificial general intelligence.