Turner on grown AI, misalignment, and superintelligence risk
Turner says humanity grows AI rather than builds it, nobody knows how to reliably instill priorities, and severe misalignment is always possible. He describes runaway progress and recursive self-improvement potentially yielding intelligence beyond comprehension.
This goes beyond one contract.
As Jacob said, humanity doesn't build and understand AI systems the way we build and understand bridges.
Rather, we grow them.
Nobody knows how to reliably instill a designer's priorities into a new model.
Severe misalignment is always possible.
AI companies are racing to make their AIs as smart as possible, increasingly trusting their AIs with the process of improving the next crop of AIs, and it's working.
Today's rate of AI progress is staggeringly fast, which means even faster progress tomorrow, driven by tomorrow's even smarter AIs.
This progress may soon enter a feedback loop called recursive self-improvement.
Recursive self-improvement could quickly yield AIs that are intelligent beyond our comprehension.
Of course, smarter AI means more potential benefits, but more risk when things go wrong.