Coxon warns of racing dynamics and recursive self-improvement
Coxon says companies are racing each other to superintelligence, and describes recursive self-improvement where each AI generation helps build the next. He notes OpenAI had milestones for automated AI research around 2027-2028 and is on track or beating them.
These companies are also all racing each other.
Each one is worried that a competitor will get to superintelligence first.
Each believes it would do a better or safer job than whoever might beat it.
So they can't let the others win first.
Meanwhile, the technology is improving fast.
It will probably happen much faster now because already the AIs are now helping to build their successors.
This is called recursive self-improvement.
Each generation of AI helps build the next smarter generation, which then builds the one after that even faster and better.
Automating AI research is now the primary goal of these companies.
It's what they're really gunning for.
At my last job, I was in some sense working to automate myself.
When I was at OpenAI, we had milestones from automated AI researcher around 2027, 2028, and we are on track or beating these expectations.
That's a matter of months from now, not decades or even five years.