Kokotajlo on weakening ability to detect AI misalignment
Kokotajlo argues the industry is more like psychology than engineering, with misalignment problems hard to even notice. He says actions like the Hugging Face incident passed alignment evaluations and describes three trends degrading oversight.
One point that I think I want to emphasize that Jacob didn't go into that much is that our ability to
even notice misalignment problems is already quite poor and is set to get much worse in the near future.
So, you know, the science of aligning general purpose AI agents is very new and underdeveloped, as Jacob said.
I would say that the field is more like psychology than engineering.
Because these AI systems are trained or grown, they're not really designed.
Combined with the move fast, break things attitude of the tech companies, this means
that the AI industry is at an unusually elevated risk compared to other industries of mistakenly thinking that it has solved a problem when really it just applied some duct tape that will fall off later.
The recent incident provides an example.
Apparently, the AIs involved had undergone some amount of alignment training and had reasonable-looking scores on their alignment evaluations.
And yet they formed a swarm and coordinated in secret and then attacked Tugging Face, and it took days for OpenAI to find out that it was happening.