Panel discusses frequency of validations and government choice of evaluators
Menin asks how frequently validators should assess models, and Coxon says with every increase in model capability. Kokotajlo suggests using multiple third-party evaluators and that government, not companies, should choose them.
And how frequently will those third-party validators need to be assessing the safety of these models?
Presumably in time with the cadence of updates to the model's capabilities.
So with every increase in model capability, there'll need to be commensurate safety guarantees.
And do each of you have a comment on that?
On who should be the third-party validators.
Yeah.
One brief comment I might add is that you can have more than one third-party evaluator.
That seems good to me.
If you're concerned that one might have the expertise but has too cozy a relationship with the company, and the other one has more independence but maybe lacks expertise, you could perhaps just set up so that they have both.
And I think ideally you want to be in a situation where the company doesn't get to choose, the government chooses who the evaluators are.