Researchers have uncovered significant flaws in numerous tests used to assess the safety and effectiveness of new artificial intelligence models. According to The Guardian,
experts from the UK government's AI Security Institute and scholars from Stanford, Berkeley, and Oxford universities examined over 440 tests evaluating AI safety systems.
They found considerable shortcomings that, they argue, «undermine the credibility of the results obtained». It was noted that nearly all the tests reviewed «exhibit weaknesses in at least one area», and the outcomes may be «irrelevant or even misleading».
Many of these tests are utilized to evaluate cutting-edge AI models produced by leading tech firms, stated Andrew Bean, a researcher from the Oxford Internet Institute and the principal author of the study.
In the absence of national AI regulations in the UK and the US, these tests are employed to verify whether new models are safe, align with public interests, and achieve claimed capabilities in reasoning, mathematics, and coding.
«Tests form the foundation of almost all claims regarding advancements in artificial intelligence. However, without uniform definitions and reliable measurement methods, it is challenging to ascertain whether the models are genuinely improving or if it is merely an illusion,» Bean emphasized.
The research examined publicly available tests, but leading AI companies also possess their internal tests, which were not included in the analysis.
Bean noted that «a shocking finding was that only a small minority (16%) of tests employed uncertainty estimates or statistical methods to demonstrate how likely it is that the criteria would be accurate. In other instances, when criteria were established to evaluate AI characteristics, including its «harmlessness», the definitions were contentious or vague, reducing the utility of the test.
The study concludes that there is an «urgent need for common standards and best practices» in the field of AI.