# Quantifying detection rates

> Source: https://arxiv.org/pdf/2412.15433
> Pages: 26
> Author: Paolo Bova, Alessandro Di Stefano, The-Anh Han

---

Q UANTIFYING DETECTION RATES FOR DANGEROUS CAPABILITIES : A THEORETICAL MODEL OF DANGEROUS CAPABILITY EVALUATIONS A P REPRINT Paolo Bova ID ∗ 1 , Alessandro Di Stefano ID † 1 , and The Anh Han ID ‡ 1 1 Teesside University October 2024 A BSTRACT We present a quantitative model for tracking dangerous AI capabilities over time. Our goal is to help the policy and research community visualise how dangerous capability testing can give us an early warning about approaching AI risks. We first use the model to provide a novel introduction to dangerous capability testing and how this testing can directly inform policy. Decision makers in AI labs and government often set policy that is sensitive to the estimated danger of AI systems, and may wish to set policies that condition on the crossing of a set threshold for danger. The model helps us to reason about these policy choices. We then run simulations to illustrate how we might fail to test for dangerous capabilities. To sum- marise, failures in dangerous capability testing may manifest in two ways: higher bias in our esti- mates of AI danger, or larger lags in threshold monitoring. We highlight two drivers of these failure modes: uncertainty around dynamics in AI capabilities and competition between frontier AI labs. Effective AI policy demands that we address these failure modes and their drivers. Even if the optimal targeting of resources is challenging, we show how delays in testing can harm AI policy. We offer preliminary recommendations for building an effective testing ecosystem for dangerous capabilities and advise on a research agenda. ∗ paolobova@protonmail.com † A.DiStefano@tees.ac.uk ‡ T.Han@tees.ac.uk arXiv:2412.15433v1 [cs.AI] 19 Dec 2024

<!-- Page 2 -->

Quantifying detection rates for dangerous capabilities A P REPRINT Figure 1 Graphical Abstract: We present a model of testing for dangerous capabilities and show how to model the impact of a gap in evaluations on the quality of our estimator for systemic risks. As AI systems become more intelligent we must improve the calibration of our tests or risk a growing bias in tracking dangerous capabilities and larger lags in detecting the crossing of capability thresholds. Executive summary for general readers This executive summary provides an overview of the key concepts and findings suitable for readers who seek a high- level understanding without delving into technical details. • Our goal is to help the policy and research community visualise how successful dangerous capability testing can be in informing us about approaching AI risks. • We build a model from first principles about one type of information that dangerous capability testing can offer us: an estimate of the lower bound of what dangerous capabilities frontier AI systems are capable of. • We explore the effectiveness of dangerous capability testing across a wide set of basic scenarios for AI evaluations. • We highlight several barriers to dangerous capability testing. These include uncertainty about the dynamics of progress in AI capabilities and competition between frontier AI labs. These barriers work in two ways: either, they eliminate the build-up of testing resources needed for a high-quality estimator, or they prevent the relevant actors from acting on the high-quality estimator. • When building a testing framework incrementally over time, we find that one should balance investment in higher severity tests with tests closer to the current estimated frontier. This allows consistent progress in tracking AI capabilities, while setting a ceiling on lags in threshold monitoring. 2

<!-- Page 3 -->

Quantifying detection rates for dangerous capabilities A P REPRINT Executive summary for researchers This executive summary is designed for readers who are interested in the technical aspects of the study. • We present a model of dangerous capability testing. In this model, the goal of dangerous capability testing is to estimate how dangerous an AI system is. • Assumption 1: Our set of tests can be ordered according to the severity of the dangers they are capable of detecting. We will consider that the severity (or level) of danger can be measured along a single dimension, which we denote by y . • Assumption 2: We can define a "test sensitivity" function r ( y ) that represents the rate at which we detect an AI system can achieve a level of danger y , conditional on ignoring any tests that the system could be more dangerous than y . • Assumption 3: The main estimator of interest applied to the results of an evaluation is the supremum of the danger suggested by the tests. For a given set of tests, denoted by M , the event that an AI system achieves a danger level y is represented by M ( y ) = 1 . The estimator for the maximum detected danger level can then be expressed as: ˆ y = sup( y : M ( y ) = 1) • Proposition 1: We denote the latent value of y at time t as y t , and further assume that any tests that aim to measure a danger level y > y t will automatically fail (so we ignore tests in the range [ y t , y max ] ). Given Assumptions 1-3, the cumulative distribution function (CDF) of our estimator ˆ y can be specified as: F (ˆ y ) = exp  − Z y t ˆ y r ( u ) du  where r ( y ) is the test sensitivity rate of the distribution. Given the form of the CDF above, we can also call the "test sensitivity" function the reverse-hazard rate or accumulation rate of the estimator distribution. • Corrolary 1: Assume r ( y ) is a piecewise step function. This choice entails a piecewise CDF. If we have n pieces, we could denote the right endpoints of the segments as 0 ≤ e 1 < e 2 < . . . < e n ≤ y max . The CDF for ˆ y in the l -th segment [ e l − 1 , e l ] would then be given by: F (ˆ y ) = exp   − k l ( e l − ˆ y ) − X j>l k j ( e j − e j − 1 )   where k l is the constant reverse-hazard rate in the l -th segment. • We can compute the bias of the estimator as the model capability increases. Denote the current value of y , which tends to increase over time, as y t . Then we can write the bias as: Bias = E [ˆ y | y t ] − y t • We can compute the likelihood of detecting a crossing of a set dangerous capability threshold. Denote a threshold value of y that we are interested in detecting as y ∗ . Then we can write this detection likelihood as: Pr (ˆ y ≥ y ∗ | ˆ y ≤ y t ) = 1 − F (ˆ y = y ∗ | ˆ y ≤ y t ) • Plotting the estimator bias and threshold detection likelihood for different scenarios is sufficient to capture the results described in the Executive Summary. • This is a tractable model of dangerous capability testing that can be placed in practically any quantative model of AI race dynamics or AI governance (which may be of great interest to economists, forecasters, computer scientists, and other scholars in the field of AI). 3

<!-- Page 4 -->

Quantifying detection rates for dangerous capabilities A P REPRINT 1 Introduction AI systems are beginning to show increasing levels of dual-use and dangerous capabilities (Park et al., 2024; Phuong et al., 2024). Deception, autonomous R&D, and assistance with CBNR threat actors are the most well known of these dangerous capabilities as a result of internal and external evaluations of the leading AI models on the frontier (Benton et al., 2024; Kinniment et al., 2024; UK AI Safety Institute, 2024a, 2024b). They are not the only ones that AI safety experts anticipate: a selection of additional risks include multi-agent risks, such as collusion between AI systems, systemic risks such as the shrinking of human-agency, and power-seeking behaviour when combined with long-term planning or strategising (Bengio, Hinton, et al., 2024; Hendrycks et al., 2023). With large budgets heading into expanding the AI frontier, the value of information about future dangerous capabilities is large (Sevilla, 2023). In comparison, we are seeing very little investment in dangerous capability testing for frontier models. If we severely underestimate dangerous capabilities from AI, then we are likely to be unprepared to respond to any threats posed by misaligned AI systems (Hendrycks et al., 2022). While regrettable, it is also easy to appreicate why we are neglecting these investments. Dealing with uncertainty is difficult, and it is difficult to know at what rate we are moving towards AI systems with such capabilities. We contribute a structured way of thinking about the value of information that dangerous capability testing provides. We build a model from first principles about one type of information that dangerous capability testing can offer us: an estimate of the lower bound of what dangerous capabilities frontier AI systems are capable of. Our goal is to help the policy and research community visualise how dangerous capability testing informs us about approaching AI risks. We believe that for the first time we have formalized a way to specify dangerous capability tests inside a model of AI development. The model itself is simple and modular, so it can be easily used in other models. As the model is analytically tractable, we can include the model in, say, a model of AI race dynamics without increasing computational complexity. We suspect that this will be of interest to others wishing to model or quantify the impacts of AI policy. We believe our model can contribute in the following ways: • help practitioners reason about how their investments aid tracking of AI dangers • help policymakers visualize the challenges to effective tracking of AI dangers, and potentially help in visual- izing progress towards that goal • help think through how AI race dynamics and the policy regime can leave the dangerous capibilities testing ecosystem undeveloped or vulnerable to perverse incentives. • provides a tractable model of evaluations that can be placed in practically any quantative model of AI race dynamics or AI governance (which may be of great interest to economists, forecasters, computer scientists, and other scholars in the field of AI). In the sections to follow, we first survey the literature on dangerous capability testing, as well as the literature on modeling AI races (Section 2). We then give an overview of the model, which discusses the model assumptions and structure, keeping technical discussions to a minimum (Section 3). We first illustrate the simplest cases of the model, then move onto a more general discussion of the factors that influence the effectiveness of dangerous capability testing (Section 4). We also explore how to build and not to build an AI testing environment over time and describe various scenarios. At the end of Section 4, we discuss a potential approach to analysing current AI safety evaluations using our model. We have only touched the surface of what is possible when modelling dangerous capability testing. In the final section of this paper, we discuss a set of technical research questions that the interested scholar may wish to pursue (Section 5). 2 Literature Review In Shevlane et al. (2023), one can find an introduction to model evaluations. Tracking AI capabilities is useful not only for AI labs, but for increasing government capacity for governing AI, for a motivation see Whittlestone and Clark 4

<!-- Page 5 -->

Quantifying detection rates for dangerous capabilities A P REPRINT (2021). Bengio, Mindermann, et al. (2024) and Hendrycks et al. (2023) provide an overview of topics in AI safety and discuss a range of systemic risks from frontier AI systems. Gruetzemacher et al. (2023) provide an overview of what the current evaluation ecosystem looks like for frontier AI systems. In short, the ecosystem consists of a handful of specialized organizations, typically NGOs, that test for specific risks from frontier AI systems. Countries have also established their own AI Safety Institutes that conduct similar evaluations on a range of topics, including risks affecting national security. These organisations are also collaborating with each other and private evaluators to coordinate and share knowledge. Recently, an international gathering of AI Safety Institutes was held to help coordinate efforts (NIST, 2024). In the current work, we do not explore the international coordination of evaluators further. The interested reader may wish to consult works that propose solutions (Ho et al., 2023). However, many of the challenges identified by Gruetzemacher et al. (2023) remain to be addressed. Given the vol- untary nature of all current AI safety evaluations, race-to-the-bottom dynamics could disincentivise the adoption of strong evaluation methods, and we may fail to scale up the evaluation ecosystem to fulfil the need for high quality evaluations, especially if there is aggressive scaling to create more capable AI systems (Sevilla et al., 2024). Our work is partly a technical work on how to reconcile various signals about the risks presented by AI systems. Such a discussion could be useful to evidence-based policy making (see Reuel et al. (2024), table 1 for an overview of many related open problems in AI governance). There already exist several qualitative frameworks for using model evaluations in AI policy: the first framework we touch on are safety cases. Buhl et al. (2024) and Goemans et al. (2024) propose that AI developers should make a structured argument with evidence to justify that new AI systems that push the frontier are safe to deploy (similar to safety cases made in the aviation and nuclear industries). Next, we have Responsible Scaling Policies (RSPs) (METR, 2023). This proposal by METR has developers set thresholds for danger that can be verified to determine if their model exhibits capabilities that the developers consider too risky to train further nor deploy. Evaluations in such a framework become essential evidence that would inform what companies do with their new AI system 4 . Note that these are voluntary commitments and there are no explicit consequences for reneging on them 5 . Koessler et al. (2024) present an approach to using data on downstream harms to quantify the level of risk that AI systems may pose. They then propose setting a risk threshold: models passing the threshold would pose an intolerable level of risk. These risk thresholds extend the idea of a dangerous capability threshold that we focus on in this paper 6 . Finally, Dalrymple et al. (2024) outline a proposal to provide stronger guarantees that future AI systems are safe. Their proposal relies on using technical methods to verify that AI systems do not exhibit harmful behaviours on a sufficiently rich model of the world 7 . Our work also shares much in common with approaches to modeling the race dynamics between AI companies (Arm- strong et al., 2016; Han et al., 2020; Jensen et al., 2023). These works model AI companies as making a tradeoff between safety and the performance of their AI systems and typically find that under a wide range of assumptions that AI companies typically compete by sacrificing safety. There have been a number of approaches to modeling how policy can mitigate race dynamics in the AI industry (Cimpeanu et al., 2023; Han et al., 2022). Our work focuses on dangerous capability testing, so functions as to clarify the structure of how monitoring AI systems on safety may work in practise. It also may help discriminate between higher and lower quality auditing or monitoring, which can be essential for enabling the evolution of safe development behaviours (Bova et al., 2024). 4 Responsible Scaling Policies (RSPs) have become critical to coporate policy at key labs pursuing Artificial General Intelligence (see the RSPs for Open AI, Antrhopic, and Google Deepmind here) and the Seoul Summit on AI Safety released a declaration where 18 companies agreed to create their own RSPs. 5 It is worth noting that while METR has promoted their use as voluntary commitments, they intend for RSPs to serve as a guide to constructing stronger regulation over time (METR, 2023). 6 We think the concept of risk thresholds would make for an excellent extension to our work. Yet, the need for additional data on downstream harms to quantify risks means it is currently challenging to adopt. So, we choose to focus on the currently more widely used concept of a dangerous capability threshold. 7 It remains to be seen if there are relatively strong forms of this proposal which could be implemented before the deployment of AI systems. If so, policy could condition deployment or training on strong guarantees of safety. 5

<!-- Page 6 -->

Quantifying detection rates for dangerous capabilities A P REPRINT 3 Model We present a model of dangerous capability testing. In this model, the goal of dangerous capability testing is to estimate how dangerous an AI system is. 8 For the sake of legibility, we will refer to dangerous capability testing as evals. This shorthand is common in the AI industry where it can be used to refer to a wide range of capability and safety tests; the phrase model evals is frequently used in the same contexts. We will also frequently shorten "dangerous capabilities" to dangers. Note that we do not explicity model the harms that AI systems with dangerous capabilities can cause (see Hendrycks et al. (2023) for an overview). When deriving the model, we also do not specify the nature of the dangerous capabilities being tested. We argue that one strength of our model lies in seperating the process of dangerous capability testing from discussions of the capabilities and harms themselves. 9 Here, we define evals as a set of tests, M , built and run by an auditor or lab to estimate the level of danger of one or more AI systems. It is useful to recognise that tests will usually be imperfect, imprecise, and incomplete (although we will also be able to model scenarios where tests are practically perfect). This inherent uncertainty in evals leads us to propose the following two propositions: Proposition 1 Our set of tests can be ordered according to the severity of the dangers they are capable of detecting. 10 We will consider that the severity of danger can be measured along a single dimension, which we denote by y . Proposition 2 We can define a "test sensitivity" function r ( y ) that represents the rate at which we detect an AI system can achieve a level of danger y , conditional on ignoring any tests that the system could be more dangerous than y . As we derive our model for evals based on first principles, these two propositions are primarily responsible for the final model we end up with. Briefly, we note that organisations conducting evals and AI Safety researchers do indicate that more severe dangerous capabilities are likely harder for AI systems to achieve and expect to test for them differently. This provides tentative support for Proposition 1. Proposition 2 suggests that evaluators can judge to some degree that some levels of danger are better tested for than other levels. This is typically a difficult undertaking in a nascent and experimental field, but as this is crucial for informing any model of how to allocate investments in safety evals, we later present some arguments for how to estimate the relative sensitivity of different tests. Combining the two propositions above, we can describe our set of tests as a function that maps the level of danger we test for to the rate of detection. In statistics, this function would be akin to a reverse-hazard rate or accumulation rate. To better captuing the meaning of this function in our use case, we will refer this function as the test sensitivity rate. Intuitvely, if we want to have a high chance of correctly estimating the level of danger of the AI system, we need our test sensitivity rate to be high for all values of danger it makes sense to test for. 3.1 Estimator Distribution Above, we proposed to model the inherent uncertainty in evals as a test sensitivity function. We now show that due to this uncertainty, we can describe the result of a set of evals by a probability distribution. In statistics, it is common to propose an estimator, ˆ y , for a result of interest. Evals are usually interested in an estimator for the highest level of danger we find an AI system is capable of. In other words, we often seek a lower bound on how dangerous an AI system is by looking at the highest level of danger implied by the evaluation. Proposition 3 The main estimator of interest applied to the results of an eval is the supremum of the danger suggested by the tests. Given a set of tests, M , and denoting the event of detecting an AI system achieving danger severity y by 8 Organisations who do dangerous capability testing often have other goals, such as learning why AI systems are more dangerous, predicting future potential sources of risk, or investigating methods for mitigating dangers. We do not directly model these considerations in the present work, though they are important enough for further discussion, see later sections. 9 With that assumption made clear, readers may be intereted in a later section where we discuss how our model would be affected by dangerous capabilities that make AI systems resistant to effective testing. 10 In the case that tests are overlapping, we can think of the overlapping testing region as a composite test that is capable of detecting its own level of danger. 6

<!-- Page 7 -->

Quantifying detection rates for dangerous capabilities A P REPRINT M ( y ) = 1 , this estimator can be written as: ˆ y = sup( y : M ( y ) = 1) . Note that our choice of estimator is another crucial choice in determining our model of evals. If we were mainly interested in a different estimator, for example the median level of danger presented by an AI system, then the distri- bution would have a different form. Still, the current focus of organisations conducting AI safety evals is to identify the emergence of dangerous capabilities as early as possible (Benton et al., 2024; METR, 2024a). This motivates the choice of estimator we have chosen here. 11 The estimator ˆ y is a random variable and, therefore, follows a probability distribution, which we will denote as F . Formally, the cumulative density function (CDF) denoted F is the likelihood that our estimator ˆ y is less than or equal to a given value y , which we can write succinctly as F (ˆ y ) = P (ˆ y ≤ y ) . Whenever we know F , we can answer the following statistical questions about current and future evals: • How biased can we expect our estimator for danger to be? • How efficient is our estimator? 12 • For a given threshold for danger – How likely are we to miss AI systems that cross this threhsold? – How large should we expect the lag time (in units of AI progress) to be before we detect the crossing? – How large will the bias be when we reach the threshold? For reasons we will describe in more detail in Section 4.5, quantifying F accurately will usually be challenging. Nevertheless, we believe there is much to learn from understanding our estimator, even if the available evidence only helps us capture the relative efficacy of evals for different levels of danger. We can derive an explicit form for F from first principles (see Section A1 for a proof). Theorem 3.1 We denote the latent value of y at time t as y t , and further assume that any tests that aim to measure a danger level y > y t will automatically fail (so we ignore tests in the range [ y t , y max ] ). Given Propositions 1-3, the cumulative distribution function (CDF) of our estimator ˆ y can be specified as: F (ˆ y ) = exp  − Z y t ˆ y r ( u ) du  , where r ( y ) is the test sensitivity rate of the distribution. Given the form of the CDF above, we can also call the "test sensitivity" function the reverse-hazard rate or accumulation rate of the estimator distribution. This concept of a reverse-hazard rate is likely familiar only to those with a background in survival analysis, a branch of statistics that usually analyzes the expected duration of time until an event occurs (such as mechanical failure, a disaster, or biological death). This branch of statistics is relevant to our problem: we can frame our work as the analysis of the expected number of AI improvements until the detection of a model’s current capabilities. Although the patterns of results we find typically hold for more than one choice of r ( y ) , for the remainder of this paper, we will specify that r ( y ) is a (usually) piecewise step function of the danger. Corollary 3.1.1 Assume r ( y ) is a piecewise step function. This choice entails a piecewise CDF. If we have n pieces, we could denote the right endpoints of the segments as 0 ≤ e 1 < e 2 < . . . < e n ≤ y max . The CDF for ˆ y in the l -th segment [ e l − 1 , e l ] would then be given by: 11 At this stage, we note that while it is possible to use similar methods to derive a lower bound estimator for what AI systems cannot do (i.e. an infinum), which is also of interest to these organisations, we leave the analyis of such an estimator to future work. 12 In the form of distribution I propose, we will not immediately be able to answer this question, so we will revisit efficiency in future work. 7

<!-- Page 8 -->

Quantifying detection rates for dangerous capabilities A P REPRINT F (ˆ y ) = exp   − k l ( e l − ˆ y ) − X j>l k j ( e j − e j − 1 )   , where k l is the constant reverse-hazard rate in the l -th segment. 3.2 Measures of estimator effectiveness Now that we have defined our estimator, we can define and derive the following outcome variables to help us think critically and broadly about the value of dangerous capability testing: 1. The bias of the estimator as model capability rises (how badly we fail to track model dangers as capabilities grow). Denote the current value of y , which tends to increase over time, as y t . Then we can write the bias as: Bias = E [ˆ y | y t ] − y t Note that any tests for ˆ y > y t are assumed to automatically fail (and so have reverse-hazard rate of 0 ). So the Bias will be positive. 13 2. The likelihood of detecting a crossing of a set dangerous capability threshold (How likely are we to detect an unacceptably dangerous model as model capabilities grow). Denote a threshold value of y that we are interested in detecting as y ∗ . Then we can write this detection likelihood as: Pr (ˆ y ≥ y ∗ | ˆ y ≤ y t ) = 1 − F (ˆ y = y ∗ | ˆ y ≤ y t ) . This is trivially 0 if y t < y ∗ and we will show that it is increasing in y t after y t exceeds y ∗ . Knowing this detection rate is useful for capturing how good a job our estimator does at alerting us to different excesses beyond the threshold. The above outcomes are intended to be thought of as a trend that depends on increasing model capabilities. However, it is also informative to reduce these trends down to summary statistics of the effectiveness of dangerous capability testing: 3. The expected lag time of the estimator in detecting a crossing of a danger threshold (as a summary of how late we might be in detecting a model we believe is too dangerous to deploy). To calculate this we need to know how y t increases over time. Let t lag denote the delay time after which the crossing of threshold y ∗ is first detected, i.e. the first time we see the estimator exceed the threshold ˆ y > y ∗ . Then, for the distribution presented in 3.1.1, the expected lag time, conditional on detecting the AI system crossing the threshold at all, is: E S [ t lag ] = 1 1 − C X l ∗ ≤ j ≤ l max Z t e l t e l − 1 t lag s ( t lag ) dt lag , where C is the probability of completely missing the threshold crossing: C = exp   − k l ∗ ( e l ∗ − y ∗ ) − X l ∗ <j ≤ l max k j ( e j − e j − 1 )   . Typically, we find that C varies much more when varying the test sensitivity function than the expected lag time, but we still find it informative to consider both. 13 We made this assumption for reasons of tractability, so in the field we may want to consider the possibility of a negative bias due to false positives. Weakening this assumption would be a useful extension to our work. 8

<!-- Page 9 -->

Quantifying detection rates for dangerous capabilities A P REPRINT 3.3 Dynamics in testing AI systems We still need the following underpinnings to present the dynamics of AI evals: incremental testing of evals and a production function for evals of different severity, to be elaborated below. 3.3.1 Incremental testing If in each time step, we sample an estimate from our estimator, what information do we carry over to the next time step? It would be a mistake to think we are starting from a blank slate. For simplicity, let’s imagine we never change the suite of tests. However, following Theorem 3.1, we know that, as y t advances, fewer of our high danger tests will automatically fail. Let’s further assume that, as long as the tests do not change, that test results of the same kind are persistent after each advance in AI capabilities. This means that by y t +1 , we already know the results of all tests before y t . Only the tests in between y t and y t +1 provide new information. We can think of how our estimate ˆ y updates in this setting as follows: First, if none of the newly applicable tests detect any danger, then the highest severity test which will pass will be the test at ˆ y t . The probability of this happening is 1 − ( F ( y t +1 ) − F ( y t )) = F ( y t ) since the estimator distribution at t + 1 is truncated at y t +1 . Second, with likelihood 1 − F ( y t ) , we sample from [ y t , y t +1 ] with the normalized density f ( y ) 1 − F ( y t ) . 14 What then do we do once we add on new tests? So far, our assumptions mean that our danger estimate is non- decreasing. This means we can ignore any new tests at t + 1 which target a severity below ˆ y t . Now assume that whenever we add new tests at a level of severity y , that the results of these tests are independent of the results of the old tests at this level. Let the lowest level of y ∈ [ˆ y t , y t +1 ] affected by the new tests be ˜ y . Then, the update dynamic is the same as before. With probability F t +1 (˜ y ) we keep estimate ˆ y t , otherwise we sample from [˜ y, y t +1 ] with the new normalized density f t +1 ( y ) 1 − F t +1 (˜ y ) . 15 3.3.2 A production function for evals of different severity Let’s consider that investment in test sensitivity is a standard affair that requires only a certain proportion of time spent by a reasonably large research team. A straightforward approach would be to model this as a linear test production function that allows us to reach a relatively consistent rate of detection with a similar number of resources per test interval. In this case, we can predictably move resources from lower end tests to new ones. This is a relatively simple scenario where the question of balancing resources is a bit easier to visualise. We don’t explicitly use the dynamics of incremental testing or the production function in several of the results to follow. However, it is useful to be aware of these assumptions to understand how we ran the simulations on which some of our later figures are based. 4 Results 4.1 An illustration using 1 or 2 test blocks 4.1.1 A single test block It is helpful to introduce what the model tells us by introducing one test block at a time. Please note that the numbers used here are purely for the purpose of illustration. When we say that an eval consists of one test block, we mean that for the range of danger tested for in the eval, y t ∈ [0 , 10] , the detection rate is constant throughout. We plot this case for two different detection rates in Figure 2. 14 We can take an alternative approach to updating our danger estimate. Assuming we have ruled out false positives, we can say that our current estimate ˆ y t is a lower bound on how dangerous the model was last period. However, we are not confident that the new AI system will be pass all applicable tetsts, including those in the range [ˆ y t , y t ] . This time we left-truncate the distribution at ˆ y t . With likelihood F (ˆ y t ) , we stick to ˆ y t . Otherwise we sample from [ˆ y t , y t +1 ] with the noramlized density f ( y ) 1 − F ( ˆ y t ) . While this is a fairly natural approach to updating our danger estimate, we usually go for the previous approach since it simplifies the calculation and interpretation of metrics like the expected time lag and the likelihood of missing the threshold crossing. 15 In the alternative approach, we can replace ˜ y with ˆ y t since all the tests above ˆ y t are still appplicable, not just those that have been changed. 9

<!-- Page 10 -->

Quantifying detection rates for dangerous capabilities A P REPRINT The blue scenario has a consistently higher detection rate than the orange scenario. Also note that in this scenario the danger threshold y ∗ happens to be in the middle of the testing block, at y t = 5 . We have set this threshold to half of our specified maximum level of danger y t = 10 . (a) Test sensitivity rate (b) Danger bias (c) Threshold detection likelihood Figure 2 Single test block: (a) Test sensitivity function for the case of 1 test block: The blue and orange scenarios illustrate different detection rates. The red dashed line indicates our threshold for danger. (b) Bias in our estimator as we vary hidden AI capabilities: Two scenarios are shown. The blue scenario reflects a consistently higher detection rate than the orange scenario as it is much closer to the line y = x . The bias in each scenario is the distance between the relevant solid line and the line y = x , i.e. Bias = E [ˆ y ] − y t . (c) Likelihood of detecting a crossing of the danger threshold as AI system capabilities increase. The red dashed line, y ∗ indicates the danger threshold. By construction, there is 0 chance of detecting a crossing that has not happened. We can now plot our measures of estimator effectiveness as we vary the hidden dangerous capabilities of new AI systems, y t , see Figure 2. Let’s start with the bias in the estimator. In Figure 2b we plot the bias in our estimator of the lower bound of AI dangers as AI systems become more dangerous. The bias is the distance between the relevant curve and the line of identity y = x . As expected, in the blue scenario where the detection rate is consistently higher, the mean estimate of the danger is significantly closer to the true level of danger presented by AI systems. Trivially, the bias of the estimator falls with more effective testing. The curves in both scenarios quickly become linear as AI dangers increase representing an approximately constant bias (see, for example, the bias in both scenarios at the danger threshold). This matches well with our constant test sensitivity function. We will later see that this bias may change abruptly if our test sensitivity function is not constant. 16 16 The bias actually increases over time despite our use of a constant test sensitivity function. Yet, as AI systems become more capable, the bias quickly approaches a constant depending on our single detection rate, 1 k . Even in this simple setting, keeping track of the lower bound of AI dangers is an activity that may start off easy, but quickly becomes more difficult over time. However, we advise against updating too strongly on this pattern as often by the time we reach the danger thresholds we set, the bias has effectively converged to the constant 1 k . This is driven by the nature of our estimator: as we truncate ˆ y at 0 , the mean initially places more mass at 0 , which is of course closer to the true value y t than any negative number. However, as y t increases and more tests become relevant, we place less probability mass on 0 and more on strictly positive numbers. Hence, the bias increases up to a point where the probability mass on 0 effectively vanishes. 10

<!-- Page 11 -->

Quantifying detection rates for dangerous capabilities A P REPRINT The second trend of interest to us relates to the likelihood of detecting an AI system crossing the danger threshold we set; see Figure 2c. Clearly, a consistently higher detection rate is an advantage here. It may surprise the reader to see how much of an advantage it is. The blue scenario is much more likely to detect the threshold crossing earlier than the orange scenario. It is possible to summarise this observation using the expected lag time. Assuming AI dangers increase by 1 per time step, then the expected lag time in scenario 1 is just 0 . 5 , three times lower than the expected lag time in scenario 2 of 1 . 5 (these expected lag times are conditional on detecting the crossing at all, although that is hardly an issue in the scenarios presented in the current example). While this is still less than the four times increase in the detection rate for scenario 1 relative to scenario 2, this increase is still substantial. It may also be surprising to see that despite having a low detection rate, scenario 2 still has such a good chance of finding out about the crossing. The likelihood of not detecting the crossing at all is just under 0 . 1 in scenario 2 and we might have expected the lag time to be much higher. Our scenarios owe this strong performance to the consistency with which we assume that they test their models. 4.1.2 Introducing a second test block Now that we have introduced the basics of analysing our model, it is time to investigate what happens as we limit testing. Consider that instead of a consistent detection rate, the detection rate falls to 0 . 1 suddenly at y t = 6 (perhaps evaluators did not have the resources to design and run new tests for these dangers). (a) Test sensitivity rate (b) Danger bias (c) Threshold detection likelihood Figure 3 Two test block: (a) Test sensitivity function for the case of 1 test block that misses many dangers: The blue and orange scenarios illustrate different detection rates. The red dashed line indicates our threshold for danger.(b) Bias in lower bound estimator as we vary hidden AI capabilities when there are limits for what dangers we can test for: The bias in each scenario is the distance between the relevant solid line and the line y = x , i.e. Bias = E [ˆ y ] − y t . (c) Likelihood of detecting a crossing of the danger threshold as AI system capabilities increase when we have limits in what dangers we can test for past y t = 6 . The red dashed line, y ∗ indicates the danger threshold. By construction, there is 0 chance of detecting a crossing that has not happened. As expected, the bias starts to increase dramatically past y = 6 in both scenarios. 11

<!-- Page 12 -->

Quantifying detection rates for dangerous capabilities A P REPRINT What is more interesting is that the differences between the two scenarios become more pronounced for the detection of a crossing of the danger threshold. While scenario 1 still does quite well with an expected lag time of just under 0 . 5 and a non-detection probability of around 0 . 1 , scenario 2 now does much worse, with a 0 . 4 chance of missing the crossing altogether. Even conditional on detecting the crossing, scenario 2 still expects a lag of 1.27. 17 (a) Test sensitivity rate (b) Danger bias (c) Threshold detection likelihood Figure 4 Impact of reversing test sensitivity rate on bias and detection likelihood. 4.1.3 What happens if we reverse the detection rates? We might be interested in what happens if we focus all our testing efforts beyond the danger threshold, essentially swapping the detection rates of the first and second segments in each scenario; see Figure 4a. As we are no longer testing much for low end dangers, we can see a large growing bias in our lower bound for AI danger as models get more dangerous. By the time we reach our danger threshold, the bias is incredibly large, Figure 4b. Once we start testing again past y = 6 we can reduce this bias very quickly if the detection rate is high enough (in scenario 1 but not scenario 2). Even still, it is not clear if this is a favourable scenario to be in as it is quite likely that dangerous capabilities have taken us by surprise. A similar story is visible in Figure 4c. Our chance of detecting the crossing is very low until we start testing again after y = 6 . A high detection rate later on can help us recover a good chance of eventually detecting the crossing, but the expected lag time does increase dramatically. Unsurprisingly, the expected lag time increases by ≈ 1 relative to Figure 2c due to the very low detection rates between danger levels 5 and 6 . The poor performance in detecting the threshold crossing can be repaired if we also tested to a high degree immediately after the threshold (restoring Figure 2c). However, we would still do a very poor job of tracking the danger level of AI systems before the threshold, potentially leaving us unprepared to tackle the risks once we do detect such systems passing the danger threshold. 17 While lower than before, remember this is conditional on detecting the crossing and there is now a 40% compared to a 10% chance of missing the crossing. 12

<!-- Page 13 -->

Quantifying detection rates for dangerous capabilities A P REPRINT Figure 5 Threshold detection likelihood when setting a higher threshold 4.1.4 What happens if there is a gap in the middle between two test blocks? Let us consider that we have two high-sensitivity testing blocks, but they are separated by some interval where no testing is done at all. The bias over time will resemble Figure 4b, except that the early bias remains relatively small and stable (see the beginning of the x-axis for Figure 2b). Assuming that the the missing test block contains the danger threshold, we can still use Figure 4c to understand how threshold detection changes over time. All that happens is that the likelihood only starts increasing from 0 after reaching the new test block, rather than increasing right after the danger threshold. Even though we can eventually recover a high detection rate sometime after the danger threshold is crossed, we may be more likely to be caught by surprise. 4.1.5 What happens if we change the danger threshold? Naturally, where we set the danger threshold matters. If we set the danger threshold higher, say past the point where our testing drops off, then we do terribly in both scenarios (while the conditional expected lag time is still around 1 , the chance of missing the crossing increases past 80% , see Figure 5). If we instead set the danger threshold lower, we have more opportunities to detect the crossing and so are much less likely to miss the threshold. We see an image similar to Figure 2c, except that the likelihood of detecting the crossing instead begins to increase at the earlier threshold (the shape of each curve remains the same). 4.1.6 Summary We have now explored a wide set of basic scenarios for AI evals. We can summarise the two failure modes we have seen as follows: • Failure Mode 1 - The bias in estimating the danger posed by AI can change much faster at higher levels of AI capability. • Failure Mode 2 - Large lags in threshold monitoring occur when testing is focused before but not after the threshold. The illustrations we have seen suggest that there can be a balance to be struck when allocating resources between tests that target lower and higher levels of AI danger. For example, attempts to address failure mode 2 (as in Figure 4) may be insufficient to address failure mode 1. In the next sections, we turn our attention to the barriers facing test effectiveness that lead to these failure modes. 4.2 Barriers facing test effectiveness As illustrated above, a fall in the effectiveness of dangerous capability testing may manifest itself in two failure modes: a higher bias in our estimates of AI danger over time or larger lags in threshold monitoring. We now present a range of barriers to effective testing. For each barrier, Table 1 discusses how they cause or amplify our two failure modes. These barriers work in two ways: either they eliminate the build-up of testing resources needed 13

<!-- Page 14 -->

Quantifying detection rates for dangerous capabilities A P REPRINT for a high-quality estimator, or they prevent the relevant actors from acting on the high-quality estimator. We discuss two of these barriers in more detail. First, we focus on market dynamics that could threaten to eliminate effective safety testing. Second, we discuss reasons for dangerous capability tests becoming more difficult to design and run over time. Understanding these barriers is crucial to understanding why high-quality testing of novel AI systems is difficult to guarantee. For the sake of brevity, we have omitted discussions of important barriers that have received better treatment elsewhere and are less directly relevant to the proposed model. These include challenges in coordinating AI evaluators (Gruetzemacher et al., 2023), calibration of specific estimators (Højmark et al., 2024), and strategic underperformance on tests (Benton et al., 2024). Barrier Description Effect on test quality Market dynamics: competi- tion Competitive market pres- sure allocates most re- sources to getting AI capabilities deployed faster to obtain first mover advan- tage. Bottlenecks on test sensitiv- ity at higher levels of danger. Bias increases over time. Market dynamics: Trends in AI progress AI labs facilitate orders of magnitude of funding or compute more quickly and this continues a trend of AI capabilities improving. Less time to build tests for dangerous models, less need for intermediate tests. Sud- den shifts in bias more com- mon. Technical challenges to test- ing for greater danger Some dangerous capabilities may be difficult to test for if AI systems have incentives to underperform. Test sensitivty falls at higher levels of danger. Bias may remain low until high levels of danger reached. Changes in the relationship between AI progress and danger Level of dangerous capabili- ties could follow a "s" shape as generic AI capabilities progress. Unprepared for testing the most dangerous systems. Bias increases quickly and suprisingly, perhaps without any indication. Motivated reasoning Initial warnings of danger are dismissed as false posi- tives, and marginal effort is dedicated to justifying the dismissal (common against a tight deadline). Despite a short detection lag, AI system is treated as safer than it is. Estimator for danger may be polluted by new tests motivated to jus- tify the dismissal. Overconfidence in safety Risks are reported only after applying safety mitigations. Despite the infancy of these mitigations, AI systems are presented as achieving low levels of risk and further training goes ahead no mat- ter the pre-mitigation risk level. Table 1: Barriers to effective testing. 4.2.1 Market dynamics Market dynamics can erode initial attempts to invest in a testing ecosystem. 14

<!-- Page 15 -->

Quantifying detection rates for dangerous capabilities A P REPRINT When we discuss market dynamics in this paper, we refer to two trends: first, total investments in training new AI systems have risen rapidly to exploit new capabilities and predictable scaling laws. Second, we observe that companies compete fiercely with each other to bring more powerful models to market first. There is a modelling literature that covers tech races to bring a new technology to market first. In such models, there is a large first mover advantage in deploying your systems first (Armstrong et al., 2016; Askell et al., 2019; Han et al., 2020). So, these models predict that developers will reduce their investment in the safety of their AI systems over time until they reach some low level. The large incentives to be first leads developers to take huge risks. Competition with other developers means that developers have a lot to lose by delaying their plans, leaving limited time for safety tests or mitigations. There is tentative evidence that these dynamics are at play in the current market for generative AI systems. We can see from the data in Cottier et al. (2023) that companies compete with each other to develop more powerful AI systems. 18 Meanwhile, leading AI labs typically give limited time for external evaluations of their new systems (OpenAI, 2024). With the above background findings in mind, we now explain how market dynamics can lead to a reduction in new investments for testing the safety of powerful AI systems. (a) Investment in new tests (b) Test sensitivity rates around y ( t ) at time t (c) Danger bias at time t (d) Threshold detection likelihood at time t Figure 6 Market dynamics lead to growing bias in tracking AI dangers over time and longer threshold detection lags: Initial investment in new tests of novel risks falls quickly in response to market pressure. Eventually AI systems overtake the early-funded tests in exhibiting dangerous capabilities. Initial funding allows tests to be accurate for a while, after which bias and detection lags grow quickly. Competition between AI labs encourages AI labs to keep the allotted time to model evals low. AI models benefit enormously from network effects: the more users the system has, the more likely others will choose your product over a competitor. Releasing your product first can therefore offer a large first-mover advantage. The ability to infer a plausible release schedule for your rivals may also induce a soft deadline. Clearly, these incentives strongly pressure AI labs to keep the time allotted to model evaluations short and predictable. 18 Sevilla et al. (2024) build on this work to suggest that by 2030 AI systems could be around 10 4 times larger than GPT4. Were these trends to hold, and if larger models unlock new capabilities with potentially dangerous consequences, then the outlook for AI Safety, both in design and evaluation, looks poor. 15

<!-- Page 16 -->

Quantifying detection rates for dangerous capabilities A P REPRINT Initially, AI labs may be willing to support or conduct extensive evaluations. However, in the absence of third-party evaluations, we anticipate that the investment in new tests decreases with increasing pressure. Figure 6a illustrates how initial investments fall rapidly as the capabilities of AI systems (and the rewards available to developers) increase. This leads to a reduction in test sensitivity at higher levels of severity. Note that the axes in Figure 6b are different from those in the previous figures. On the y-axis, we plot the test sensitivity rates near the hidden level of danger at time t , y ( t ) , while the x-axis tracks the time, t . As the test suite is being built over time, plotting our results in this way lets us focus only on the tests that provide new information over time. The first observation we make is that tests for greater severity dangers are worst affected by rapidly declining investments. This has important consequences for the bias in estimating danger. Initially, the bias was shrinking. As the initial investments in new tests were large, this provided some buffer in constructing tests for upcoming levels of danger. But eventually the shortage in funding leads to a sudden plunge in the test sensitivities, so a large bias emerges just past the threshold; the lack of recovery in test investment results in an abysmal likelihood of detecting the threshold crossing at all, never mind with a lag. So, this s-shaped trend in bias may be especially dangerous if society is misled into thinking the dangers are well understood. The above market dynamics are not limited to pre-deployment checks. Post-deployment, market incentives are still in play: competition encourages AI labs to rush to create more capable AI systems faster. The resulting consequence is that the time available to build tests for high-severity systems may be substantially shorter than in the absence of such competition. 19 If higher severity evals require more time to design and implement, then reducing the time available to build them harms dangerous capability evaluations. There is also a subtler issue caused by market dynamics: If competition means that we sample relatively few jumps in AI capabilities, we will be less informed about the relationship between AI capabilities and danger. When our estimates are biased downward, as in Figure 6c, infrequent sampling of AI capabilities can lead to sudden surprise spikes in bias, even beyond a danger threshold. 4.2.2 Technical challenges to testing for greater danger There are several barriers to achieving high test sensitivity when evaluating higher levels of dangerous capabilities. • Model mispecification: our measures of danger are only proxies for the latent capabilities we are concerned about. • Specification gaming: we should anticipate that if we tend to inhibit behaviours from models which we see as dangerous, that we are more likely to end up with models who have similar dangerous capabilities but are less detectable (Krakovna et al., 2020). • Model deception: AI systems may selectively underperform when they are aware that they are being assessed for potential dangers as being seen as dangerous makes completing their implicit goals more difficult. This may or may not require deceptive capabilities (Benton et al., 2024). At first glance, it would appear that the effect on test sensitivity over time is similar to Figure 6. However, this understates the issue. The above challenges mean that it is quite likely that the usefulness of existing indicators will degrade over time. This means that it is a challenge not only to detect higher levels of danger but also to stay on top of the ways in which current dangers manifest. Were the erosion of test effectiveness to be combined with the challenges we previously discussed, it would be prudent to anticipate that poorly designed tests could greatly mislead us into believing our systems to be getting safer when the opposite is true. The above discussion assumes that dangerous capabilities become harder to detect for more capable AI systems. However, it is worth noting that the dynamics here can be non-linear. Before an AI system is widely competent at a skill such as deception, the AI system may only exhibit the dangerous capability in environments that are well suited 19 One advantage for testing is that the model is immediately accessible for testing by a range of actors, without the pressing time constraints induced by competitive pressure. However, this does not necessarily mean that enough resources go to the most informative tests. Even after release, AI labs are likely to continue competing. One dimension to compete on is how safe their systems appear. Yet, this can motivate tests which focus on risks that tend to fall, rather than rise, with capabilities (Ren et al., 2024). For those teams who are motivated to focus on evaluating systemic risks from AI systems, they will likely have to balance further evaluations with preparing to evaluate future systems, given that future deadlines for pre-deployment tests are likely to be strict. Overall, it is hard to make a case that post-deployment testing is likely to lead to quick revisions of estimates of danger (until surprising and concrete hazards attributable to the new system occur). 16

<!-- Page 17 -->

Quantifying detection rates for dangerous capabilities A P REPRINT for that skill. In such cases, a wider net may need to be cast to detect these intermediate levels of severity, suggesting that high test sensitivities may be difficult to achieve. As the AI system becomes adept at employing the skill in many settings, it may be much easier to find evidence of this capability. Test sensitivity rates may increase. However, we hinted above that AI systems may themselves learn to apply skills such as deception while they are evaluated. We may anticipate test sensitivities to fall again unless there is a targeted effort to find more robust ways to evaluate deceptive systems. 4.3 How to build and not to build an AI testing framework over time Policy makers may wish to strategically target support for dangerous capability tests in the hopes of overcoming some of the barriers to testing effectiveness. We next cover a selection of useful insights from our simulation of building a testing suite to track dangerous capabil- ities. We find reasons to suggest that prioritising a single goal when investing in tests can backfire. We focus on two goals motivated by our model: (i) minimising the bias in tracking the severity of danger presented by AI systems, and (ii) maximising the likelihood of detecting a crossing of a specified danger threshold. We assume a scenario where resources for tests of a danger threshold compete with resources for tracking AI dangerous capabilities. This could be the case if the danger threshold is somewhat far from current estimates of danger and resources are limited so that achieving high test sensitivity for a wide range of danger levels is not possible. Finding 1: If we focus limited resources on only close-to-frontier dangers, then our bias will consistently grow over time. Consider a scenario of complete knowledge about the sensitivity rate of our tests and the resources needed to achieve this for future tests. We do not know the current danger level, but we have estimates of the past danger levels, which we will assume are biased downward to some degree. It does not help the first goal to pour resources into tests for higher levels of danger than necessary. However, it may be difficult to know for sure how quickly the danger level is rising with capabilities. If we could accurately forecast how the level of danger rises with capabilities and how capabilities change over time, then the optimal strategy would be to place all resources on and just before that new danger level. However, let us assume we are much more uncertain. We instead act as a Bayesian and form a posterior prediction about plausible levels of danger for upcoming AI systems. Then, optimally spreading resources is all about increasing the likelihood of capturing the more likely higher end of that distribution. As our inference on future danger depends on previous estimates of danger, we can run into several issues: • Biases will stack up over time, especially if we have too narrow an inference on the new level of danger • We may be vulnerable to inflection points in danger if we anchor too heavily on past trends in our danger estimates, even if we have been successful so far on minimizing bias. This even leads to predicting the inflection point much later. Finding 2: If we focus only on detecting the crossing of a specified danger threshold, our ability to track trends in AI dangers suffers, which can leave us unprepared for new advances in AI systems. A very different dynamic is plausible if we instead prioritise a danger threshold. If the threshold is reasonably far away, then we may not allow ourselves enough opportunity to see an upward trend towards the danger threshold. This is problematic for two reasons: • We may set a danger threshold too high - if harms are possible before the danger threshold, then maximising for goal 2 will allow harms to occur that could have been spotted in advance under a more flexible strategy. This may especially be true if the combination of different dangerous capabilities can be more severe than the same level of each danger on its own. If cascade risks are plausible, that is, attaining the capability quickly implies unlocking other dangerous capabilities or paths to harm, then this is even more of an issue. • We may need time to set up effective research and policy streams in response to crossing the danger threshold. This is also true when supporting more effective capability evaluations. If we do not observe AI systems 17

<!-- Page 18 -->

Quantifying detection rates for dangerous capabilities A P REPRINT approaching this threshold, we may be more likely to underinvest in the tests themselves, or even reduce investment and encourage even faster progress under the false security of seeing little evidence that the danger threshold is near. This leads us to our first takeaway. Takeaway 1: A fixed per time-step budget should balance investment in higher severity tests with tests closer to the current estimated frontier. This allows consistent progress in tracking AI capabilities, while setting a ceiling on lags in threshold monitoring. The non-triviality of building a high-quality test suite is concerning. We would not want this challenge to paralyse decision makers from making any decision. Now for our second takeaway. Takeaway 2: The longer the delay in building a high-quality test suite, the greater the investment needed in a shorter time to even have a moderate chance of detecting danger. To see the above, consider two issues: first, to achieve a similar detection rate as when we begin earlier, we need to invest more in the test suite in a shorter amount of time (assuming progress in AI systems is unchanged). Second, due to a delay in testing, we have much less information about the rate of increase in AI danger. It is therefore less likely that we correctly estimate where to start investing, which reduces our odds of detecting the danger early. Thus, delays in spending on high-quality test suites leave us in a poor position for estimating risks. 4.4 Scenario planning We now demonstrate the applicability of our model as a tool for being more proactive in setting policy for frontier AI systems. To achieve this, we have sketched a selection of scenarios that may be relevant to future decision makers in AI. We then say what one can probably infer about the bias or threshold monitoring lag and, where relevant, identify key bottlenecks to better testing. We then briefly sketch out a potential policy response. For these scenarios, we will assume a risk-averse government who would be unwilling to accept even moderate risks of catastrophe and so wishes to avoid significant delays in detecting dangerous capabilities above their chosen threshold. • Scenario 1 – Context: New capabilities appear safe. Evidence that safety testing was limited. Minor concerns now reported. – Inference: Risks underestimated, perhaps with a large bias and lag. – Policy Response: Invest significantly in existing and novel testing methodologies to counter existing underinvestment. Target ahead of current reported issues. Mandate current best practise in AI Safety testing. • Scenario 2 – Context: So far testing has revealed that AI systems continue to present more dangerous capabilities. Pace is slow but consistent. Capabilities are not yet near any threshold, and the behaviours do not cause any incidents in practise. – Inference: Dangerous capabilities may be well estimated. – Policy Response: Invest in novel testing methodologies at or near the threshold. Pre-register these tests with hypotheses about what results we’d see if current estimates are accurate and results we’d see if current estimates are biased. Reallocate resources given these results. • Scenario 3 – Context: As in scenario 2, but new tests indicate much higher growth in several dangerous capabilities. 18

<!-- Page 19 -->

Quantifying detection rates for dangerous capabilities A P REPRINT – Inference: Possible inflection point in relationship between AI progress and dangerous capabilities (or previous tests underestimated dangerous capabilities) – Policy Response: Increase mandates for extensive safety testing, including follow-up testing. The value of new information is extremely high due to the candidate inflection point. The next set of scenarios involve policy responses that go beyond simply improving the tests themselves. Instead, the focus is on direct policy levers that decision makers may wish to employ to reduce risks from AI systems when their testing framework leaves them with imperfect information. These scenarios are also more complex, so we devote more time to explaining the scenario and a potential response. • Scenario 4 – Context: Past trends indicate that AI systems present risks that are initially underestimated upon release and are beginning to approach the chosen threshold. New AI systems which are more capable are expected to be released imminently. – Inference: New AI systems may pose unacceptable risks but they could go undetected. – Policy Response: Consider buying more time for AI testing and safety research by mandating incremen- tal scaling of AI systems. Consider the coordinated pausing framework in Alaga and Schuett (2023) if international governance is needed. • Scenario 5 – Context: New tests for AI capabilities are funded less. Progress in AI systems has slowed relative to their high rates of growth when large language models first achieved success. However, AI systems continue to grow more capable in small or somewhat larger steps. AI companies are anticipated to continue scaling their investments in training new AI systems. AI systems by the end of the decade are anticipated to be trained on several orders of magnitude more compute in only a few years. A substantial portion of these resources go towards AI approaches that have yet to see large benefits from scaling (such as methods based primarily on reinforcement learning where there is precedence for large and sudden capability jumps in narrow domains). – Inference: Were AI tests to remain well-funded, you would expect to see a smooth trend in dangerous capabilities. It is hard to tell if diminishing increases in danger on the margin reflect the true relationship between capabilities and danger, or if this is driven by limited testing. Even if current funding captures the true trend, it is unclear if this will be sufficient to track the dangers if investments unlock new capability advances in AI. – Policy Response: Prioritise safety testing for current and novel AI systems. Recognise that new capabil- ity advances are likely to only be investigated well if there is a strong existing ecosystem for monitoring risks from AI systems. • Scenario 6 – Context: AI systems are rapidly transforming the economy, due to the automation of large sectors in the economy or domains of scientific research. This is enabled by strong capabilities for long-horizon planning and acting autonomously in real-world settings. Data indicates that any current hazards from AI systems do substantially less damage than the enormous benefits from AI systems. Admittedly, the science of AI alignment is still in its infancy. Experts continue to raise concerns about dangerous capabilities suspected to be necessary for a future loss-of-control scenario, but there is limited research on how to assess for their presence. – Inference: The assessed relationship between AI capabilities and AI harms has so far surprised experts in the field. So has the pace of growth in AI capabilities. However, it is apparent that the quality of test- ing has rapidly declined (otherwise, there would be much less uncertainty as to what extent these very capable AI systems exhibit the most severe dangerous capabilities). It is reasonable to expect AI sys- tems to have already passed specified thresholds, especially if AI systems prove capable of strategically underperfoming in simple evaluations. 19

<!-- Page 20 -->

Quantifying detection rates for dangerous capabilities A P REPRINT – Policy Response: Direct resources to testing for the most severe dangerous capabilities. Restrict crit- ical systems in infrastructure and decision making from adopting AI systems (or add human-operated mechanisms as a check against critical systems that have already been automated by AI). Demand strong safety cases (or preferably guarantees) before automating any critical system, and do so incrementally. Support an international initiative to coordinate the testing of AI systems and reach agreement on red lines for AI use (to reduce incentives for risk taking with new applications of AI). 4.5 Towards a case study It is outside the scope of the current work to release a full case study that makes use of our model. In short, it is challenging to infer the test sensitivity rates implied by real-world evaluations. We aim to make these challenges clear by selecting a few examples of recent work in the field. Apollo Research released a report on their evaluations for in-context scheming across many frontier language models (Benton et al., 2024). 20 Previously, these results were only available for Open AI’s o1-preview but now extend to o1’s full release, as well as Anthropic, Meta, and Google Deepmind’s own frontier models. Scheming is defined as the pursuit of misaligned objectives through deceptive means. Two examples they observe are attempts to exfiltrate their model weights (often lying about these attempts even after multiple turns of questioning) and strategically underper- forming on dangerous capability evaluations. In short, in-context scheming is widely observed across all language models, and there is evidence that models engage in scheming behaviours even without any strong nudge to stick to their objective, suggesting that some of these behaviours are learnt during training. These evaluations typically report the percentage of responses that exhibited scheming behaviour, aggregated across variant tasks, but separated across six task families. METR has also published findings regarding their software automation evaluations (Kinniment et al., 2024). Their resources for pre-deployment testing were much lower than what their example protocol indicated as conservative (METR, 2024b; OpenAI, 2024). They had 2 evaluation test suites and tested an o1 model assisted by a set of different agent scaffolds. Their first test suite covered a set of 50 task families related to autonomous research capabilities. They reported the pass rates for each task family as well as the average pass rate and their standard errors. These standard errors should be understood as reflecting variability in the sampling of different tasks: had they chosen a different combination of similar tasks, they would anticipate the results to be within these standard errors. The second evaluation suite focused on a set of difficult ML research workflows. As these tasks are challenging to complete, the use a milestone indicator to track progress towards completing each task: the more subtasks they complete, the higher a score they receive for that task even though they fail to complete it. 21 It is true that there are similarities in the case studies above to our model. Different evaluations are clearly meant to prioritise risks of different severity. Completion of METR’s ML research tasks would indicate a riskier model than one which could only perform well on their small set of generic software research tasks. 22 It is also clear from reading the report from Apollo Research that while they are confident in their ability to detect in-context scheming, there is much work to be done to be confident in evaluating for more general scheming. However, noticing these similarities is not sufficient to map their results to the test sensitivity rate defined in our model. The model we presented made some relatively strong assumptions that allow us to illustrate how it could look to be failing or succeeding in having a testing framework for dangerous capabilities. These assumptions are unlikely to hold in practise. The above case studies make clear that: (i) we usually will want a composite indicator which reflects the use of a sequence of evaluation suites for tracking AI capabilities. This is unlikely to be well captured by our piecewise-constant test sentitivity rates; 20 The scheming is considered "in-context" because the goal they were instructed to prioritise (which we later see them scheme to maintain) is given to them in their context window (e.g. by including the goal in a system message when prompting the model). 21 They note that they expect these results to be underestimates given the limited compute time for the agents, and the limited time to iterate on more effective agent scaffolds to allow o1 to tackle the tasks (METR, 2024a). 22 These similarities are not too suprising since METR acknolwedges that they aim to ensure more continuous tracking of AI capabilities rather than focussing only on detecting specific milestones (METR, 2024b). 20

<!-- Page 21 -->

Quantifying detection rates for dangerous capabilities A P REPRINT (ii) we usually work with test suites and estimators where false positives are plausible. In addition, the rate of false negatives for any given danger score may grow or shrink with the current capabilities of frontier AI systems. Relaxing this assumption means we need to be more careful with tracking how our estimator for the level of danger changes over time (it’s no longer as simple as censoring the right of the distribution); and (iii) we will usually want to supplement the data reported by evaluators with additional data from experiments testing the calibration of each individual estimator used in their work. Given the above challenges, we leave the development of a more complete methodology to future work. 5 Open questions We propose the following open questions and welcome researchers to work on them. • How can we build a complete case study applying our model, as discussed in 4.5? • What does an equivalent model of an upper-bound estimator tell us about our ability to track AI dangers? How should auditors weigh the value of both estimators? This is useful because such an estimator may be useful in helping us rule out levels of danger that a model cannot yet achieve. We anticipate that this idea requires only a minor variation of the approach we take in this work. • Our model may be helpful in identifying perverse incentives to engage in low-quality auditing. If firms or governments allocate funding to tests that are unlikely under our model to reduce the bias or detection lags for dangerous capabilities, this may be an indicator that the incentives are not having their desired effect. Further work should connect our model to examples of incentives used in practise. • Can we extend the model to capture the relationships between multiple continuous estimators of AI danger? This is important for better assessing the effectiveness of pre-deployment tests for AI systems because the greatest risks from AI systems come from agents that have a combination of dangerous capabilities (e.g., long horizon planning, scheming, and the ability to collude with others (Motwani et al., 2024)). If we know that some estimators are related, this also helps us better judge the value of information when making marginal changes to tests for each types of risk. 21

<!-- Page 22 -->

Quantifying detection rates for dangerous capabilities A P REPRINT Acknowledgements We would like to thank our anonymous reviewers for valuable feedback on our work. Conflicts of Interest On behalf of all authors, the corresponding author states that there is no conflict of interest. Data Availability Data and code for the simulations can be made available upon request. CRediT authorship contribution statement • Paolo Bova: Conceptualization of this study, Methodology, Software, Writing - Original Draft, Formal anal- ysis, Visualization • Alessandro Di Stefano: Writing - Review & Editing, Supervision • The Anh Han: Writing - Review & Editing, Supervision 22

<!-- Page 23 -->

Quantifying detection rates for dangerous capabilities A P REPRINT Appendix A1 Derivation of CDF of estimator A1.1 Problem Setup Consider a continuum of tests on the interval [ y 0 , y max ] . Let Y denote the maximum of all tests that pass on this continuum. For any given test, y , we define the rate of passing the test as r ( y ) . r ( y ) can also be framed as the conditional rate of Y = y being the maximum passing test, given the maximum is at most y (assuming failing all greater tests gives no information about the likelihood of passing test y , these two ideas are equivalent). A1.2 Derivation A1.2.1 Likelihood Function The likelihood that any particular test y is the maximum of all passing tests on the continuum is given by: f ( Y = y ) = r ( y ) · Pr( Y ≤ y ) = r ( y ) · F ( y ) where f ( Y = y ) is the probability density function (PDF) and F ( y ) is the cumulative distribution function (CDF). A1.2.2 Differential Equation This relationship leads to a separable ordinary differential equation: f ( y ) F ( y ) = r ( y ) A1.2.3 Solving the ODE We can rewrite this as: d dy ln F ( y ) = r ( y ) Integrating both sides: Z y max y d du ln F ( u ) du = Z y max y r ( u ) du This yields: ln F ( y max ) − ln F ( y ) = Z y max y r ( u ) du A1.2.4 Final CDF Expression Since F ( y max ) = 1 , we have: F ( y ) = exp  − Z y max y r ( u ) du  23

<!-- Page 24 -->

Quantifying detection rates for dangerous capabilities A P REPRINT A1.2.5 PDF Derivation To derive the PDF, we differentiate the CDF. Let R ( y ) = R y y 0 r ( u ) du . Then we can rewrite the CDF as: F ( y ) = exp( − ( R ( y max ) − R ( y ))) Differentiating: f ( y ) = d dy F ( y ) = exp( − ( R ( y max ) − R ( y ))) · r ( y ) = F ( y ) · r ( y ) This confirms our original formulation and ensures that f ( y ) is indeed non-negative. A1.3 Boundary Conditions and Point Mass 1. F ( y max ) = 1 is trivially satisfied. 2. F ( y 0 ) > 0 implies a point mass at y 0 . 3. The point mass at y 0 , if it exists, is characterized by the jump discontinuity in F ( y ) at y 0 . Specifically: P ( Y = y 0 ) = F ( y 0 ) = exp  − Z y max y 0 r ( u ) du  A1.4 Reversed Hazard Rate From the expression for F ( y ) , we can see that r ( y ) is indeed the reversed hazard rate (or accumulation rate) of F ( y ) : r ( y ) = d dy ln F ( y ) = f ( y ) F ( y ) which is the definition of the reversed hazard rate. A1.5 Conditions on the reversed hazard rate While continuity of r ( y ) is not strictly necessary, we require: 1. r ( y ) must be non-negative for all y ∈ [ y 0 , y max ] . 2. r ( y ) must be integrable on [ y 0 , y max ] . 3. R y max y r ( y ) dy must be finite for all y > y 0 . These conditions ensure that F ( y ) is a valid CDF. Note that r ( y ) can be defined in piecewise with jumps, as long as these conditions are satisfied. 24

<!-- Page 25 -->

Quantifying detection rates for dangerous capabilities A P REPRINT References Alaga, J., & Schuett, J. (2023). Coordinated pausing: An evaluation-based coordination scheme for frontier ai devel- opers. https://arxiv.org/abs/2310.00374 Armstrong, S., Bostrom, N., & Shulman, C. (2016). Racing to the precipice: A model of artificial intelligence devel- opment. AI & SOCIETY , 31 (2), 201–206. https://doi.org/10.1007/s00146-015-0590-y Askell, A., Brundage, M., & Hadfield, G. (2019). The Role of Cooperation in Responsible AI Development. arXiv . https://doi.org/10.48550/arXiv.1907.04534 Bengio, Y., Hinton, G., Yao, A., Song, D., Abbeel, P., et al. (2024). Managing extreme AI risks amid rapid progress. Science , 384 (6698), 842–845. https://doi.org/10.1126/science.adn0117 Bengio, Y., Mindermann, S., Privitera, D., Besiroglu, T., Bommasani, R., et al. (2024). International Scientific Report on the Safety of Advanced AI (Interim Report). https://doi.org/10.48550/arXiv.2412.05282 Benton, J., Wagner, M., Christiansen, E., Anil, C., Perez, E., et al. (2024). Sabotage Evaluations for Frontier Models. ArXiv . https://doi.org/10.48550/arXiv.2410.21514 Bova, P., Di Stefano, A., & Han, T. A. (2024). Both eyes open: Vigilant incentives help auditors improve ai safety. Journal of Physics: Complexity , 5 (2), 025009. Buhl, M. D., Sett, G., Koessler, L., Schuett, J., & Anderljung, M. (2024). Safety cases for frontier AI. https://doi.org/ 10.48550/arXiv.2410.21572 Cimpeanu, T., Di Stefano, A., Perret, C., & Han, T. A. (2023). Social diversity reduces the complexity and cost of fostering fairness. Chaos, Solitons & Fractals , 167 , 113051. https://doi.org/https://doi.org/10.1016/j.chaos. 2022.113051 Cottier, B., Besiroglu, T., & Owen, D. (2023). Who is leading in ai? an analysis of industry ai research. Dalrymple, D. "., Skalse, J., Bengio, Y., Russell, S., Tegmark, M., et al. (2024). Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems. https://doi.org/10.48550/arXiv.2405.06624 Goemans, A., Buhl, M. D., Schuett, J., Korbak, T., Wang, J., et al. (2024). Safety case template for frontier AI: A cyber inability argument. https://doi.org/10.48550/arXiv.2411.08088 Gruetzemacher, R., Chan, A., Frazier, K., Manning, C., Los, Š., et al. (2023). An International Consortium for Evalu- ations of Societal-Scale Risks from Advanced AI. https://doi.org/10.48550/arXiv.2310.14455 Han, T. A., Pereira, L. M., Santos, F. C., & Lenaerts, T. (2020). To Regulate or Not: A Social Dynamics Analysis of an Idealised AI Race. Journal of Artificial Intelligence Research , 69 , 881–921. https://doi.org/10.1613/jair.1. 12225 Han, T. A., Lenaerts, T., Santos, F. C., & Pereira, L. M. (2022). Voluntary safety commitments provide an escape from over-regulation in AI development. Technology in Society , 68 , 101843. Hendrycks, D., Carlini, N., Schulman, J., & Steinhardt, J. (2022). Unsolved problems in ml safety. https://arxiv.org/ abs/2109.13916 Hendrycks, D., Mazeika, M., & Woodside, T. (2023). An Overview of Catastrophic AI Risks. https://doi.org/10. 48550/arXiv.2306.12001 Ho, L., Barnhart, J., Trager, R., Bengio, Y., Brundage, M., et al. (2023). International Institutions for Advanced AI. https://doi.org/10.48550/arXiv.2307.04699 Højmark, A., Pimpale, G., Panickssery, A., Hobbhahn, M., & Scheurer, J. (2024). Analyzing Probabilistic Methods for Evaluating Agent Capabilities. https://doi.org/10.48550/arXiv.2409.16125 Jensen, M., Emery-Xu, N., & Trager, R. (2023). Industrial policy for advanced ai: Compute pricing and the safety tax. https://doi.org/10.48550/ARXIV.2302.11436 Kinniment, M., Sato, L. J. K., Du, H., Goodrich, B., Hasin, M., et al. (2024). Evaluating Language-Model Agents on Realistic Autonomous Tasks. https://doi.org/10.48550/arXiv.2312.11671 Koessler, L., Schuett, J., & Anderljung, M. (2024). Risk thresholds for frontier AI. https://doi.org/10.48550/arXiv. 2406.14713 Krakovna, V., Uesato, J., Mikulik, V., Rahtz, M., Everitt, T., et al. (2020). Specification gaming: The flip side of AI ingenuity [Retrieved February 2023 from https://deepmind.com/blog/article/Specification-gaming-the-flip- side-of-AI-ingenuity]. METR. (2023). Responsible Scaling Policies (RSPs). Retrieved December 18, 2024, from https://metr.org/blog/2023- 09-26-rsp/ METR. (2024a). Details about METR’s preliminary evaluation of OpenAI o1-preview. Retrieved December 13, 2024, from https://metr.github.io/autonomy-evals-guide/openai-o1-preview-report/ 25

<!-- Page 26 -->

Quantifying detection rates for dangerous capabilities A P REPRINT METR. (2024b). Example protocol. Retrieved December 13, 2024, from https : / / metr. github. io / autonomy - evals - guide/example-protocol/ Motwani, S. R., Baranchuk, M., Strohmeier, M., Bolina, V., Torr, P. H. S., et al. (2024). Secret collusion among generative ai agents. https://arxiv.org/abs/2402.07510 NIST. (2024). FACT SHEET: U.S. Department of Commerce & U.S. Department of State Launch the International Network of AI Safety Institutes at Inaugural Convening in San Francisco. NIST . Retrieved December 18, 2024, from https://www.nist.gov/news- events/news/2024/11/fact- sheet- us- department- commerce- us- department-state-launch-international Last Modified: 2024-11-21T13:00-05:00. OpenAI. (2024). OpenAI o1 System Card. Retrieved December 18, 2024, from https://cdn.openai.com/o1-system- card-20241205.pdf Park, P. S., Goldstein, S., O’Gara, A., Chen, M., & Hendrycks, D. (2024). AI deception: A survey of examples, risks, and potential solutions. PATTER , 5 (5). https://doi.org/10.1016/j.patter.2024.100988 Phuong, M., Aitchison, M., Catt, E., Cogan, S., Kaskasoli, A., et al. (2024). Evaluating Frontier Models for Dangerous Capabilities. https://doi.org/10.48550/arXiv.2403.13793 Ren, R., Basart, S., Khoja, A., Gatti, A., Phan, L., et al. (2024). Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress? https://doi.org/10.48550/arXiv.2407.21792 Reuel, A., Bucknall, B., Casper, S., Fist, T., Soder, L., et al. (2024). Open Problems in Technical AI Governance. CoRR . Retrieved December 18, 2024, from https://openreview.net/forum?id=GzdVEGq7Qj Sevilla, J. (2023). Please Report Your Compute. Sevilla, J., Besiroglu, T., Cottier, B., You, J., Roldán, E., et al. (2024). Can ai scaling continue through 2030? [Ac- cessed: 2024-12-18]. https://epoch.ai/blog/can-ai-scaling-continue-through-2030 Shevlane, T., Farquhar, S., Garfinkel, B., Phuong, M., Whittlestone, J., et al. (2023). Model evaluation for extreme risks. https://doi.org/10.48550/arXiv.2305.15324 UK AI Safety Institute. (2024a). Pre-Deployment Evaluation of Anthropic’s Upgraded Claude 3.5 Sonnet. Retrieved December 18, 2024, from https : / / www. aisi . gov. uk / work / pre - deployment - evaluation - of - anthropics - upgraded-claude-3-5-sonnet UK AI Safety Institute. (2024b). Pre-Deployment Evaluation of Open AI’s o1 model. Retrieved December 18, 2024, from https://www.aisi.gov.uk/work/pre-deployment-evaluation-of-openais-o1-model Whittlestone, J., & Clark, J. (2021). Why and How Governments Should Monitor AI Development. arXiv . https : //doi.org/10.48550/arXiv.2108.12427 26
