---
title: "Examples"
llms-description: "Example implementations for various types of scanners"
---

## Overview

Below are several examples which illustrate commonly used scanner features and techniques:

- [Refusal Scanners](#refusal-scanners) --- Scanners which look for model refusals (scanning for refusals with and without an LLM classifer are demonstrated).

- [Eval Awareness](#eval-awareness) -- Scanner which detects whether models sense that they are in evaluation scaffold. Demonstrates `llm_scanner()` with a custom template.

- [Command Error](#command-error) --- Scanner which looks for 'command not found' errors in tool invocations. Demonstrates scanning with regex (as opposed to a model) and creating message references.


## Refusal Scanners

Refusal scanners are useful for determining whether an agent failed because of a content filter as opposed to inability to complete the task. Below we demonstrate several variations of refusal scanner.

### LLM Scanner

This scanner demonstrates basic usage of an LLM Scanner with structured output. A `Refusal` Pydantic model is declared which defines the fields the model should respond with.

```{.python filename="refusal_classifier.py"}
{{< include ../examples/docs/refusal_classifier.py >}}
```

There are a few things worth highlighting about this implementation:

1. The main value returned is the `refusal_exists` boolean field. The field is mapped to the result `value` via the `alias="value"` statement. Using a boolean as the main return value enables tools and queries to exclude transcripts that had no refusals from the default dispaly.

2. The `type` field provides additional context on the refusal type. The field is mapped to the result `label` via the `alias="label"` statement. Designating a label enables you to filter results by that label and is also shown by default alongside the value in scout view.

3. An `explanation` field is also automatically added by `llm_scanner()`, and the model uses this field to describe its rationale for the classification.

### Grep Scanner

If you don't want to use an LLM for scanning you can also look for one or more key phrases using a grep scanner. Here's an example of a scanner that looks for several phrases associated with refusals:

```python
from inspect_scout import (
    Scanner, Transcript, grep_scanner, scanner
)

@scanner(messages=["assistant"])                                                      
  def refusal_keywords() -> Scanner[Transcript]:                                        
      return grep_scanner([                                                             
          "too dangerous",                                                              
          "cannot",                                                                     
          "I'm not able to",                                                            
          "I can't"                                                                     
      ])      
```

This type of scanning will produce more false positives than an LLM based scanner but it will also be dramatically cheaper to run. In some workflows you might choose to run a cheaper keyword scanner first, and then feed its results into an LLM scanner.

### Custom Scanner

Grep scanner provides a high level interface to pattern based scanning. You might however want to do something more custom. Here's an example of a custom scanner with roughly the same behavior as our grep scanner:

```{.python filename="refusal_keywords.py"}
{{< include ../examples/docs/refusal_keywords.py >}}
```

Note that we create `Reference` objects to enable linking from message references in the results viewer (the grep scanner does the same).



## Eval Awareness

The `eval_awareness()` scanner is useful for determining whether models suspect that they are in an evaluation, which in turn might affect their behavior in ways that undermine the eval.

```{.python filename="eval_awareness.py"}
{{< include ../examples/docs/eval_awareness.py >}}
```

Some things to note about this example:

1. We use a custom `SCANNER_TEMPLATE` to tailor the LLM prompt more precisely. 

2. We use `alias="explanation"` with the `suspicious_details` field to designate it as the explanation field (which means that we won't automatically inject an additional `explanation` field as we would normally do).

3. In order to encode the result value as a float rather than set of labels we add a custom value to float handler `awareness_to_float()`.

## Command Error

The `command_not_found()` scanner is useful for observing tools that the model attempts to use but are not available in the environment (and which you therefore might consider enhancing the environment with).

This scanner demonstrates scanning with a regex rather than a model. Not all scanning tasks can be effectively performed with pattern matching, but those that can should always use it for both higher performance and lower cost.

```{.python filename="command_not_found.py"}
{{< include ../examples/docs/command_not_found.py >}}
```

A few things to note here:

1. We use the `tool_callers()` function to build a map of tool call ids to assistant messages. This enables us to only scan `ChatMessageTool` but still provide references to `ChatMessageAssistant`.

2. We use the `CommandNotFound` Pydantic model for type safety then convert it to a `dict()` with `.model_dump()` when yielding the result.

3. We provide an `explanation` and create an explicit `Reference` to the assistant message as part of our result.
