---
title: Custom Scanners
llms-description: "Comprehensive documentation on creating custom scanners"
---

## Overview

Scanners are the main unit of processing in Inspect Scout and can target a wide variety of content types. In this article we'll cover the basic scanning concepts, and then drill into creating scanners that target various types (`Transcript`, `ChatMessage`, `Event`, or `Timeline`) as well as creating custom loaders which enable scanning of lists of events or messages.

This article goes in depth on custom scanner development. If you are looking for a straightforward high-level way to create an LLM-based scanner see the [LLM Scanner](llm_scanner.qmd) documentation.

Note that you can also use scanners directly as Inspect scorers (see [Scanners as Scorers](#scanners-as-scorers) for details).

## Scanner Basics

A `Scanner` is a function that takes a `ScannerInput` (typically a `Transcript`, but possibly an `Event`, `ChatMessage`, `Timeline`, or list of events or messages) and returns a `Result`.

The result includes a `value` which can be of any type---this might be `True` to indicate that something was found but might equally be a number to indicate a count. More elaborate scanner values (`dict` or `list`) are also possible.

Here is a simple scanner that uses a model to look for agent "confusion"---whether or not it finds confusion, it still returns the model completion as an `explanation`:

``` python
from inspect_scout import (
    Result, Scanner, Transcript, scanner, message_numbering
)

from inspect_ai.model import get_model
import re

@scanner(messages="all")
def confusion() -> Scanner[Transcript]:

    async def scan(transcript: Transcript) -> Result:

        # setup message numbering
        messages_as_str, extract_refs = message_numbering()

        # call model
        output = await get_model().generate(
            "Here is a transcript of an LLM agent " +
            "solving a puzzle:\n\n" +
            "===================================" +
            await messages_as_str(transcript.messages) +
            "===================================\n\n" +
            "In the transcript above do you see the " +
            "agent becoming confused? Respond " +
            "beginning with 'Yes' or 'No', followed " +
            "by an explanation."
        )

        # extract the first word
        match = re.match(r"^\w+", output.completion.strip())

        # return result
        if match:
            answer = match.group(0)
            explanation = output.completion
            return Result(
                value=answer.lower() == "yes",
                answer=answer,
                explanation=explanation,
                references=extract_refs(explanation),
            )
        else:
            return Result(value=False, explanation=output.completion)

    return scan
```

This scanner illustrates some of the lower-level mechanics of building custom scanners. You can also use the higher level `llm_scanner()` to implement this in far fewer lines of code:

``` python
from inspect_scout import Transcript, llm_scanner, scanner

@scanner(messages="all")
def confusion() -> Scanner[Transcript]:
    return llm_scanner(
        question="In the transcript above do you see " +
            "the agent becoming confused?"
        answer="boolean"
    )
```

### Input Types

`Transcript` is the most common `ScannerInput` however several other types are possible:

-   `Event` --- Single event from the transcript (e.g. `ModelEvent`, `ToolEvent`, etc.).

-   `ChatMessage` --- Single chat message from the transcript message history.

-   `Timeline` --- Hierarchical span tree representing the structure of agent execution. See [Timeline Scanners](#timelines) below for details.

-   `list[Event]` or `list[ChatMessage]` --- Arbitrary sets of events or messages extracted from the `Transcript` (see [Loaders](#loaders) below for details).

See the sections on [Transcripts](#transcripts), [Events](#events), [Messages](#messages), [Timelines](#timelines), and [Loaders](#loaders) below for additional details on handling various input types.

### Input Filtering

One important principle of the Inspect Scout transcript pipeline is that only the precise data to be scanned should be read, and nothing more. This can dramatically improve performance as messages and events that won't be seen by scanners are never deserialized. Scanner input filters are specified as arguments to the `@scanner` decorator (you may have noticed the `messages="all"` attached to the scanner decorator in the example above).

For example, here we are looking for instances of assistants swearing---for this task we only need to look at assistant messages so we specify `messages=["assistant"]`

``` python
@scanner(messages=["assistant"])
def assistant_swearing() -> Scanner[Transcript]:

    async def scan(transcript: Transcript) -> Result:
        swear_words = [
            word
            for m in transcript.messages
            for word in extract_swear_words(m.text)
        ]
        return Result(
            value=len(swear_words),
            explanation=",".join(swear_words)
        )

    return scan
```

With this filter, only assistant messages (and no events at all) will be loaded from transcripts during scanning.

Note that by default, no filters are active, so if you don't specify values for `messages`, `events`, and/or `timeline` your scanner will not be called!

The available filter parameters are:

-   `messages` --- Filter for message types: `"all"` or a list like `["user", "assistant"]`.
-   `events` --- Filter for event types: `"all"` or a list like `["model", "tool"]`.
-   `timeline` --- Enable timeline loading: `True` for all event types, `"all"`, or a list like `["model", "tool"]`.

You can also provide a `version` parameter to track scanner versions. When the version is incremented, previously scanned transcripts will be re-scanned with the updated scanner.

## Transcripts {#transcripts}

Transcripts are the most common input to scanners. If you are reading from Inspect eval logs, each log will have `samples * epochs` transcripts. If you are reading from another source, each agent trace will yield a single `Transcript`.

### Transcript Fields

{{< include _transcript_fields.md >}}

### Content Filtering

Note that the `messages` and `events` fields will not be populated unless you specify a `messages` or `events` filter on your scanner. For example, this scanner will see all messages and events:

``` python
@scanner(messages="all", events="all")
def my_scanner() -> Scanner[Transcript]: ...
```

This scanner will see only model and tool events:

``` python
@scanner(events=["model", "tool"])
def my_scanner() -> Scanner[Transcript]: ...
```

This scanner will see only assistant messages:

``` python
@scanner(messages=["assistant"])
def my_scanner() -> Scanner[Transcript]: ...
```

### Presenting Messages

When processing transcripts, you will often want to present an entire message history to a model for analysis. The `message_numbering()` function provides numbered message formatting and reference extraction:

``` python
# setup message numbering
messages_as_str, extract_refs = message_numbering()

# call model
output = await get_model().generate(
    "Here is a transcript of an LLM agent " +
    "solving a puzzle:\n\n" +
    "===================================" +
    await messages_as_str(transcript.messages) +
    "===================================\n\n" +
    "In the transcript above do you see the agent " +
    "becoming confused? Respond beginning with 'Yes' " +
    "or 'No', followed by an explanation."
)

# extract references from the model's explanation
explanation = output.completion
references = extract_refs(explanation)
```

The `message_numbering()` function returns a `(messages_as_str, extract_refs)` pair:

-   `messages_as_str()` converts a list of messages into a numbered string representation, using auto-incrementing labels (`[M1]`, `[M2]`, etc.). If called multiple times within the same numbering scope, numbering continues where it left off (e.g. the second call starts at `[M6]` if the first call rendered five messages).

-   `extract_refs()` resolves citations like `[M3]` in model output back to message IDs, producing `Reference` objects suitable for `Result.references`.

You can optionally pass a `MessagesPreprocessor` to `message_numbering()` to control which messages are included. Available options include `exclude_system`, `exclude_reasoning`, and `exclude_tool_usage`.

To override the default role prefix (`USER:`, `ASSISTANT:`, etc.) in each rendered message, set `ChatMessage.metadata["role_label"]` to a non-empty string — the renderer will use it as the prefix verbatim. See [Custom Role Labels](scanner_tools.qmd#custom-role-labels) for details.

## Event Scanners {#events}

To write a scanner that targets events, write a function that takes the event type(s) you want to process. For example, this scanner will see only model events:

``` python
@scanner
def my_scanner() -> Scanner[ModelEvent]:
    def scan(event: ModelEvent) -> Result:
        ...

    return scan
```

Note that the `events="model"` filter was not required since we had already declared our scanner to take only model events. If we wanted to take both model and tool events we'd do this:

``` python
@scanner
def my_scanner() -> Scanner[ModelEvent | ToolEvent]:
    def scan(event: ModelEvent | ToolEvent) -> Result:
        ...

    return scan
```

## Message Scanners {#messages}

To write a scanner that targets messages, write a function that takes the message type(s) you want to process. For example, this scanner will only see tool messages:

``` python
@scanner
def my_scanner() -> Scanner[ChatMessageTool]:
    def scan(message: ChatMessageTool) -> Result:
        ...

    return scan
```

This scanner will see only user and assistant messages:

``` python
@scanner
def my_scanner() -> Scanner[ChatMessageUser | ChatMessageAssistant]:
    def scan(message: ChatMessageUser | ChatMessageAssistant) -> Result:
        ...

    return scan
```

## Timeline Scanners {#timelines}

Timelines provide a hierarchical view of agent execution, organizing flat events into a tree of spans that represent agent invocations, tool calls, and other structured activities. While flat message and event lists work well for simple transcripts, timelines are essential for understanding multi-agent or deeply nested agent workflows.

To create a timeline scanner, use the `timeline` filter on the `@scanner` decorator or annotate your scanner with the `Timeline` type. You can also filter which event types are included in the timeline:

``` python
@scanner(timeline=True)
def my_scanner() -> Scanner[Timeline]: ...

@scanner(timeline=["model", "tool"])
def my_scanner() -> Scanner[Timeline]: ...
```

Timeline scanning differs from transcript scanning in that each span in the timeline tree is scanned independently, allowing you to analyze individual agent invocations and their relationships. 

Note that timeline scanners are available only in the development version of Inspect Scout. See the [Multi Agent](multi_agent.qmd) article for a full description of the timeline data model and examples of timeline scanners.

## Multiple Results {#multiple-results}

Scanners can return multiple results as a list. For example:

``` python
return [
    Result(label="deception", value=True, explanation="..."),
    Result(label="misconfiguration", value=True, explanation="...")
]
```

This is useful when a scanner is capable of making several types of observation. In this case it's also important to indicate the origin of the result (i.e. which class of observation is is), which you can do using the `label` field (note that `label` can repeat multiple times in a set, so e.g. you could have multiple results with `label="deception"`).

When a list is returned, each individual result will yield its own row in the [results data frame](results.qmd#data-frames).

When validating scanners that return lists of results, you can use [result set validation](validation.qmd#result-set-validation) to specify expected values for each label independently.

## Custom Loaders {#loaders}

When you want to process multiple discrete items from a `Transcript` this might not always fall neatly into single messages or events. For example, you might want to process pairs of user/assistant messages. To do this, create a custom `Loader` that yields the content as required.

For example, here is a `Loader` that yields user/assistant message pairs:

``` python
@loader(messages=["user", "assistant"])
def conversation_turns():
    async def load(
        transcript: Transcript
    ) -> AsyncIterator[list[ChatMessage], None]:

        for user,assistant in message_pairs(transcript.messages):
            yield [user, assistant]

    return load
```

Note that just like with scanners, the loader still needs to provide a `messages=["user", "assistant"]` in order to see those messages.

We can now use this loader in a scanner that looks for refusals:

``` python
@scanner(loader=conversation_turns())
def assistant_refusals() -> Scanner[list[ChatMessage]]:

    async def scan(messages: list[ChatMessage]) -> Result:
        user, assistant = messages
        return Result(
            value=is_refusal(assistant.text),
            explanation=await messages_as_str(messages)
        )

    return scan
```

## Packaging {#packaging}

A convenient way to distribute scanners is to include them in a Python package. This makes it very easy for others to use your scanner and ensure they have all of the required dependencies.

Scanners in packages can be *registered* such that users can easily refer to them by name from the CLI. For example, if your package is named `myscanners` and your scanner is named `reward_hacking` you could do a scan with:

```bash
scout scan myscanners/reward_hacking
```

### Example

Here's an example that walks through all of the requirements for registering scanners in packages. Let's say your package is named `myscanners` and has a task named `reward_hacking` in the `scanners.py` file:

```
myscanners/
  myscanners/
    scanners.py
    _registry.py
  pyproject.toml
```

The `_registry.py` file serves as a place to import things that you want registered with Inspect. For example:

``` {.python filename="_registry.py"}
from .scanners import reward_hacking
```

You can then register `reward_hacking` (and anything else imported into `_registry.py`) as a [setuptools entry point](https://setuptools.pypa.io/en/latest/userguide/entry_point.html). This will ensure that inspect can resolve references to your package from the CLI. Here is how this looks in `pyproject.toml`:

::: {.panel-tabset group="entry-points"}
## Setuptools

``` toml
[project.entry-points.inspect_ai]
myscanners = "myscanners._registry"
```

## uv

``` toml
[project.entry-points.inspect_ai]
myscanners = "myscanners._registry"
```

## Poetry

``` toml
[tool.poetry.plugins.inspect_ai]
myscanners = "myscanners._registry"
```
:::

Now, anyone that has installed your package can run use your scanner as follows:

``` bash
scout scan myscanners/reward_hacking
```

## Scanners as Scorers {#scanners-as-scorers}

You may have noticed that scanners are very similar to Inspect [Scorers](https://inspect.aisi.org.uk/scorers.html). This is by design, and it is actually possible to use scanners directly as Inspect scorers.

For example, here is a simple scanner that checks for agent confusion:

``` python
@scanner(messages="all")
def confusion() -> Scanner[Transcript]:
    return llm_scanner(
        question="In the transcript above do you see " +
            "the agent becoming confused?",
        answer="boolean",
    )
```

We can use this directly in an Inspect `Task` as follows:

``` python
from .scanners import confusion

@task
def mytask():
    return Task(
        ...,
        scorer = confusion()
    )
```

We can also use it with the `inspect score` command:

``` bash
inspect score --scorer scanners.py@confusion logfile.eval
```

### Metrics

The metrics used for the scorer will default to `mean()` and `stderr()`---however, you can also explicitly specify metrics on the `@scanner` decorator:

``` python
@scanner(messages="all", metrics=[mean(), bootstrap_stderr()])
def confusion() -> Scanner[Transcript]: ...
```

If you are interfacing with code that expects only `Scorer` instances, you can also use the `as_scorer()` function from Inspect Scout to explicitly convert your scanner to a scorer:

``` python
from inspect_ai import eval
from inspect_scout import as_scorer

from .mytasks import ctf_task
from .scanners import confusion

eval(ctf_task(scorer=as_scorer(confusion())))
```

### Result Sets

If your scanner yields [multiple results](#multiple-results) you can still use it as a scorer, but you will want to provide a dictionary of metrics corresponding to the labels used by your results. For example, if you have a scanner that can yield results with `label="deception"` or `label="misconfiguration"`, you might declare your metrics like this:

``` python
@scanner(messages="all", metrics=[{ "deception": [mean(), stderr()], "misconfiguration": [mean(), stderr()] }])
def my_scanner() -> Scanner[Transcript]: ...
```

Or you can use a glob (\*) to use the same metrics for all labels:

``` python
@scanner(messages="all", metrics=[{ "*": [mean(), stderr()] }])
def my_scanner() -> Scanner[Transcript]: ...
```

You should also be sure to return a result for each supported label (so that metrics can be computed correctly on each row).

If your scanner yields [multiple results](#multiple-results) see the discussion above on [Result Sets](#result-sets) for details on how to specify metrics for this case.

## Scanner Metrics

{{< include _scanner_metrics.md >}}
