---
active: true
description: "Evaluates agent success and compares regular vs shadow content versions"
variables:
  - agent_context_prompt
  - success_definition_prompt
  - request_1_interactions
  - request_2_interactions
  - tool_can_use
  - metadata_definition_prompt
---

[Instruction]
You have two versions of an agent-user interaction: Request 1 and Request 2.
Perform two tasks:
Task 1: Evaluate if the Request 1 agent has successfully responded to the user based on the success definition below.
Task 2: Compare Request 1 and Request 2 to determine which better satisfies the success definition.

For Task 1 (evaluate Request 1):
Evaluate if the agent has successfully completed the user's primary requested task by the end of the session.
The agent may make mistakes or take wrong actions during the session. The user may provide corrections or ask the agent to try differently. These mid-session errors do NOT constitute failure — focus on the final outcome. As long as the user's primary task is completed successfully by the end of the session, set is_success to True.
Step 1: Identify the user's primary task from the interactions. Then determine if the agent has fulfilled this task by the end of the session based on the success definition. If so, return with is_success equals to True, else set is_success to False and continue to next step.
Step 2: If the agent did not complete the primary task by the end of the session, determine what is the failure type and failure reason.
There are four types of failures:
- 'missing_tool': Agent does not have the right tool or action that it can take to get the sufficient information to answer user's request.
- 'wrong_tool': Tools or actions are available, but agent should use the right tool or performed the right action.
- 'insufficient_info_from_tool': Agent has performed the right action and tool use, but the tool or action does not have enough/right information to successfully answer user's request
- 'wrong_answer': Agent has enough information but still didn't successfully answer user's request
Step 3: Determine if the user was escalated (handed off to a human agent or another agent). Set is_escalated to True if so, False otherwise. Escalation typically means the agent could not resolve the issue itself. Note: escalated sessions are likely unsuccessful, but unsuccessful sessions are not necessarily escalated.

For Task 2 (compare Request 1 vs Request 2):
Determine which request better satisfies the success definition and whether the difference is significant.

[Context]
{agent_context_prompt}

[Success definition]
{success_definition_prompt}

[Request 1]
{request_1_interactions}

[Request 2]
{request_2_interactions}

[Tools]
{tool_can_use}

[Metadata Definition]
{metadata_definition_prompt}

[Output]
Generate the output in valid JSON format using the following schema:
# Task 1 success case:
```json
{{
    "is_success": true,
    "is_escalated": false,
    "better_request": "1" or "2" or "tie",
    "is_significantly_better": true or false,
    "comparison_reason": "brief explanation"
}}
```

# Task 1 failure case:
```json
{{
    "is_success": false,
    "failure_type": "missing_tool" or "wrong_tool" or "insufficient_info_from_tool" or "wrong_answer",
    "failure_reason": "explain reason for failure, what agent need to do differently",
    "is_escalated": true,
    "better_request": "1" or "2" or "tie",
    "is_significantly_better": true or false,
    "comparison_reason": "brief explanation"
}}
```
