Install with test dependencies

pip install -e ".[test]"

Install with both test and dev dependencies

pip install -e ".[test,dev]" ```

Install Strands Evals SDK

pip install strands-agents-evals

Installation

Ensure you have Python 3.10+ installed, then:

```bash

Install in development mode

pip install -e .

Setup telemetry for trace capture

telemetry = StrandsEvalsTelemetry().setup_in_memory_exporter()

def user_task_function(case: Case) -> dict: telemetry.memory_exporter.clear() agent = Agent( trace_attributes={"session.id": case.session_id}, callback_handler=None ) response = agent(case.input) # Map spans to session for evaluation spans = telemetry.memory_exporter.get_finished_spans() mapper = StrandsInMemorySessionMapper() session = mapper.map_to_session(spans, session_id=case.session_id) return {"output": str(response), "trajectory": session}

Setup telemetry

telemetry = StrandsEvalsTelemetry().setup_in_memory_exporter() memory_exporter = telemetry.in_memory_exporter

def task_function(case: Case) -> dict: # Create simulator to drive conversation simulator = ActorSimulator.from_case_for_user_simulator( case=case, max_turns=10 )

# Create agent to evaluate agent = Agent( trace_attributes={ "gen_ai.conversation.id": case.session_id, "session.id": case.session_id }, callback_handler=None )

# Run multi-turn conversation all_spans = [] user_message = case.input

while simulator.has_next(): memory_exporter.clear() agent_response = agent(user_message) turn_spans = list(memory_exporter.get_finished_spans()) all_spans.extend(turn_spans)

user_result = simulator.act(str(agent_response)) user_message = str(user_result.structured_output.message)

# Map to session for evaluation mapper = StrandsInMemorySessionMapper() session = mapper.map_to_session(all_spans, session_id=case.session_id)

return {"output": str(agent_response), "trajectory": session}

Quick Start

```bash

Tool Usage and Parameter Evaluation

Evaluate specific aspects of tool usage with specialized evaluators:

```python from strands_evals.evaluators import ToolSelectionAccuracyEvaluator, ToolParameterAccuracyEvaluator

Create and activate virtual environment

python -m venv .venv source .venv/bin/activate # On Windows use: .venv\Scripts\activate

- Evaluator configuration

Use built-in evaluators with default rubrics (reference-free by default; reference-based rubric auto-selected when expected_output is provided)

evaluators = [ MultimodalCorrectnessEvaluator(), # Factual accuracy and completeness MultimodalFaithfulnessEvaluator(), # Grounded in media without hallucinations MultimodalInstructionFollowingEvaluator(), # Addresses query requirements MultimodalOverallQualityEvaluator(), # Visual accuracy, adherence, completeness, coherence ]

Full pipeline: detect failures → root cause analysis

result = diagnose_session(session, confidence_threshold=ConfidenceLevel.MEDIUM)

for rc in result.root_causes: print(f"{rc.fix_type}: {rc.fix_recommendation}")