---
type: email
from: sebbe42 <notifications@github.com>
subject: Re: [openai/openai-agents-python] How do I handle image outputs from

 tools and feed it back to the LLM during an agent run? (Issue #1553)
received: 2026-03-06 07:13:47
date_header: Sun, 24 Aug 2025 23:37:35 -0700
priority: normal
status: awaiting_approval
uid: 28601
---

## Email Content

**From:** sebbe42 <notifications@github.com>
**Subject:** Re: [openai/openai-agents-python] How do I handle image outputs from

 tools and feed it back to the LLM during an agent run? (Issue #1553)
**Date:** Sun, 24 Aug 2025 23:37:35 -0700

### Body (preview)

sebbe42 left a comment (openai/openai-agents-python#1553)



Ok so. Even if you cache images in the tool via the context wrapper and you catch the tool call via hooks, you cannot append any user messages to the runners items as they are not accessible. if you are really eager you'd have to modify the source code, but it'll be difficult.



Alternatives:

* use an llm call within the tool to evaluate the image and return that as part of the string. the tool can take as input any evaluation criteria from the main agent.

* grab the images from context after the run and automatically insert them before any next user messages, to ensure they end up in the history. you could go as far as prompting for 'after calling <tool> as the user to send you the images' with a certain syntax your application checks for, then automatically run the agent with the images without prompting the user for inputs.

* switch to pydantic_ai that natively supports the user message behavior, it works. Main drawback is they dont support carrying over reasoning from previous responses. (this was my motivation to switch to openai agents)



@seratch would you mind explaining how relaying the URL to the model will make the image end up in the conversation history? Also, since agents on the openai website can operate with images via code and reason about the outcomes (like cropping/zooming), agents are acting as if they have multimodal tool responses, at least on the website. So there is an internal workaround?



I have to say, given that models nowadays do tool calling within reasoning, its a substantial limitation that tool returns are not multimodal. its easily fixed when using the api and appending user messages, but while its not intuitive, it actually works and is simple. 



-- 

Reply to this email directly or view it on GitHub:

https://github.com/openai/openai-agents-python/issues/1553#issuecomment-3219016566

You are receiving this because you are subscribed to this thread.



Message ID

---

## Suggested Actions
- [ ] Reply
- [ ] Archive
- [ ] Escalate
