Skip to content

Conversation

Wh1isper
Copy link
Contributor

@Wh1isper Wh1isper commented Jun 24, 2025

closes #2034

This PR allows us to implement more complex tool return like computer use

Example response:

  • content: Screenshots, including before and after
  • tool_return: some console log

previous: #1517

Copy link
Contributor

hyperlint-ai bot commented Jun 24, 2025

PR Change Summary

Added support for customizable multi-modal content presentation using the new MultiModalContent feature.

  • Introduced MultiModalContent for enhanced content customization.
  • Provided an example of using MultiModalContent with a custom image tool.
  • Explained the automatic replacement of placeholders in prompts.

Modified Files

  • docs/tools.md

How can I customize these reviews?

Check out the Hyperlint AI Reviewer docs for more information on how to customize the review.

If you just want to ignore it on this PR, you can add the hyperlint-ignore label to the PR. Future changes won't trigger a Hyperlint review.

Note specifically for link checks, we only check the first 30 links in a file and we cache the results for several hours (for instance, if you just added a page, you might experience this). Our recommendation is to add hyperlint-ignore to the PR to ignore the link check for this PR.

@Wh1isper Wh1isper force-pushed the feat-extend-multi-modal-content branch from 4f6d660 to 3b14091 Compare June 24, 2025 12:03
@Wh1isper Wh1isper marked this pull request as ready for review June 24, 2025 12:42
@Wh1isper Wh1isper changed the title feat: add MultiModalContent for customizable multi-modal tool feat: add MultiModalToolResponse for customizable multi-modal tool Jun 24, 2025
@DouweM DouweM self-assigned this Jun 24, 2025
@Wh1isper Wh1isper changed the title feat: add MultiModalToolResponse for customizable multi-modal tool feat: add ToolReturn for customizable tool return Jun 25, 2025
@Wh1isper
Copy link
Contributor Author

@DouweM I did some refactoring and now I use the ToolReturn type for multimodal and artifact requirements.

Copy link

@thisisarko thisisarko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@Wh1isper Wh1isper requested a review from DouweM June 26, 2025 16:06
@DouweM DouweM changed the title feat: add ToolReturn for customizable tool return Let tools return ToolReturn to pass additional content to model, or attach metadata that's not passed to the model Jun 27, 2025
@DouweM DouweM merged commit b650238 into pydantic:main Jun 27, 2025
19 checks passed
@DouweM
Copy link
Collaborator

DouweM commented Jun 27, 2025

@Wh1isper Thanks a ton!

@nathan-gage
Copy link

@Wh1isper is there a specific reason return_value/metadata are Any typed rather than Generic?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

How to store 'artifacts' with tool responses?
4 participants