Support Pydantic output from tools

[skylarbpayne]

Had a brief interaction with @samuelcolvin on X about this: https://x.com/samuel_colvin/status/1864058372099580112

Let me know if I misunderstood, but this doesn't work the way I thought it would.

Minimal reproduction

from __future__ import annotations

import asyncio
import logging
from typing import Any

from pydantic import BaseModel
from pydantic_ai import Agent, RunContext

# Set up logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

class Location(BaseModel):
    latitude: float
    longitude: float


# Initialize agent with our complex model
agent_pydantic_output_tool = Agent("openai:gpt-4o-mini")
agent_dict_output_tool = Agent("openai:gpt-4o-mini")

@agent_pydantic_output_tool.tool
def get_location(ctx: RunContext[None], place_name: str) -> Location:
    return Location(latitude=36.53, longitude=-116.93)

@agent_dict_output_tool.tool
def get_location_dict(ctx: RunContext[None], place_name: str) -> dict[str, Any]:
    return {"latitude": 36.53, "longitude": -116.93}

async def main():
    """Run example that should trigger an error due to schema complexity."""
    prompt = "What is the latitude and longitude of Death Valley, California?"

    result_dict = await agent_dict_output_tool.run(prompt)
    print(f"Dict result: {result_dict.data}")

    result_pydantic = await agent_pydantic_output_tool.run(prompt)
    print(f"Pydantic result: {result_pydantic.data}")

if __name__ == "__main__":
    asyncio.run(main())

This raises an error from the tool use with agent_pydantic_output_tool.
However, the little decorator I wrote to map pydantic output to dict](https://gist.github.com/skylarbpayne/7d52269e1790e3f74076af71e68ff5c3) works (and doesn't have a type error; though I haven't carefully checked for type erasure in the solution)

Error

pydantic_core._pydantic_core.ValidationError: 5 validation errors for nullable[union[str,int,float,json-or-python[json=list[...],python=chain[is-instance[Sequence],function-wrap[sequence_validator()]]],dict[str,...]]]
str
  Input should be a valid string [type=string_type, input_value=Location(latitude=36.53, longitude=-116.93), input_type=Location]
    For further information visit https://errors.pydantic.dev/2.10/v/string_type
int
  Input should be a valid integer [type=int_type, input_value=Location(latitude=36.53, longitude=-116.93), input_type=Location]
    For further information visit https://errors.pydantic.dev/2.10/v/int_type
float
  Input should be a valid number [type=float_type, input_value=Location(latitude=36.53, longitude=-116.93), input_type=Location]
    For further information visit https://errors.pydantic.dev/2.10/v/float_type
`json-or-python[json=list[...],python=chain[is-instance[Sequence],function-wrap[sequence_validator()]]]`
  Input should be an instance of Sequence [type=is_instance_of, input_value=Location(latitude=36.53, longitude=-116.93), input_type=Location]
    For further information visit https://errors.pydantic.dev/2.10/v/is_instance_of
`dict[str,...]`
  Input should be a valid dictionary [type=dict_type, input_value=Location(latitude=36.53, longitude=-116.93), input_type=Location]
    For further information visit https://errors.pydantic.dev/2.10/v/dict_type

Environment

Device

16-inch, Nov 2023 Macbook Pro
Apple M3 Max
36 GB Memory
macOS Sonoma 14.6.1

Python

Python version: 3.10.14

uv pip freeze:

annotated-types==0.7.0
anyio==4.6.2.post1
asttokens==2.4.1
asynciolimiter==1.0.0
cachetools==5.5.0
certifi==2024.8.30
cfgv==3.4.0
chardet==5.2.0
charset-normalizer==3.4.0
colorama==0.4.6
decorator==5.1.1
deprecated==1.2.15
devtools==0.12.2
distlib==0.3.9
distro==1.9.0
eval-type-backport==0.2.0
exceptiongroup==1.2.2
executing==2.1.0
filelock==3.16.1
google-api-core==2.23.0
google-api-python-client==2.154.0
google-auth==2.36.0
google-auth-httplib2==0.2.0
google-auth-oauthlib==1.2.1
googleapis-common-protos==1.66.0
griffe==1.5.1
groq==0.13.0
h11==0.14.0
html2text==2024.2.26
httpcore==1.0.7
httplib2==0.22.0
httpx==0.28.0
identify==2.6.3
idna==3.10
importlib-metadata==8.5.0
iniconfig==2.0.0
ipdb==0.13.13
ipython==8.18.1
jedi==0.19.2
jiter==0.8.0
logfire==2.6.0
logfire-api==2.6.0
markdown-it-py==3.0.0
matplotlib-inline==0.1.7
mdurl==0.1.2
mypy==1.13.0
mypy-extensions==1.0.0
nodeenv==1.9.1
oauthlib==3.2.2
openai==1.56.0
opentelemetry-api==1.28.2
opentelemetry-exporter-otlp-proto-common==1.28.2
opentelemetry-exporter-otlp-proto-http==1.28.2
opentelemetry-instrumentation==0.49b2
opentelemetry-proto==1.28.2
opentelemetry-sdk==1.28.2
opentelemetry-semantic-conventions==0.49b2
packaging==24.2
parso==0.8.4
pexpect==4.9.0
platformdirs==4.3.6
pluggy==1.5.0
pre-commit==4.0.1
prompt-toolkit==3.0.48
proto-plus==1.25.0
protobuf==5.29.0
ptyprocess==0.7.0
pure-eval==0.2.3
pyasn1==0.6.1
pyasn1-modules==0.4.1
-e file:///Users/skylarbpayne/projects/pydai_agent_hw/pydai-hw
pydantic==2.10.2
pydantic-ai==0.0.8
pydantic-ai-slim==0.0.8
pydantic-core==2.27.1
pygments==2.18.0
pyparsing==3.2.0
pyproject-api==1.8.0
pytest==8.3.4
python-dotenv==1.0.1
pyyaml==6.0.2
requests==2.32.3
requests-oauthlib==2.0.0
rich==13.9.4
rsa==4.9
ruff==0.8.1
six==1.16.0
sniffio==1.3.1
stack-data==0.6.3
tomli==2.2.1
tox==4.23.2
tox-uv==1.16.0
tqdm==4.67.1
traitlets==5.14.3
typing-extensions==4.12.2
uritemplate==4.1.1
urllib3==2.2.3
uv==0.5.5
virtualenv==20.28.0
wcwidth==0.2.13
wrapt==1.17.0
zipp==3.21.0

[skylarbpayne]

If I'm not mistaken, I think the issue is here: https://github.com/pydantic/pydantic-ai/blob/main/pydantic_ai_slim/pydantic_ai/messages.py#L75

(excerpt)

def model_response_str(self) -> str:
        if isinstance(self.content, str):
            return self.content
        else:
            content = json_ta.validate_python(self.content) # <----- This line
            return json_ta.dump_json(content).decode()

json_ta.validate_python seems to expect a python object... but a python representation of a JSON structure (eg dict).

For example, if we create a minimal repro of this code:

from typing import Mapping, Sequence, Union

from pydantic import TypeAdapter
from typing_extensions import TypeAliasType

from pydantic import BaseModel

JsonData = TypeAliasType('JsonData', 'Union[str, int, float, None, Sequence[JsonData], Mapping[str, JsonData]]')
json_ta: TypeAdapter[JsonData] = TypeAdapter(JsonData)

class SimpleModel(BaseModel):
    name: str
    age: int

simple_model = SimpleModel(name="John", age=30)

json_ta.validate_python(simple_model) # error like the one in the original issue
json_ta.validate_python(json_ta.dump_python(simple_model)) # works

So the modification would be something like:

def model_response_str(self) -> str:
        if isinstance(self.content, str):
            return self.content
        else:
            content = json_ta.validate_python(json_ta.dump_python(self.content))
            return json_ta.dump_json(content).decode()

Or perhaps it's more desirable to update the implementation of validate_python?
Also I am unsure of whether the other use of validate_python requires a similar change.

[samuelcolvin]

Thanks.

I agree, should be easy to fix.

[samuelcolvin]

Should be fixed via #136.