Get new OpenAI features like web_search tool working

[simonw]

Part of the new Responses API.

• https://openai.com/index/new-tools-for-building-agents/
• https://platform.openai.com/docs/guides/responses-vs-chat-completions
• https://platform.openai.com/docs/guides/tools-web-search?api-mode=responses

[simonw]

Incomplete not-working prototype:

diff --git a/llm/default_plugins/openai_models.py b/llm/default_plugins/openai_models.py
index ddd5676..5b78b78 100644
--- a/llm/default_plugins/openai_models.py
+++ b/llm/default_plugins/openai_models.py
@@ -325,6 +325,11 @@ class SharedOptions(llm.Options):
         default=None,
     )
 
+    web_search_preview: Optional[bool] = Field(
+        description="Enable web search preview tool",
+        default=None,
+    )
+
     @field_validator("logit_bias")
     def validate_logit_bias(cls, logit_bias):
         if logit_bias is None:
@@ -537,8 +542,9 @@ class _Shared:
                 "type": "json_schema",
                 "json_schema": {"name": "output", "schema": prompt.schema},
             }
-        if stream:
-            kwargs["stream_options"] = {"include_usage": True}
+        if prompt.options.web_search_preview:
+            kwargs.pop("web_search_preview", None)
+            kwargs["tools"] = [{"type": "web_search_preview"}]
         return kwargs
 
 
@@ -561,9 +567,9 @@ class Chat(_Shared, KeyModel):
         client = self.get_client(key)
         usage = None
         if stream:
-            completion = client.chat.completions.create(
+            completion = client.responses.create(
                 model=self.model_name or self.model_id,
-                messages=messages,
+                input=messages,
                 stream=True,
                 **kwargs,
             )
@@ -580,9 +586,9 @@ class Chat(_Shared, KeyModel):
                     yield content
             response.response_json = remove_dict_none_values(combine_chunks(chunks))
         else:
-            completion = client.chat.completions.create(
+            completion = client.responses.create(
                 model=self.model_name or self.model_id,
-                messages=messages,
+                input=messages,
                 stream=False,
                 **kwargs,
             )
@@ -614,9 +620,9 @@ class AsyncChat(_Shared, AsyncKeyModel):
         client = self.get_client(key, async_=True)
         usage = None
         if stream:
-            completion = await client.chat.completions.create(
+            completion = await client.responses.create(
                 model=self.model_name or self.model_id,
-                messages=messages,
+                input=messages,
                 stream=True,
                 **kwargs,
             )
@@ -633,9 +639,9 @@ class AsyncChat(_Shared, AsyncKeyModel):
                     yield content
             response.response_json = remove_dict_none_values(combine_chunks(chunks))
         else:
-            completion = await client.chat.completions.create(
+            completion = await client.responses.create(
                 model=self.model_name or self.model_id,
-                messages=messages,
+                input=messages,
                 stream=False,
                 **kwargs,
             )

[simonw]

That's interesting - you CAN use these features with the older chat completions API, but only with new custom model names gpt-4o-search-preview and gpt-4o-mini-search-preview:

https://platform.openai.com/docs/guides/tools-web-search?api-mode=chat&lang=curl

When using Chat Completions, the model always retrieves information from the web before responding to your query. To use web_search_preview as a tool that models like gpt-4o and gpt-4o-mini invoke only when necessary, switch to using the Responses API.

I tried it just now and it worked as advertised:

curl -X POST "https://api.openai.com/v1/chat/completions" \
    -H "Authorization: Bearer $OPENAI_API_KEY" \
    -H "Content-type: application/json" \
    -d '{
        "model": "gpt-4o-search-preview",
        "web_search_options": {},
        "messages": [{
            "role": "user",
            "content": "What was a positive news story from today?"
        }]
    }'

This worked for me just now:

{
  "id": "chatcmpl-3c3f6ad7-306c-4370-8c3b-45642bf61cae",
  "object": "chat.completion",
  "created": 1742340859,
  "model": "gpt-4o-search-preview-2025-03-11",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Here are some positive news stories from March 18, 2025:\n\n**Enterprise Ireland Opens Atlanta Office**\n\nEnterprise Ireland, the agency supporting Irish companies' global growth, officially unveiled its eighth U.S. office in Atlanta. This expansion reflects Atlanta's growing appeal to Irish firms and strengthens economic ties between Ireland and the southeastern United States. ([georgiatrend.com](https://www.georgiatrend.com/2025/03/18/georgia-trend-daily-march-18-2025/?utm_source=openai))\n\n**Special Olympics Michigan Athletes Shine at World Games**\n\nMichigan athletes returned from the Special Olympics World Games in Turin, Italy, with impressive achievements. Jeremiah Sumpfig of Houghton Lake earned gold and silver in cross-country skiing, while Christina Danieli from Grosse Pointe Farms secured two gold medals in alpine skiing. Their successes highlight the dedication and talent of Special Olympics participants. ([fox17online.com](https://www.fox17online.com/morning-mix/morning-buzz-march-18-2025?utm_source=openai))\n\n**Vienna Vegetable Orchestra Recognized by Guinness World Records**\n\nThe Vienna Vegetable Orchestra, known for crafting instruments from fresh produce, received official recognition from Guinness World Records. Performing for nearly three decades, the group uses about 70 pounds of vegetables per concert, creating unique musical experiences and promoting sustainability. ([wng.org](https://wng.org/podcasts/the-world-and-everything-in-it-march-18-2025-1742241586?utm_source=openai))\n\nThese stories showcase achievements in international business expansion, athletic excellence, and innovative artistry. ",
        "refusal": null,
        "annotations": [
          {
            "type": "url_citation",
            "url_citation": {
              "end_index": 492,
              "start_index": 378,
              "title": "Georgia Trend Daily - March 18, 2025 - Georgia Trend Magazine",
              "url": "https://www.georgiatrend.com/2025/03/18/georgia-trend-daily-march-18-2025/?utm_source=openai"
            }
          },
          {
            "type": "url_citation",
            "url_citation": {
              "end_index": 1031,
              "start_index": 926,
              "title": "Morning Buzz: March 18, 2025",
              "url": "https://www.fox17online.com/morning-mix/morning-buzz-march-18-2025?utm_source=openai"
            }
          },
          {
            "type": "url_citation",
            "url_citation": {
              "end_index": 1516,
              "start_index": 1405,
              "title": "",
              "url": "https://wng.org/podcasts/the-world-and-everything-in-it-march-18-2025-1742241586?utm_source=openai"
            }
          }
        ]
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 9,
    "completion_tokens": 331,
    "total_tokens": 340,
    "prompt_tokens_details": {
      "cached_tokens": 0,
      "audio_tokens": 0
    },
    "completion_tokens_details": {
      "reasoning_tokens": 0,
      "audio_tokens": 0,
      "accepted_prediction_tokens": 0,
      "rejected_prediction_tokens": 0
    }
  },
  "system_fingerprint": ""
}

This does seem like a very different feature though - the responses API one gets to do tool usage, which I imagine produces better results because it's not doing a single RAG-style context population before running the prompt.

[simonw]

Related:

• Annotations abstraction for responses that are not just a stream of plain text llm#716

Because OpenAI's docs say:

When displaying web results or information contained in web results to end users, inline citations must be made clearly visible and clickable in your user interface.

[simonw]

Pricing information: https://platform.openai.com/docs/pricing#web-search

[simonw]

Since gpt-4o-search-preview and gpt-4o-mini-search-preview are separate models I think it makes sense to support them directly as model selections (chat completions only). I'll file a new tickt for that.

[simonw]

Migrating this issue to the new plugin.

[simonw]

This work is blocked on annotations:

• Annotations abstraction for responses that are not just a stream of plain text llm#716

[simonw]

Pushed a prototype. I used it for this research here:

• llm mlx remove-model command  llm-mlx#14 (comment)

[simonw]

Prototype still needs me to figure out which models it works with (trial and error showed that o3-mini didn't).