Assistant Responds Before Function Call Completes

[TheoGibbons]

We are encountering an issue with function calls where the assistant responds prematurely, before the function has a chance to complete its execution and return the correct data.

This behavior occurs:

1. Locally, when running the repository from this GitHub repo.
2. On our own staging servers.
3. Also, here is an example of the bug on the "Developers Digest" YouTube channel showcasing this repo: Video link at 4:25 mark.

Example scenario:

1. I ask:
   "What's the weather in Seattle?"
2. The assistant responds prematurely:
   "It seems I'm having a little trouble fetching the weather right now. How about I try something else for you? Is there anything else I can help you with?"
   However, the function call is triggered correctly and returns the weather.
3. I ask again:
   "What's the weather in Seattle?"
4. This time, the assistant correctly responds with the weather.

Additional Information:
We are using the latest version of the repository code and have observed this behavior consistently.

Is there any way around this issue? It would be helpful if there were a way to configure something like functionTimeout: 10 seconds, which would allow more time for the function call to complete before the assistant responds.

One potential workaround could be to ignore all responses from OpenAI until the function call finishes, and then manually trigger a response.create event afterward. However, this approach feels a bit hacky. Is there a more reliable or recommended solution to handle this?

[stefanskiasan]

I Have the same Issue. I call a API Request which need 5 to 10 seconds after 1 seconde i get the message He dont can use Internal .... I debug this in the Libary:

this.realtime.on('server.response.output_item.done', async (event) => { const { item } = handlerWithDispatch(event); if (item.status === 'completed') { this.dispatch('conversation.item.completed', { item }); } if (item.formatted.tool) { callTool(item.formatted.tool); } });

This Event is calling for execute the function and after 1 second later again.. But with the todo to complete the message.

I think is Important to make the Timout higher. and Maybe to give the opportunity to config this. Thanks @TheoGibbons you are opend this ticket. Lets hope this will will fixed very fast. Or we get a workaround. Thanks guys for your support.

[meng-x]

Got same issue here when using relay websocket server, here is the log:

user
How's the weather in Tokyo?
function call
get_weather({"lat":35.6895,"lng":139.6917,"location":"Tokyo"})
function call output
{"error":"Tool \"get_weather\" has not been added"}
assistant
Oh, it looks like I couldn't retrieve the weather information for Tokyo right now. Is there anything else I can help you with?
function call output
{"latitude":35.7,"longitude":139.6875,"generationtime_ms":0.028967857360839844,"utc_offset_seconds":0,"timezone":"GMT","timezone_abbreviation":"GMT","elevation":40,"current_units":{"time":"iso8601","interval":"seconds","temperature_2m":"°C","wind_speed_10m":"km/h"},"current":{"time":"2024-10-21T04:30","interval":900,"temperature_2m":18.7,"wind_speed_10m":6.1}}

interestingly, it has "function call output" event twice, the first one comes earlier than the response, said "get_weather not registered", then the second comes later.

Then I doubt that maybe I should register the function call on relay server side RealtimeClient too, basically add below code to the relay.js:

client.addTool(
      {
        name: 'get_weather',
        description:
          'Retrieves the weather for a given lat, lng coordinate pair. Specify a label for the location.',
        parameters: {
          type: 'object',
          properties: {
            lat: {
              type: 'number',
              description: 'Latitude',
            },
            lng: {
              type: 'number',
              description: 'Longitude',
            },
            location: {
              type: 'string',
              description: 'Name of the location',
            },
          },
          required: ['lat', 'lng', 'location'],
        },
      },
      async ({ lat, lng, location }) => {
        const result = await fetch(
          `https://api.open-meteo.com/v1/forecast?latitude=${lat}&longitude=${lng}&current=temperature_2m,wind_speed_10m`
        );
        const json = await result.json();
        return { ...json, origin: 'relay' };
      }
    );

I tried that then it worked everytime. like log below:

user
How's the weather like in Tokyo?
function call
get_weather({"lat":35.6895,"lng":139.6917,"location":"Tokyo"})
function call output
{"origin":"relay","latitude":35.7,"longitude":139.6875,"generationtime_ms":0.04398822784423828,"utc_offset_seconds":0,"timezone":"GMT","timezone_abbreviation":"GMT","elevation":40,"current_units":{"time":"iso8601","interval":"seconds","temperature_2m":"°C","wind_speed_10m":"km/h"},"current":{"time":"2024-10-21T05:00","interval":900,"temperature_2m":19.6,"wind_speed_10m":6.4}}
function call output
{"latitude":35.7,"longitude":139.6875,"generationtime_ms":0.026941299438476562,"utc_offset_seconds":0,"timezone":"GMT","timezone_abbreviation":"GMT","elevation":40,"current_units":{"time":"iso8601","interval":"seconds","temperature_2m":"°C","wind_speed_10m":"km/h"},"current":{"time":"2024-10-21T05:00","interval":900,"temperature_2m":19.6,"wind_speed_10m":6.4}}
assistant
In Tokyo, the temperature is currently 19.6°C with a wind speed of 6.4 km/h. How's that for a lovely day?

again, got 2 "function call output" event printed, 1 from relay server RealtimeClient (used for create response), 1 from client side RealtimeClient (used for updating UI) . you can differ the 2 with '"origin":"relay"'.

so as a conclusion, if using relay server, we need to register the function call on the relay server side RealtimeClient, and if need to update UI (like the sample code updating map pins and stuff), we register the same function call on client side RealtimeClient.

[meng-x]

I also tried remove the client.addTool from the frontend side RealtimeClient, and it never worked.

so it seems that:
1, the frontend side RealtimeClient will handle the function call registry
2, both side RealtimeClients handle the callback, however
3, only the callback in relay server side RealtimeClient will be used for generating the response

hopefully above issue and fix is temporary and openai can resolve this in the new version of '@openai/realtime-api-beta' library.

[devanarchitect]

A Solution using a patch was posted for this on YouTube @fullstackdesk

…

On Mon, Oct 21, 2024 at 12:51 AM Joe M ***@***.***> wrote: hopefully above issue and fix is temporary and openai can resolve this in the new version of ***@***.***/realtime-api-beta' library. — Reply to this email directly, view it on GitHub <#49 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BHADPMOAXSK4SYFM4AXN3GTZ4SB3NAVCNFSM6AAAAABQGVXKUCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRVGU3TSNZWGA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[meng-x]

#14 (comment)

A Solution using a patch was posted for this on YouTube @fullstackdesk
…
On Mon, Oct 21, 2024 at 12:51 AM Joe M @.> wrote: hopefully above issue and fix is temporary and openai can resolve this in the new version of @./realtime-api-beta' library. — Reply to this email directly, view it on GitHub <#49 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/BHADPMOAXSK4SYFM4AXN3GTZ4SB3NAVCNFSM6AAAAABQGVXKUCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRVGU3TSNZWGA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[TheoGibbons]

Thanks all I've created a pull request with the change.
In the mean time you can update your package.json file to point to the fork which should work around the issue.

"@openai/realtime-api-beta": "github:TheoGibbons/openai-realtime-api-beta#bd4e74aaa20d09a069f058eb6aaa76205871f28a",