-
Notifications
You must be signed in to change notification settings - Fork 13.5k
Open
Labels
Description
Describe the bug
In the sample code presented in https://github.com/Azure-Samples/azureai-samples/blob/main/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Tool_Call_Accuracy.ipynb., The type is missing from the tool_definition. And it seems that only 'function' type tool can be evaluated by this evaluator.
To Reproduce
Steps to reproduce the behavior:
- Clone 'https://github.com/Azure-Samples/azureai-samples/blob/main/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Tool_Call_Accuracy.ipynb'
- Run the first 2 code cells.
- See the evaluation result stating 'not applicable'.
- Adding "type": "function", to tool_definition in cell added the lesson folders and READMEs #2 and rerun, it gives valid evaluation result.
Expected behavior
Please update the sample code. Also, any tool_definition related sample code might need to be updated with tool type definition. I saw it break the AIAgentConverter because of missing type.