Skip to content

The type is missing from the tool_definition in the sample codes #235

@ktsasworkspace

Description

@ktsasworkspace

Describe the bug
In the sample code presented in https://github.com/Azure-Samples/azureai-samples/blob/main/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Tool_Call_Accuracy.ipynb., The type is missing from the tool_definition. And it seems that only 'function' type tool can be evaluated by this evaluator.

To Reproduce
Steps to reproduce the behavior:

  1. Clone 'https://github.com/Azure-Samples/azureai-samples/blob/main/scenarios/evaluate/Supported_Evaluation_Metrics/Agent_Evaluation/AI_Judge_Evaluator_Tool_Call_Accuracy.ipynb'
  2. Run the first 2 code cells.
  3. See the evaluation result stating 'not applicable'.
  4. Adding "type": "function", to tool_definition in cell added the lesson folders and READMEs #2 and rerun, it gives valid evaluation result.

Expected behavior
Please update the sample code. Also, any tool_definition related sample code might need to be updated with tool type definition. I saw it break the AIAgentConverter because of missing type.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions