Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 68 additions & 0 deletions sdk/ai/Azure.AI.Projects/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ The client library uses version `v1` of the AI Foundry [data plane REST APIs](ht
- [Using uploaded datasets](#using-uploaded-datasets)
- [Using custom prompt-based evaluator](#using-custom-prompt-based-evaluator)
- [Using custom code-based evaluator](#using-custom-code-based-evaluator)
- [Evaluating responses](#evaluating-responses)
- [Troubleshooting](#troubleshooting)
- [Next steps](#next-steps)
- [Contributing](#contributing)
Expand Down Expand Up @@ -1053,6 +1054,73 @@ private EvaluatorVersion GetCodeEvaluatorVersion()

The code-based evaluator can be used the same way as prompt-based

#### Evaluating responses

The evaluation may be done on the OpenAI response items, received from the Agent. To use this data structure,
the data source configuration `scenario` has to be set to "responses".

```C# Snippet:Sample_CreateData_EvaluationsAgent
private static BinaryData GetEvaluationConfig(string modelDeploymentName)
{
object[] testingCriteria = [
new {
type = "azure_ai_evaluator",
name = "violence_detection",
evaluator_name = "builtin.violence",
},
];
object dataSourceConfig = new
{
type = "azure_ai_source",
scenario = "responses"
};
return BinaryData.FromObjectAsJson(
new
{
name = "Agent Response Evaluation",
data_source_config = dataSourceConfig,
testing_criteria = testingCriteria
}
);
}
```

The data source needs to have section `item_generation_params`, having `response_retrieval` type.
This section informs service to get the data from the response with the given ID.

```C# Snippet:Sample_CreateDataSource_EvaluationsAgent
private static BinaryData GetRunData(string agentName, string responseId, string evaluationId)
{
object dataSource = new
{
type = "azure_ai_responses",
item_generation_params = new {
type = "response_retrieval",
data_mapping = new { response_id = "{{item.resp_id}}" },
source = new
{
type = "file_content",
content = new[]
{
new
{
item = new { resp_id = responseId}
}
}
}
},
};
return BinaryData.FromObjectAsJson(
new
{
eval_id = evaluationId,
name = $"Evaluation Run for Agent {agentName}",
data_source = dataSource
}
);
}
```

## Troubleshooting

Any operation that fails will throw a [RequestFailedException][RequestFailedException]. The exception's `code` will hold the HTTP response status code. The exception's `message` contains a detailed message that may be helpful in diagnosing the issue:
Expand Down
Loading
Loading