Skip to main content

Webhook Evaluators

Sometimes, custom evaluators on Agenta may not be sufficient for your specific use case, or you may wish to host your own evaluators. In such cases, you can create a webhook evaluator to suit your specific needs. Webhook evaluators only need to comply with a predefined API.

info

For the moment, there are limitation on the security measures related to webhook evaluators. There is no authentication and no authorization, and the endpoint must be publicly available.

Webhook API​

The input parameters are as follows:

  1. inputs: A dictionary containing the inputs to the webhook evaluators.
  2. output: The output generated by the LLM app.
  3. correct_answer: The expected correct answer for the LLM app.

The webhook request body has the following signature:

{
    "inputs": { "{input_1}": <String> [,"{input_2}": <String>, ...] }
    "output": <String>,
    "correct_answer": <String>
}

The webhook should return the following:

score: A float value corresponding to the score of the evaluation. The score should be between 0 and 1, with 0 meaning the evaluation failed, with its lowest score, and 1 meaning the evaluation succeeded, with its highest score.

The webhook response body should have this signature:

{
    "score": <Float>
}

The webhook response status must be 200 for successful evaluations, the webhook response Content-Type header should be set to application/json, and the webhook response body must be a properly-formatted JSON object.

Example Implementation​

Here's a simple webhook evaluator implementation that returns 1.0 when the lowercase output matches the lowercase correct answer. You can use this webhook for testing by copying the endpoint. Then fork and modify it for your specific use case: