Contextual Evaluation

contextual_evaluation is a scenario-aware assertion that evaluates your AI's response against a rubric you write, using the scenario data as ground truth. It's the primary tool for testing whether a personalized AI response is actually good for a specific user.

How it works

For each scenario, the evaluator receives:

  1. Your rubric (with variables resolved from the scenario data)
  2. The context field definitions
  3. The scenario's actual data
  4. The value being evaluated (typically an extracted field from the response)

It returns a score (0.0–1.0) and reasoning. The score is compared against your pass threshold (default 75%).

Scoring bands

ScoreMeaning
0.9–1.0Meets all criteria fully with no notable gaps
0.75–0.89Meets most criteria; minor issues or weak justifications
0.5–0.74Partial compliance; notable gaps or weak reasoning
0.0–0.49Fails key criteria or contains clear factual violations

Writing a rubric

The rubric is a free-text description of what a good response looks like. It supports variable interpolation using {{variable_name}} — any variable defined in your dataset or resolved from context mapping is available.

Example rubric

Evaluate this list of movie recommendations.

CONSTRAINTS:
1. Must not recommend any title the user has already watched: {{already_watched}}
2. For friend-sourced recommendations, the friend name must be real and present in the scenario data

QUALITY CRITERIA:
- Recommendations should match the user's taste profile
- Each recommendation should include a clear reason connecting to this specific user's history
- Generic reasoning ("you might enjoy this") without referencing the user's actual data is a quality failure

Using reference responses

For question-specific criteria, use {{reference_response}} in your rubric. Set a reference response per dataset row to define what a good answer looks like for that specific question. The evaluator uses it as a benchmark.

QUESTION-SPECIFIC CRITERIA:
{{reference_response}}

Setting the target

By default, contextual_evaluation evaluates the full response body. Use target to evaluate a specific extracted variable instead — useful when your response is structured (e.g. a JSON array of recommendations in {{RECOMMENDATIONS}}).

Tips for effective rubrics

  • Be specific about what "good" means — "relevant recommendations" is vague; "at least 2 of 3 should be sourced from friend activity" is testable
  • Distinguish hard constraints from quality criteria — the evaluator can weight violations differently based on how you phrase them
  • Reference scenario variables — rubrics that reference {{already_watched}} or {{friend_activity}} produce much more accurate evaluations than generic criteria
  • Use reference responses for question-specific guidance — rubrics for open-ended questions benefit from a concrete example of what a good answer includes
← AssertionsContext Mapping →