Assertions

Assertions are checks that run against your API responses. Litmus Lab supports three categories: deterministic, qualitative (LLM-based), and scenario-aware.

Deterministic assertions

These always produce the same result for the same input — no LLM involved.

Type	What it checks
`status_code`	HTTP status matches an expected value
`response_time`	Response completes within a max millisecond threshold
`json_path`	A JSONPath expression on the response body matches an expected value
`contains`	Response body contains a specific string
`not_contains`	Response body does not contain a specific string
`schema`	Response body matches a JSON schema
`extract_match`	A previously extracted variable matches an expected value

on_failure

Any assertion can set on_failure:

continue (default) — record the failure, keep running
stop_request — skip remaining assertions for this step
stop_flow — stop the entire flow (all subsequent steps)

Qualitative assertions (LLM-based)

These use an LLM to evaluate the response and return a confidence score. A threshold (default 75%) determines pass/fail.

Type	What it checks
`llm_rubric`	Evaluates against a free-text rubric you write
`tone`	Checks that the response matches a specified tone
`factual`	Compares against a ground-truth statement
`no_hallucination`	Checks that the response doesn't contradict provided context
`answers_question`	Checks that the response adequately answers a given question

Scenario-aware assertions

These run when a context and scenario are attached. They have access to the scenario data as ground truth.

Type	What it checks
`consistent_with_scenario_data`	Response doesn't contradict the scenario data
`accurate_with_scenario_context`	Response aligns with an approved reference response
`no_hallucination_vs_scenario`	Response doesn't invent facts beyond the scenario data
`contextual_evaluation`	Evaluates against a rubric you write, with scenario data as context

contextual_evaluation is the most powerful — see Contextual Evaluation for a full guide.

Pass thresholds

LLM-based assertions have a configurable pass threshold (default 75%). The score returned by the evaluator must meet or exceed this threshold for the assertion to pass.

If the suite has an Intent set (on the Objective tab), it's included as context in qualitative and scenario-aware assertion prompts, so evaluation stays anchored to the suite's overall goal rather than judging each check in isolation.

Note that assertion (check) results don't directly determine a Flow run's pass/fail — that comes from the run's Intent Evaluation instead. The suite's pass threshold, set on the Objective tab, is the percentage of Flow runs whose Intent Evaluation must pass for the suite to be considered passing overall.