Getting Started with Litmus Lab
Litmus Lab is an AI testing platform that lets you define test suites, run them against live endpoints, and evaluate responses — including with LLM-based qualitative assertions.
Core concepts
| Concept | What it is |
|---|---|
| Project | A workspace containing tests, environments, contexts, and collections |
| Test suite | A named set of steps — API Workflow (HTTP requests) or UI Workflow (browser actions) — with checks |
| Intent | The overarching goal of a suite's Flow, set on the Objective tab and evaluated holistically after each run |
| Dataset row | One "test case" — a set of variables run through the suite |
| Scenario | A persona or situation (e.g. a specific user) applied during evaluation |
| Context | A schema defining what scenario data looks like |
| Collection | A batch run of multiple test suites against an environment |
| Environment | A named set of variables (e.g. HOSTNAME) for a target environment |
Your first test
1. Create a project
From the dashboard, click New Project. Give it a name that reflects the API or product you're testing.
2. Add an environment
In your project, go to Environments and create one. At minimum, add a HOSTNAME variable pointing to your API (e.g. https://api.example.com).
3. Create a test suite
Click New Test Suite. Give it a name and choose a suite type — API Workflow or UI Workflow. This can't be changed after creation, since the two differ fundamentally in what a step is.
On the Objective tab, optionally add a description and an Intent — the goal a real user should be able to accomplish, e.g. "A first-time visitor should be able to understand what the product does and request early access without confusion, missing information, or dead ends." See Objective for how this drives evaluation.
4. Build the flow
Switch to the Flow tab and add steps:
- Set the URL using
{{HOSTNAME}}/your/endpoint - Choose the HTTP method and add a body if needed
- Add Checks — start with
status_code: 200 - Use Extract to capture values from responses for use in later steps
5. Add test cases
In the Test Cases tab, add dataset rows. Each row is one run of the entire flow, with its own variable values.
6. Run it
Click Run to execute the suite against your chosen environment. Results appear in the Results panel — each run's pass/fail comes from its Intent Evaluation, not a raw percentage of checks passing.
Next steps
- Objective — suite types and how Intent Evaluation determines pass/fail
- UI Workflow Steps — navigation, headers, and browser isolation for browser-driven suites
- Contexts & Scenarios — test against multiple user personas
- Assertions — deterministic and LLM-based checks
- Collections & CI — run multiple suites together and gate CI