Collections

A Collection is a batch run of multiple test suites against a single environment. Collections are the primary way to run your full test coverage in one shot — and to gate CI/CD pipelines on test results.

Creating a Collection

  1. In your project sidebar, click New Collection
  2. Give it a name (e.g. "Full Regression" or "Pre-deploy Check")
  3. Add the test suites you want to include
  4. Set a pass threshold — the percentage of suites that must pass for the collection run to be considered passing (default: 100%)

Running a Collection

Select an environment and click Run. Litmus Lab runs all included suites in parallel. Results appear in the Runs panel as they complete.

Each run shows:

  • Overall pass/fail
  • Per-suite results
  • Drill-down into individual test case and step results

Pass threshold

The collection pass threshold controls when the overall run is considered passing. For example, at 80%, a collection with 10 suites passes if 8 or more pass.

Each suite also has its own pass threshold (set on the Objective tab, and evaluated via each run's Intent Evaluation) — both must be met.

Using Collections for CI/CD

Collections expose a run API that returns a pass/fail result, making them easy to integrate into CI pipelines as a quality gate.

See CI / GitHub Actions Setup for a complete setup guide.

← JMESPathCI / GitHub Actions →