CI / GitHub Actions Setup

Use a Litmus Lab Collection as a quality gate in your CI pipeline. The Collection page gives you everything you need — a ready-to-use trigger URL and a webhook secret. No IDs to look up, no URLs to construct.

How it works

The trigger endpoint supports a synchronous ?wait=N mode that blocks until the run completes and returns an HTTP status code you can act on directly:

Response codeMeaning
200Run completed — passed the collection's pass threshold
422Run completed — failed to meet the pass threshold
202Still running when the timeout elapsed

The environment to run against is configured on the Collection itself.

Step 1 — Copy your credentials from the Collection page

Open the Collection in Litmus Lab. In the Webhook section you'll find:

  • Trigger URL — the full endpoint to POST to (already includes your collection ID)
  • Webhook Secret — the X-Litmus-Secret value used to authenticate the request

Both have copy buttons. Use Regenerate if you ever need to rotate the secret.

Step 2 — Add secrets to GitHub

In your GitHub repo, go to Settings → Secrets and variables → Actions and add:

SecretValue
LITMUS_TRIGGER_URLThe trigger URL copied from the Collection page
LITMUS_WEBHOOK_SECRETThe webhook secret copied from the Collection page

Step 3 — Add the workflow

Create .github/workflows/litmus.yml:

name: AI Quality Gate

on:
  push:
    branches: [main, staging]
  pull_request:
    branches: [main]

jobs:
  litmus:
    name: Litmus Lab quality gate
    runs-on: ubuntu-latest

    steps:
      - name: Run collection
        run: |
          HTTP_CODE=$(curl -s -o /tmp/litmus.json -w "%{http_code}" -X POST \
            "${{ secrets.LITMUS_TRIGGER_URL }}?wait=120" \
            -H "X-Litmus-Secret: ${{ secrets.LITMUS_WEBHOOK_SECRET }}")

          echo "Response:"
          cat /tmp/litmus.json

          if [ "$HTTP_CODE" = "200" ]; then
            echo "Quality gate passed."
          elif [ "$HTTP_CODE" = "422" ]; then
            echo "Quality gate FAILED — collection did not meet pass threshold."
            exit 1
          elif [ "$HTTP_CODE" = "202" ]; then
            echo "Quality gate timed out — run still in progress after 120s."
            exit 1
          else
            echo "Unexpected response: $HTTP_CODE"
            exit 1
          fi

Step 4 — Verify

Push a commit or open a PR. The AI Quality Gate job will appear in your Actions tab. If the collection passes its threshold, the job succeeds. If it fails or times out, the build is blocked.

Tips

  • Set ?wait=N conservatively — collection runs with LLM evaluations typically take 30–90 seconds. ?wait=120 is a safe default; increase if your suites are large.
  • Target a staging environment — configure the collection to run against your staging environment, not production.
  • Tune the pass threshold — 100% can be brittle during active development. 90% may be more appropriate as a non-blocking signal on PRs.
  • Gate PRs, not just pushes — blocking PR merges gives developers feedback before code lands on main.
← CollectionsMonitors →