LitmusLab

In the Lab

Research, field notes, and lab reports from the teams building with AI. What we're testing, what we're finding, and what it means.

What's New

What's New — October 16, 2026

Sign in to LitmusLab with your Google account, alongside your existing password rather than instead of it. Plus org admins can now reset a teammate's password themselves instead of waiting on support, self-service password changes from your avatar menu, unified and tightened monitor listings across the dashboard/API/MCP, and a sortable Sessions table.

The LitmusLab Team

The LitmusLab Team

October 16, 2026

3 min read
Read article
Research

MCP Was Not Designed for This: What a Structural Security Flaw Reveals About Agent-to-Agent Trust

A security researcher found the same server-side request forgery flaw inside Google, JPMorgan Chase, and a national government's MCP servers, independently of each other. The bug itself is patchable. What isn't: a protocol built for a human calling an agent, now running agent-to-agent, with write access as the default.

The LitmusLab Team

The LitmusLab Team

October 13, 2026

5 min read
Read article
What's New

What's New — October 9, 2026

LitmusLab now has a public, documented API — trigger runs and fetch results programmatically with an Org API key — and you can connect Claude directly to your projects as an MCP server, scoped to your own login rather than your whole org. Plus a holistic, Intent-based pass/fail for API Monitoring, and a batch of API Monitoring UX fixes: deduplicated Overview/Sessions tabs, a clearer breakdown for unmatched captures, and tab names that now match UI Monitoring's.

The LitmusLab Team

The LitmusLab Team

October 9, 2026

4 min read
Read article
What's New

What's New — October 2, 2026

API Monitors can now correlate with UI Monitor sessions and continue into a second real API call — verify a live chat answer against the document it actually cited, using real production traffic instead of a scripted test. Plus a fix for Monitor Constants not applying to real API calls, header-based extraction in test suite steps, and a Monitor results column that never showed anything useful.

The LitmusLab Team

The LitmusLab Team

October 2, 2026

4 min read
Read article
What's New

What's New — September 25, 2026

Vision checks can now find and scroll through content anywhere on the page, not just what's in the initial viewport. Plus Re-run that replaces your current results instead of piling up new ones, a heads-up when your session expires or a new version is available, and a fix for UI Workflow steps that could get stuck repeating an already-completed action.

The LitmusLab Team

The LitmusLab Team

September 25, 2026

4 min read
Read article
What's New

What's New — September 18, 2026

Manage project variables from a dedicated drawer with autocomplete and inline resolved status. Plus copy and paste steps between suites and projects, scroll support for UI Workflow steps, Test Builder remembering your run and tab in the URL, and fixes for empty-dataset results, unlabeled empty runs, and random-choice steps re-deciding mid-step.

The LitmusLab Team

The LitmusLab Team

September 18, 2026

4 min read
Read article
What's New

What's New — September 11, 2026

A project can now run multiple UI Monitors, each scoped to its own part of your site with independent test suites and settings. Plus configurable Privacy Controls for session capture, a client API for controlling capture from your own code, automatic exclusion of your own test traffic, and sharper, more trustworthy Graph tab and check results.

The LitmusLab Team

The LitmusLab Team

September 11, 2026

4 min read
Read article
What's New

What's New — September 4, 2026

Live sessions that match a Flow step now get checked against your existing test suite assertions automatically, using what actually happened in production. Plus faster, more accurate matching, clearer Graph tab labels and stability status, and Reset Test & Monitoring History in Project Settings.

The LitmusLab Team

The LitmusLab Team

September 4, 2026

3 min read
Read article
What's New

What's New — August 28, 2026

UI Monitoring lands: Production Monitoring now covers UI Workflows, with real session capture, automatic Flow coverage matching, and Uncovered Activity surfacing untested states. Plus alternate viewport sizes for UI Workflow steps.

The LitmusLab Team

The LitmusLab Team

August 28, 2026

3 min read
Read article
Research

The Missing Piece in Harness Engineering: The Behavioral Harness

Harness engineering has mapped the maintainability and architecture layers with impressive speed. But the most credible voices in the discipline — Böckeler, the CAAF paper, RepoRescue — all point to the same unsolved piece: the behavioral harness.

The LitmusLab Team

The LitmusLab Team

August 25, 2026

6 min read
Read article
What's New

What's New — August 21, 2026

Collection automation — schedule runs to happen automatically instead of triggering them by hand. Plus pass-rate trend charts, run completion emails, a Stop button for in-progress runs, and faster concurrent execution.

The LitmusLab Team

The LitmusLab Team

August 21, 2026

3 min read
Read article
What's New

What's New — August 14, 2026

UI Workflows land — test through the browser with plain-language steps and vision-based checks. Plus the new Objective tab with Intent Evaluation, Excel results export, and a landing page refresh.

The LitmusLab Team

The LitmusLab Team

August 14, 2026

3 min read
Read article
Research

When AI Writes the Code and the Tests, What Does Passing Actually Mean?

Code throughput is up 59% year-over-year. Main-branch success rates fell to a five-year low of 70.8%. If AI is writing the code and writing the tests, what does a passing test suite actually tell you?

The LitmusLab Team

The LitmusLab Team

August 12, 2026

5 min read
Read article
What's New

What's New — August 7, 2026

Production Monitoring lands — evaluate live traffic against your Test Suite checks with sampling, dedup, and per-step bypass modes. Plus mobile nav on the public site and a new logo.

The LitmusLab Team

The LitmusLab Team

August 7, 2026

3 min read
Read article
Research

Why AI Quality Is Hard to Test Even When You're Trying

More AI features ship faster. The review cycles that used to catch things are getting compressed. Here are the four underlying problems that make testing AI quality hard — and why most teams are stuck even when they're trying.

The LitmusLab Team

The LitmusLab Team

August 5, 2026

5 min read
Read article
What's New

What's New — July 31, 2026

Project-level YAML import/export, Orgs for shared billing and LLM config, invite activation for new members, and a last run timestamp on every test.

The LitmusLab Team

The LitmusLab Team

July 31, 2026

3 min read
Read article
Research

The Speed Trap: Why AI's Biggest Strength Is Creating Its Biggest Quality Problem

88% of agentic AI early adopters are seeing positive ROI. But the same report quietly lists "verify quality" as a core human responsibility — without explaining how anyone is supposed to do that at AI speed.

The LitmusLab Team

The LitmusLab Team

July 28, 2026

5 min read
Read article
What's New

What's New — July 24, 2026

Usage tab in project settings, a free AI Quality Assessment tool (no account needed), and public site improvements.

The LitmusLab Team

The LitmusLab Team

July 24, 2026

2 min read
Read article
Lab Notes

We Built Screened to Prove LitmusLab Isn't Just Theoretical

We built a real AI application — a social movie recommendation engine — and ran it through three levels of testing. This is what we found, what we'd do differently, and what it means for teams shipping AI features.

The LitmusLab Team

The LitmusLab Team

July 23, 2026

5 min read
Read article
What's New

What's New — July 17, 2026

Scenario Import lands — bring in test data from JSON or CSV. Plus GPT-4o as the default evaluator, in-app help, and automatic scenario sync.

The LitmusLab Team

The LitmusLab Team

July 17, 2026

3 min read
Read article
What's New

What's New — July 10, 2026

Scoring accuracy fixes, scenario field cleanup, and evaluation prompt corrections — all affecting the accuracy of results you're already seeing.

The LitmusLab Team

The LitmusLab Team

July 10, 2026

2 min read
Read article
What's New

What's New — July 3, 2026

A big week: Response Inspector, JMESPath for context mapping, live run progress, stop-on-failure, dashboard trend charts, context mapping preview, and data type support.

The LitmusLab Team

The LitmusLab Team

July 3, 2026

4 min read
Read article
What's New

What's New — June 26, 2026

Contexts & Scenarios land — the most significant change to how LitmusLab works. Define your data structure, populate scenarios, and run every test case against all of them automatically.

The LitmusLab Team

The LitmusLab Team

June 26, 2026

3 min read
Read article
What's New

What's New — June 19, 2026

LitmusLab is live. Delete and duplicate tests, variable substitution in connections, and scrollable results.

The LitmusLab Team

The LitmusLab Team

June 19, 2026

2 min read
Read article