Litmus
Test your AI like you test code.
In the constellation
Litmus highlighted in the live map — hover or tap any node to explore.
The gap
AI features ship without the safety nets every other kind of software takes for granted. No reproducible test suite, no regression gate, no way to see quality trending down before it breaks in front of a customer. You change a prompt or swap a model and learn the cost in production.
Code has tests, CI, and a build that fails when something regresses. AI quality, for most teams, is managed by hope.
What Litmus is
Litmus brings the discipline of code testing to AI. It runs evaluation suites, replays fixtures so every run is reproducible, scores model fitness, grades open-ended output against a rubric, and gates regressions in your pipeline — failing the build when quality drops.
Because runs are reproducible and tracked over time, slow drift becomes visible long before it bites. Shipping an AI change stops being a leap of faith: every change is measured against a baseline, and the ones that regress get blocked.
What it does
Eval suites for AI
Author and run test suites against any AI feature, the way you’d write tests for code.
Reproducible runs
Replay recorded fixtures so every evaluation is deterministic — the same inputs always give you the same comparison.
Regression gates in CI
Wire quality into your pipeline so a build fails when an AI change makes output measurably worse.
Fitness over time
Track quality trends across releases and catch slow degradation before it becomes an incident.
The line between judgment and machinery
AccelMars draws one hard line through every product: what an AI decides, and what runs deterministically. Litmus sits on the boundary — and keeps the two honest.
What the AI judges
- Whether an open-ended output is actually good
- How a response scores against a rubric
- Which quality changes are meaningful
What stays deterministic
- Fixture replay for reproducible runs
- Scoring and pass/fail gating
- Trend tracking across every release
The rubric and AI judge are calibrated against human judgment once, up front — so the continuous scoring that follows is both trustworthy and cheap to run on every build.
Who it’s for
- Product teams shipping AI features they need to trust
- AI and ML teams that want defensible quality gates
- Release engineers putting AI behind a CI pipeline
Planned — designed, not yet built. It builds on the model-evaluation engine and the record-and-replay layer that already gives AccelMars reproducible AI runs.