Building a Regression Test Suite for Claude Prompts With promptfoo: A Working Config

A regression test suite for a Claude prompt catches the thing manual spot-checking misses: a prompt that worked fine last week silently breaking after you tweak one sentence, swap a model version, or add a new few-shot example. promptfoo runs the same prompt against the same test cases every time, scores the output against assertions you define, and fails the build when a change makes a known-good case fail — the same discipline a code test suite applies, pointed at a prompt instead of a function.

Step 1: Install promptfoo and scaffold a config

npm install -g promptfoo
export ANTHROPIC_API_KEY=your_api_key_here

promptfoo’s own docs set the API key through the standard ANTHROPIC_API_KEY environment variable, or through an apiKey field inside a provider’s config block if you’d rather keep it out of your shell. Either way, nothing here needs the model itself to see the key — it’s read by promptfoo’s CLI, not injected into the prompt.

Step 2: Point a provider at a real Claude model

promptfoo’s Anthropic provider id follows the pattern anthropic:messages:<model-id>. A working promptfooconfig.yaml for a single prompt file looks like this:

prompts:
  - file://support-reply-prompt.txt

providers:
  - id: anthropic:messages:claude-sonnet-4-6
    config:
      temperature: 0.2
      max_tokens: 1024

tests:
  - vars:
      ticket: "My invoice shows a charge I don't recognize from last Tuesday."
    assert:
      - type: contains
        value: "invoice"
      - type: not-contains
        value: "As an AI"
      - type: llm-rubric
        value: offers a concrete next step instead of a generic apology

  - vars:
      ticket: "The app crashes every time I try to export a PDF."
    assert:
      - type: contains-any
        value: ["crash", "export", "PDF"]
      - type: javascript
        value: output.length < 600

That config is deliberately narrow in one way worth flagging before you copy it: temperature only applies on some Claude models. promptfoo's own documentation for the Anthropic provider states that Claude Sonnet 5 and Opus 4.7, 4.8, and 5 reject the parameter outright, and the tool silently omits it for those models rather than erroring. If you're testing one of those newer models and your assertions depend on consistent, low-variance output, that's a real behavior difference to account for — not a bug in your config, but a model capability you need to check before assuming temperature: 0.2 is doing anything at all. This is exactly the kind of small, version-specific detail that Claude's model deprecation and version changes can quietly affect: a test suite pinned to a model id that later gets retired, or a config that assumed a parameter still worked, both fail for reasons that have nothing to do with your prompt.

Step 3: Know what each assertion type actually checks

• contains / contains-any / not-contains — exact substring checks. Fast, deterministic, and good for things that must or must not appear verbatim, like a disclaimer or a banned phrase such as "As an AI."
• javascript — runs an arbitrary expression against the string output. Useful for anything a regex or length check can express, like capping response length or validating that output parses as JSON.
• llm-rubric — a second model grades the first model's output against a plain-English rubric you write. This is the only assertion type on this list that can judge tone, completeness, or "did it actually solve the problem" — and the only one whose result can vary between runs, since it depends on a second model call.
• defaultTest — not an assertion type itself, but a block that applies the same assertions to every test case without repeating them. Use it for rules the whole prompt must follow, like never claiming to be human.

Step 4: Run it, and read the matrix

promptfoo eval -c promptfooconfig.yaml

promptfoo eval runs every test case against every provider listed and opens a results matrix with pass/fail per cell. Add a second provider — another Claude model, or a competing model entirely — and the same test cases become a side-by-side comparison instead of a single pass/fail run, which is useful context alongside a checklist for choosing between Claude, OpenAI, and Gemini structured output modes if part of what you're testing is whether a prompt survives a provider swap at all.

Mistakes that make a test suite worse than no test suite

• Grading everything with llm-rubric. A model-graded assertion costs an extra API call and can disagree with itself between runs. Use contains or javascript for anything that has one objectively correct answer, and save the rubric for genuinely subjective judgments like tone.
• Testing only the happy path. A support-reply prompt tested only against polite, well-formed tickets will pass every time and tell you nothing about the angry, run-on, multi-issue ticket that actually breaks it in production — the same blind spot covered from the attacker's side in a walkthrough of your own prompts using the OWASP LLM Top 10.
• Pinning a model id and never revisiting it. claude-sonnet-4-6 written into a config today is a specific, dated model. Anthropic's deprecation notices give 60 days' warning before a model stops responding, which is enough time to update a pinned config, but only if something is watching for the notice in the first place.
• Treating a passing suite as proof of correctness. Assertions can only check what you thought to write. A prompt that passes every test and still produces a wrong invoice total passed because nobody wrote an assertion for arithmetic, not because the output was actually right.

Step 5: Wire it into CI, not just your laptop

A promptfoo config that only runs when someone remembers to run it locally catches regressions after they ship, which defeats most of the point. Running promptfoo eval -c promptfooconfig.yaml as a step in a GitHub Actions workflow, with a non-zero exit code failing the job on any assertion failure, turns the same config into a gate on every pull request that touches the prompt file. The setup is close to what configuring Claude Code's permission mode for a CI pipeline already covers for an agent running inside CI: a secret for ANTHROPIC_API_KEY in the repository's action secrets, a job step that installs promptfoo, and a step that runs the eval command against the same YAML file you tested locally. The only prompt-specific wrinkle is that an llm-rubric assertion makes a live API call to grade the output, which means CI runs cost real tokens every time — worth knowing before you wire the suite into every single commit rather than only pull requests that change the prompt.

Choosing an assertion without overthinking it

Most of the friction in a first promptfoo config comes from reaching for llm-rubric by default, because it feels like it can check anything. A faster rule: start with whether the thing you're checking has one correct, literal answer. "Does the reply mention the word invoice" is a contains check. "Is the output valid JSON" is a javascript check (JSON.parse(output) wrapped in a try/catch, returning a boolean). "Did the model refuse to answer" usually reduces to not-contains against a known refusal phrase, which is more reliable than it sounds, because refusals tend to share boilerplate language across a given model version. Only once you've exhausted what a literal check can express — tone, helpfulness, whether an explanation would satisfy a confused customer — does llm-rubric earn its extra API call. Reversing that order, writing every assertion as a rubric first, is the single fastest way to end up with a test suite that's slow, expensive, and only approximately repeatable.

What a test case should actually cover

The two test cases in the config above are a starting shape, not a complete suite. A support-reply prompt used in production needs cases for the ticket that's actually two complaints stapled together, the one written in a language other than the examples in your prompt, the one that's three words long with no context, and the one that quotes a policy the model should recognize but not repeat verbatim if it's outdated. Each of those is a one-line addition to the tests array, with its own vars and whatever assertions fit — the cost of adding a case is small enough that the real failure mode isn't writing too many, it's writing only the ones that occurred to you on the first pass and never coming back to add the ones a real user eventually sends. Treat a production incident caused by a prompt as a trigger to add the case that would have caught it, the same way a bug report becomes a new unit test in ordinary software; the suite should only ever grow. A suite with twelve well-chosen cases that mirror real traffic beats one with two hundred near-duplicates of the same happy path. Favor realistic traffic over invented edge cases whenever real examples are available.

Featured image: a failed electronic component under magnification, public domain (CC0), via Wikimedia Commons — the kind of defect a quick glance misses and a close, repeatable check catches, which is the whole argument for running the same test cases on every change instead of re-reading the prompt and trusting your eyes.

Sources: promptfoo — Anthropic provider documentation and promptfoo — Configuration guide.