LLM eval suite that gates PRs

# LLM eval suite that gates PRs

Catch regressions in an LLM feature before merge: 50+ cases from real traffic, the suite runs in under 5 minutes with caching, a PR fails when its pass rate drops more than 2 points below the main-branch baseline, and model-graded checks agree with human labels on at least 90% of a 30-case calibration set.

## Recommended stack

- **promptfoo 0.124** - Eval runner, assertions, caching, before/after comparison. YAML test cases with deterministic and model-graded assertions, weights, and thresholds; still open source and released regularly after joining OpenAI, with multi-provider support kept.
- **promptfoo/promptfoo-action@v1 on GitHub Actions** - PR gate and comparison comment. Runs on pull_request for prompt paths and posts a link to the before/after diff on the PR.
- **Your production LLM function as a promptfoo custom provider** - System under test. Exercises the real prompt template, tools, and parsing instead of a copy of the prompt that drifts.
- **A labeled JSONL dataset in the repo** - Test cases. Versioned with the code, reviewed in PRs, and grown from real failures.

## Build steps

1. Pull 50-100 real inputs from logs, scrub personal data, and label the expected behavior. Tag each as happy path, edge case, past bug, or adversarial.
2. Wrap the production function as a custom provider (`file://evals/provider.ts`) so evals call the same code path as users.
3. Write deterministic assertions first (is-json, javascript checks on parsed fields, contains, latency, cost). Use llm-rubric only for qualities code cannot check, set weights, and give each test a threshold.
4. Calibrate model-graded checks: hand-label 30 outputs, run the graders, and edit rubrics until agreement is 90% or higher. Pin the grader model version in config.
5. Add `promptfooconfig.yaml` with defaultTest and derivedMetrics for an overall score; run `npx promptfoo@0.124 eval -o results.json` and inspect with `npx promptfoo view`.
6. Add a workflow on pull_request for prompts/ and src/llm/: set env PROMPTFOO_CACHE_PATH=~/.cache/promptfoo and cache that path, run promptfoo/promptfoo-action@v1 (cache-path: ~/.cache/promptfoo) for the before/after comment, then run `npx promptfoo@0.124 eval -o results.json` with PROMPTFOO_PASS_RATE_THRESHOLD=0 and a short node script that computes successes/(successes+failures) from results.json results.stats, compares it with evals/baseline.json committed from main, and exits 1 when it drops more than 2 points.
7. Make it policy that every production LLM bug ships with a new test case in the same PR as the fix.

## Watch out for

- Evaluating a pasted copy of the prompt: it drifts from production within weeks. Always go through the custom provider.
- An LLM judge as the only assertion: graders are noisy and change with model updates. Keep deterministic checks as the majority and recalibrate when the grader model changes.
- Flaky results from sampling: for non-deterministic features run each case 3 times and gate on the mean, or set temperature 0 where the provider supports it.
- Cache hiding a real change: bust the cache when you switch models and run a nightly job with `--no-cache`.

## Definition of done

- `npx promptfoo eval` finishes in under 5 minutes and writes results.json with a pass rate.
- Deleting the output-format instruction from the prompt in a test PR turns the check red.
- The PR shows a before/after comparison link.
- The calibration file in the repo shows at least 90% grader-human agreement.