Your golden set is lying to you

A dataset you built in March does not describe what customers ask in September. Here is how we keep ours honest.

Evals

7 min read

Priya Raman

Founder

A stack of index cards with pencil check marks

Most teams build a golden set once, in a spreadsheet, the week before launch. It passes. Six months later it still passes, and support tickets say the bot is getting worse.

The set did not change. The questions did. New plans, new features and a pricing change all moved what people ask.

We refresh ours every night from production traces, keep the last ten snapshots and look at which cases are new each week. The new cases are where the regressions hide.

Try this on your own prompts. The free plan includes 500 runs a month.

Try this on your own prompts. The free plan includes 500 runs a month.

Use this free template

Create a free website with Framer, the website builder loved by startups, designers and agencies.