Datadog
Send grader scores and cost per request as metrics next to your latency graphs.
Fenwick exports pass rate, grader scores, tokens and cost as Datadog metrics tagged by prompt version.
Build a monitor on quality the same way you already monitor p95 latency.
Category
Observability
Setup time
10 minutes
Plans
Every plan, Hobby included
More integrations
Run your next prompt change against 200 real cases this afternoon.
Free for 500 runs a month. Team from $39 a seat.