Agent evaluations are the new unit tests
Every team that runs an agent in production eventually builds the same thing: a suite of scenarios that must not regress. The only question is whether you build it before or after the first customer-facing incident.
What happened
Agent behaviour is non-deterministic, so a prompt tweak that improves one case silently breaks three others. Teams without an eval suite discover this from a support ticket, weeks later.
Why it matters for deployment teams
An eval suite turns prompt engineering from opinion into engineering. With 30 to 50 representative scenarios scored on every change, you can actually say whether a modification was an improvement.
What to do about it
Start with the failures you have already seen. Write each one as a scenario with an explicit pass condition, wire it into CI, and refuse to merge agent changes that drop the score without a written reason.
What to do about it
- Collect 30-50 real scenarios, not synthetic ideals
- Score every prompt change in CI
- Treat a score drop as a failing build