subtopic

Score accuracy, quality, cost and speed

Run the full pipeline on the 120 test emails. Code checks: % valid JSON, category accuracy, order-ID exact match. For replies use LLM-as-judge: a second prompt scores each reply 1–5 against your 5 rules and explains why; check 20 judge scores yourself to see if you agree. Record average cost (tokens × price) and seconds per email.

Builds on: S02 · Build: An Evaluation Harness (used for every model), 5.7 Evaluate