Module 3 · Project 1: Customer Support Email Assistant (prompting, structured output, tool calling)

OBJECTIVE: build an assistant that reads a customer email, sorts it into a category, pulls out the key details as clean data (JSON), looks up the order status with a tool, and drafts a reply for a human agent to approve — and learn how to make LLM output reliable and measurable. WHAT WE COVER (in project order): • Defining the product and how we will judge it • Creating a test set of emails with the correct answers (a 'golden dataset') • Structured output: getting valid JSON every time and checking it with Pydantic • Tool (function) calling: letting the model ask your code for data • Drafting replies with rules for tone and safety • Calling APIs reliably: streaming, batching, retries, rate limits, switching providers • Evaluation: accuracy, JSON validity, LLM-as-judge for reply quality, cost and speed • Deployment: Gradio app with a human approval step AFTER THIS MODULE YOU CAN: • Turn messy text into reliable structured data with an LLM • Connect an LLM to your own functions • Measure an LLM app with numbers before you ship it DATASET: Bitext customer-support dataset (about 27,000 customer messages with category and intent) + 50 emails you write yourself. TOOLS: an LLM API (or Ollama), Pydantic, Gradio.