Module 4 · Project 3: LLMOps and MLOps — Monitor, Evaluate and Improve the Live App

OBJECTIVE: run the deployed app like a professional team: see every request, track quality, cost and speed, version prompts and models, test changes before release, get alerts when something goes wrong, and improve safely using real usage data. WHAT WE COVER (in project order): • Tracing every LLM request (inputs, retrieved chunks, outputs, tokens, cost, latency) • Prompt management and versioning • Evaluation in CI: block releases that make quality worse • Online monitoring: user feedback, quality sampling, cost and latency • Metrics and dashboards with Prometheus and Grafana; alerts • Cost and speed controls: caching, model routing, fall-backs • Guardrails in production • MLOps for the ML model: experiment tracking, model registry, drift monitoring and retraining • Safe releases: A/B tests and canary roll-outs AFTER THIS MODULE YOU CAN: • Monitor quality, cost and latency of a live AI app • Release prompt and model changes safely with tests and gradual roll-outs • Track, register and monitor ML models over time TOOLS: Langfuse (or LangSmith), promptfoo or pytest-based evals, Prometheus, Grafana, Redis, MLflow, Evidently.