Module 5 · Project 3: Fine-Tune a Small Open LLM (LoRA / QLoRA)

OBJECTIVE: fine-tune a small open model (1–3 billion parameters) to do Project 1's email extraction as well as a big API model, but cheaper, faster and fully private — and learn when and how to fine-tune an LLM. WHAT WE COVER (in project order): • When to fine-tune vs prompt vs RAG • Building an instruction dataset: format, chat templates, synthetic data from a bigger model, human review • LoRA and QLoRA: training a small part of the model on a free GPU • Training with Hugging Face TRL • Evaluation on Project 1's test set: small fine-tuned model vs base model vs big API model • Optional alignment with preference tuning (DPO) • Deployment: merge, share on the Hub, run locally with Ollama, Gradio app; serving with vLLM and estimating cost AFTER THIS MODULE YOU CAN: • Decide if fine-tuning is worth it for a problem • Prepare an instruction dataset and fine-tune an open LLM with LoRA • Prove with numbers whether the fine-tuned model is good enough to replace an API DATA: 1,500–2,000 email → JSON examples (Bitext messages labelled by the big model from Project 1, with a reviewed sample) + Project 1's golden test set for evaluation. TOOLS: Hugging Face Transformers, TRL, PEFT, bitsandbytes, Ollama. GPU: free Colab/Kaggle T4.