Skip to main content

Small language models

Not every task needs a hundred-billion-parameter model behind an API. This course is about the moment a 1 to 4 billion parameter open model, running on the workstation with no cloud call, becomes the honest answer — and about the disciplines that turn that answer into a system that ships.

Course Duration: 5h​

What You'll Learn​

  • Decide when a small local model beats an API call on the same task, and when it does not
  • Read the landscape of open small models — Phi, Gemma, Qwen, Llama, Mistral — with their licenses and language coverage
  • Distill a large teacher into a small student with synthetic data on your own task
  • Prune and quantize a model to 8 or 4 bits and measure the quality loss you actually pay
  • Choose an efficient runtime format — GGUF, ONNX, MLX — for the target hardware
  • Measure latency and throughput on a laptop CPU and an entry-level GPU with a reproducible protocol
  • Fine-tune a 1 to 4B model cheaply with LoRA on your own annotated data
  • Ship the model locally on the workstation with Ollama or llama.cpp and integrate it into an existing tool
  • Argue privacy through local execution honestly, including what local does not protect
  • Assemble the pieces into a specialized desktop assistant and compare it against the API on a real evaluation set

Prerequisites​

  • Course 16 — Large language models (tokens, context window, temperature, API arithmetic)
  • Course 19 — Fine-tuning (LoRA, adapters, evaluation before and after) is useful in module 7
  • Working Python and a laptop with 16 GB of RAM; a GPU with 8 GB VRAM is helpful but not required

Course Modules​

  1. Why size is not always the answer
  2. Landscape of open small models
  3. Knowledge distillation and compression
  4. Pruning and quantization
  5. Efficient runtime formats
  6. Measuring latency and throughput
  7. Cheap fine-tuning of a small model
  8. Local and offline execution
  9. Data privacy through local execution
  10. Project: a specialized desktop assistant

The running project​

Across the ten modules you build the same object: a support-ticket assistant that runs on the workstation of a support agent, offline, with no cloud call. It classifies incoming tickets into a small taxonomy (billing, technical, account, other), extracts a two-sentence summary and suggests a canned answer. Module 1 measures a small local model against a big API on the same 200 tickets; each following module improves one dimension — the model choice (2), the training data (3), the model size (4), the runtime (5), the measurement discipline (6), the fine-tune (7), the deployment (8), the privacy story (9) — and module 10 assembles the final assistant end to end.

Assessment and certificate​

The course ends with a 40-question exam covering every module. On success, a certificate of completion is issued; its number is verifiable on the platform.

Free courses, by contrast, end with a 5-question quiz and a preview of the certificate, without certification.