Module 9 — Merging adapters and exporting

Training is done. On disk you have the frozen 4-bit base and a 20 MB LoRA adapter file, and inference works — but only if you keep the two together and load them through the peft library. That is fine for a Jupyter notebook. Production wants a single artifact, ideally in a format that a lightweight runtime like llama.cpp or Ollama can serve without a Python environment. This module bridges the two worlds.

Merging: reversing the LoRA decomposition

Recall from module 5 that LoRA writes the update to a weight matrix as $\Delta W = B A$, and that at inference the effective weight is $W' = W + (\alpha / r) B A$. Nothing prevents you from actually computing that sum once and storing the result in place of $W$. That is what merging does.

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load the base in full precision, not 4-bit: merging into a 4-bit tensor
# is not what you want, because the merged result will not requantize cleanly.
base = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-v0.3",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

# Attach the trained adapter, then fold it into the base.
model = PeftModel.from_pretrained(base, "./out/best-checkpoint")
merged = model.merge_and_unload()

# Save as a plain Hugging Face checkpoint.
merged.save_pretrained("./merged-mistral-minutes", safe_serialization=True)
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.3")
tokenizer.save_pretrained("./merged-mistral-minutes")

Two subtleties in that snippet. First, the base is loaded in bfloat16, not in 4-bit — merging into a quantized tensor is technically possible but fragile, and the requantization step will introduce artifacts. Load full precision, merge, then requantize once if you need to. Second, merge_and_unload() returns a plain AutoModelForCausalLM — the peft wrapper is gone. From here on the checkpoint behaves exactly like the original base model, and every downstream tool sees a standard 7B checkpoint.

When to keep the adapter separate

Merging is not always the right move. If you plan to serve multiple specialized adapters on the same base — one per team, one per output format — keep them separate and swap them at inference time. The RAM saving is dramatic: a 14 GB base and ten 20 MB adapters is 14.2 GB, versus 140 GB for ten merged variants.

The decision follows one question: how many variants will you serve simultaneously? One or two: merge, get the simplicity. Ten or more: keep them separate, get the memory back. In between: pick the operationally simpler option for your team.

Exporting to GGUF

Once merged, the model is a standard Hugging Face checkpoint. To serve it with llama.cpp or Ollama — the two lightweight runtimes that dominate on-device inference — you convert it to the GGUF format, a single-file binary that packs weights, tokenizer and metadata.

The conversion tool is convert_hf_to_gguf.py, shipped with llama.cpp.

python llama.cpp/convert_hf_to_gguf.py \
  --outfile mistral-minutes-f16.gguf \
  --outtype f16 \
  ./merged-mistral-minutes

The output is a 14 GB file. For inference on modest hardware, quantize it further:

./llama.cpp/build/bin/llama-quantize \
  mistral-minutes-f16.gguf \
  mistral-minutes-q4_k_m.gguf \
  Q4_K_M

Q4_K_M is the sweet spot most teams settle on: about 4.5 GB on disk, minimal quality degradation for chat and instruction tasks, fast on a laptop CPU or a mid-range GPU. Q8_0 is closer to lossless if you have room for the 7 GB file. Below Q4_K_M, quality noticeably degrades and the space savings are marginal — Ollama's own quantization defaults settled on Q4_K_M for the same reason.

Ollama in one command, once you have the GGUF

Ollama consumes GGUF files directly. You write a small Modelfile next to the GGUF that describes the base file, the chat template and any system prompt:

FROM ./mistral-minutes-q4_k_m.gguf

TEMPLATE """{{ if .System }}<s>[INST] {{ .System }}
{{ .Prompt }} [/INST]{{ else }}<s>[INST] {{ .Prompt }} [/INST]{{ end }}"""

SYSTEM "You write meeting minutes in the fixed JSON schema."

Then:

ollama create mistral-minutes -f Modelfile
ollama run mistral-minutes "Meeting transcript: [Alice] ..."

Course 29 covers the operational side — model warmup, concurrency, monitoring — in detail. Here, the point is that once the GGUF exists, deployment is genuinely a few commands, and there is no Python involved.

Publishing to the Hub

If the fine-tune is meant to be shared (a team-internal Hub or the public huggingface.co), publish the adapter rather than the merged model. Adapters are 20 MB, they are cheap to distribute, and users can attach them to the base model they already have locally.

from huggingface_hub import HfApi

api = HfApi()
api.upload_folder(
    folder_path="./out/best-checkpoint",   # the LoRA adapter, not the merged model
    repo_id="your-team/mistral-minutes-lora",
    repo_type="model",
)

For a merged variant, upload the full checkpoint instead. Do not upload both under the same name — pick one and be explicit about which the users get.

Licensing: the step everyone forgets

Every base model comes with a license. Mistral models are Apache 2.0 (permissive, commercial use allowed). Llama 3 is under a custom Meta license that permits most uses but has restrictions for products with more than 700 million monthly active users and requires a specific attribution string. Qwen is Apache 2.0. Gemma is under the Gemma Terms of Use, which prohibit certain uses and require passing terms to downstream users.

Your fine-tuned model inherits the license of the base, and often adds constraints from your training data (if you fine-tuned on synthetic data generated by GPT-4, OpenAI's terms restrict certain commercial uses of the derived model). Before pushing to a public Hub or shipping to customers, check three things:

• The base model license and its non-standard clauses.
• The license of any dataset you used for training.
• The terms of service of any API used to generate synthetic data.

The correct place to record this is a README.md in the model repository, with the license field of the model card explicitly set. Deployment platforms parse that field and will refuse to serve models with an ambiguous or missing license.

A merged checkpoint is not "just weights"

A common misconception is that once merged, the fine-tune is legally independent of the base model. It is not. Every jurisdiction that recognizes derivative works considers the fine-tune a derivative of the base, and the base's license flows through. Attempting to relabel a Llama-3 fine-tune as a plain custom checkpoint is both legally shaky and, in most cases, a violation of the Meta license.

Summary

• Merging folds the LoRA update into the base weights with merge_and_unload(), producing a standard checkpoint that no longer requires the peft library at inference.
• Merge when you serve one or two variants; keep adapters separate when you serve ten or more, to save 12 to 14 GB of RAM per unused variant.
• GGUF conversion via convert_hf_to_gguf.py and quantization to Q4_K_M give a 4.5 GB single-file model that llama.cpp and Ollama serve directly.
• The fine-tune inherits the license of the base model and any synthetic-data source; record it explicitly in the model card before publishing.

Next module: measuring whether the fine-tune actually made the model better on the meeting-minutes task, and whether it made it worse on anything else.