Skip to main content

Module 2 — Text extraction from PDF, HTML and office documents

Module 1 laid out the four-stage pipeline. Everything from here to module 4 is preparation: turning binary files into clean, well-labelled passages the index can search. The most common cause of a bad RAG answer is a bad extraction, and that is what this module fixes.

PDF: the format that keeps its secrets​

A PDF encodes the look of a page, not its structure. Under the hood, the file describes glyphs placed at coordinates, not paragraphs. Two consequences follow immediately.

Reading order is a reconstruction. The library reads glyphs in the order they were stored, which is often close to left-to-right, top-to-bottom, but not always. A two-column layout, a boxed callout, a footnote can appear in wildly wrong positions after a naive extract. pdfplumber and PyMuPDF both reconstruct blocks, and disagree on the details.

import pymupdf   # aka fitz

doc = pymupdf.open("procedures/QUAL-047.pdf")
for page in doc:
    text = page.get_text("text", sort=True)   # sort by y then x
    print(text[:500])

Tables are lost by default. A row is a horizontal alignment, invisible to a text extractor. pdfplumber.extract_tables() does its best, but a merged cell or an image-based rule breaks it. For a procedure that lists roles and responsibilities in a two-column table, the raw text becomes an unreadable stream. Module 3 comes back to this: a table split down the middle by a naive chunker is one of the classic RAG failures.

Running headers and footers pollute every page. "Confidential — page 4 of 12" repeats 12 times, and the search index will happily return it. A cheap fix is to compare the first and last lines across pages of the same document and drop those that appear identically on more than three quarters of them.

A scanned PDF contains no text

A PDF produced from a scanner is a wrapper around images, and get_text() returns an empty string with no error. The symptom is a corpus that grows without adding any content to the index. Detect it: if text.strip() is empty on a page whose visual is not blank, it needs OCR.

OCR when the pixels are all you have​

tesseract is the open-source workhorse, and Python has a thin wrapper in pytesseract. Two settings matter more than the choice of engine.

from pytesseract import image_to_string
from pdf2image import convert_from_path

pages = convert_from_path("procedures/scan-2024.pdf", dpi=300)
text = "\n\n".join(image_to_string(p, lang="eng+fra") for p in pages)

Rendering at 300 dpi roughly doubles recognition accuracy over the default 200. Passing both languages the corpus uses (eng+fra, sometimes eng+ara) avoids the classic error of turning accented characters into look-alikes. On a page containing formulas or an unusual font, expect one to five errors per page even at that quality. OCR is never a solved problem, only a good-enough one.

HTML: strip the wrapper, keep the structure​

HTML looks textual and is not. Between the words your users care about sit navigation bars, cookie banners, related articles, and JavaScript loaders that show blank text until the browser renders. Two libraries do most of the work.

from bs4 import BeautifulSoup
import requests

html = requests.get(url, timeout=20).text
soup = BeautifulSoup(html, "html.parser")

for tag in soup(["script", "style", "nav", "footer", "aside"]):
    tag.decompose()

text = soup.get_text(separator="\n")

The four tags above catch most of the noise on a well-structured intranet page. On a badly structured one, tools like trafilatura or readability-lxml guess the "main content" heuristically and often do it better than a hand-crafted list of tags. On dynamically loaded pages, none of this works: you need a headless browser (playwright), which multiplies extraction time by ten. Reserve it for a small subset of sources where the value justifies the cost.

Preserve headings: the <h1>, <h2>, <h3> structure is exactly the signal module 3 will use to chunk on section boundaries. A flat text loses that.

Office documents: use the structured API​

Word, Excel and PowerPoint are XML in a ZIP. Do not try to read them as text.

from docx import Document          # python-docx for .docx
from openpyxl import load_workbook # for .xlsx
from pptx import Presentation      # python-pptx for .pptx

for para in Document("notes/procedure-047.docx").paragraphs:
    print(para.style.name, "|", para.text)

python-docx gives paragraphs and their styles, which lets you recognise a heading without guessing from font size. openpyxl gives cells with rows and columns intact, which is the only way to make a spreadsheet's meaning survive extraction: extract each sheet as a small table, not as a concatenation of cell values. python-pptx gives one slide at a time with its title, its bullet points and its notes — reproduce that structure in the chunk so the model knows what came from where.

Metadata is not optional​

Every extracted passage travels with a small dictionary. That dictionary is what makes a citation possible and a permission check possible. Deciding it once, at extraction, is much cheaper than reconstructing it later.

def to_chunk(text: str, source_path: str, page: int | None,
             section: str | None) -> dict:
    return {
        "text": text,
        "source_path": source_path,
        "source_name": source_path.rsplit("/", 1)[-1],
        "page": page,
        "section": section,
        "extracted_at": "2026-09-06",
        "language": "en",
    }

The minimum you regret not having on day one: source path (to open the file in a click), page (for PDFs), section (for structured documents), extraction date (for cache invalidation, module 9) and language (for filtered search, module 4). Add an access class if permissions matter — the assistant of module 10 will refuse to cite a document a user is not entitled to see.

Log every failed extraction

An extraction that silently returns an empty string is worse than one that raises. Wrap each parse in a try/except, write the failure to a log file with the path and the exception, and revisit that file weekly. A quiet 5 % failure rate turns into a 15 % coverage gap in a corpus that grows.

In summary​

  • A PDF encodes glyphs and coordinates, not structure: reading order is reconstructed, tables are fragile, and running headers must be stripped by comparing lines across pages.
  • A scanned PDF returns an empty text; detect it and OCR at 300 dpi with the languages of your corpus explicitly declared.
  • HTML needs its script, style, nav, footer and aside tags removed before extraction, and the heading hierarchy preserved so chunking can split on sections.
  • Every extracted passage carries a metadata dictionary — source, page, section, date, language, access class — which is what makes citations and permission checks possible downstream.

Next module: how to slice those long extracted texts into chunks that respect their structure and fit the model's context window.