The Chunk Was Never in the Document, Leslie Li

The common pattern for answering questions from documents looks like this. Cut the text into pieces a few hundred tokens long, turn each piece into a vector, keep the vectors, and at query time fetch the handful closest to the question. The model answers from those pieces. That is what people mean by RAG, and the chunk is the unit the rest of the pipeline is built on.

Where a chunk ends is a setting in the pipeline, not a fact about the document. Set the length to 512 tokens and a paragraph about electoral systems lands half in one piece and half in the next, and the two now compete for the same question without either being a complete thought. The table of contents was written by the author instead. Its entries have titles that say what is inside, and they stop where the subject stops.

A paper from IBM takes the second unit. STAIR gives a model the whole table of contents and the question, and asks it to name the one section that answers it. The section title is the document identifier. Retrieval returns a section, and the text of that section is what reaches the answer model.

On SearchTome, a benchmark the authors built from 18 open textbooks across six domains, STAIR reaches 82.6 percent Recall@1. The comparison points are a fine-tuned DSI at 76.9, DPR, a dense retriever, at 68.7, BM25, plain keyword search, at 59.5, and Claude Haiku 4.5 asked without any training at 45.6. The gap over the closest baseline is 5.7 points, which the paper calls 7.4 percent because it is relative.

The number is less interesting than the reason for it. DSI, the earlier system that generates a document identifier from a question, has to learn the book's structure from question and answer pairs and hold it in its weights. A section that appears in two training questions is nearly invisible to it. STAIR reads the structure from the prompt on every query, so a section seen twice is still on the menu. The authors show the gap widening as the number of training examples per section falls.

The figure rests on choices in the method. The questions were generated by Mixtral, several per paragraph, then split into training and test, so the same paragraph supplies questions to both sides and its section is always the gold answer, which makes a test question likely to be a rephrasing of one the model trained on. The same 18 books appear in training and in testing, so nothing here is measured on a document the model has not seen. And STAIR is fine-tuned while BM25 and DPR run off the shelf, which widens the margin over those two rather than over DSI, which is tuned as well. The number is a result inside its own benchmark, not an estimate for a corpus the model meets cold.

The abstract's claim of less than 0.05 percent hallucination also means less than it sounds. The paper defines hallucination as naming a section that is not a leaf in the contents, not as answering a question wrongly. Its pipeline diagram then describes inference as constrained generation restricted to valid leaf titles, which would make such an output impossible. The paper does not reconcile the two, so the figure describes the decoder as much as the model.

The move itself transfers without any of that machinery, and it takes about twenty lines. Split the document at its headings, hand the model the list of sections, ask for a number back, and fetch that section's text.

import re

HEADING = re.compile(r"^(#{1,4})\s+(.+?)\s*$", re.M)


def sections(md):
    """Split at headings. Returns {heading path: text}."""
    marks = list(HEADING.finditer(md))
    out, path = {}, []
    for i, m in enumerate(marks):
        path = path[: len(m.group(1)) - 1] + [m.group(2)]
        end = marks[i + 1].start() if i + 1 < len(marks) else len(md)
        text = md[m.end():end].strip()
        if text:
            out[" > ".join(path)] = text
    return out


def pick(question, titles, ask):
    menu = "\n".join(f"{i}. {t}" for i, t in enumerate(titles, 1))
    reply = ask(
        f"Table of contents:\n{menu}\n\nQuestion: {question}\n\n"
        f"Reply with the number of the section that answers it. Number only."
    )
    n = re.search(r"\d+", reply or "")
    return titles[int(n.group()) - 1] if n and 1 <= int(n.group()) <= len(titles) else None


# usage: ask_model(prompt: str) -> str is your own model call
doc = sections(open("handbook.md").read())
chosen = pick("How much parental leave do I get?", list(doc), ask=ask_model)
context = doc.get(chosen, "")

The headings are the index, so a corpus small enough to list needs no embedding pass at all. The model returns a number, so a bad answer is visible in the reply itself. And the retrieval unit has a boundary the author drew, so an error sends you to the wrong chapter, and the mistake is obvious.

If the reply is not a number in range, fall back to embedding the section titles and taking the nearest few, which is ordinary vector search over a much smaller and cleaner set than chunks. If the contents is too long for the window, the method stops being cheap: 400 sections is roughly 4,000 tokens and fits comfortably, and 40,000 is a different problem with a different design.

It also needs headings that mean something. A transcript, a chat log, a scanned contract with no bookmarks, or a wiki whose every page carries one title has no structure to read. The paper says as much in its limitations: the method needs a corpus with a global structure, and creating one on the fly is future work. The honest scope is documents written to be navigated: most books, standards, manuals and internal handbooks. Most of the text a company accumulates is not written that way.

Sources