Medical Agentic RAG is my search system for Vietnamese medical textbooks, built for research and study. The books are scanned PDFs, so there is no text to search until something reads every page. That sounds like a job for a vision model, and partly it is. But a model that reads a page should not also decide how the book is structured. This post covers how I split that work, and how search copes with questions typed without accents.
Why scanned textbooks are hard
A scanned page is an image. Tables, figures and headings are all pixels, and a model reading them can misplace a heading or merge two lessons. Vietnamese adds its own trap: diacritics change the meaning of a word, and models misread them.
Readers make it harder from the other side, because many type their questions without accents. And a reader who finds an answer needs the exact place it came from, with the section and the pages, so they can check the book itself. Book text can be dated or wrong, which is why every result has to point back to its source page.
Not for clinical or diagnostic use. Answers come from textbooks, which can be dated or wrong.
Let the model read, let code decide
In my pipeline the model only transcribes pages. A layout model crops figures and tables, and heading levels, ids, section splits and front matter are computed in code. Lesson boundaries are checked against the book's own table of contents.
A cheaper vision model reads every page. Flagged pages, such as tables, figures, uneven tables, missing markers, table of contents pages or truncated answers, go to a stronger model. Suspected diacritic misreads are checked against the page image, and only fixes that change diacritics alone, into words the book prints elsewhere, are applied. There is no measured accuracy yet: no reference transcriptions ship with the code.
Search that survives missing accents
Each section is one chunk. It is indexed with two embedding models and with BM25 on the accented text and on a folded copy without accents, so a question typed without diacritics still matches. The result lists are fused with RRF, then reranked. Each result comes back with its section, its page range and the original page image, and every step is traced.
Planned, not built yet: an agent that plans the search, checks the results and writes answers with citations. The lesson so far: keep the model on a narrow job and put structure in code that can be checked.
Project: Medical Agentic RAG