NeoMME Retriever: A Practical Candidate Recall Map for Visual Document Retrieval

Why NeoMME matters for page-image retrieval now

H Company released NeoMME on September 3, 2026 as a family of 260M and 800M multilingual, multimodal encoders. The release is worth attention for one practical reason: NeoMME Retriever can produce dense embeddings and late-interaction embeddings from the same forward pass. For visual document retrieval teams, that moves the model from benchmark curiosity to architecture decision.

Page-image retrieval searches rendered pages, not just extracted text. The relevant evidence may sit in a table, a diagram label, a scanned appendix, a multilingual note, or a layout where spatial context changes the meaning. OCR still matters. In clean PDFs with exact identifiers, it may be the strongest baseline. But an OCR-only design leaves out a class of document questions where the page image itself carries the signal.

The boundary is where builders need discipline. A retrieval encoder retrieves pages. It does not generate answers, prove visual RAG faithfulness, or remove the need to evaluate a downstream VLM or language model. Published retrieval scores can guide selection, but they should not be recast as answer-accuracy claims. The same logic sits behind Optijara's article on benchmark comparability and relevance testing: a metric is useful only when everyone agrees what it measures. For production planning, Optijara's small-model deployment reality test makes the same point in a different setting: measure the artifact, runtime, hardware, and workload before the architecture hardens.

The practical question is not whether an ANN index can store a lot of vectors. The question is whether first-stage dense search brings back the right pages often enough for late interaction to improve the order. If the candidate set is weak, the reranker is improving the wrong shortlist.

The architecture in plain English: one encoder, two retrieval heads

NeoMME uses a single bidirectional Transformer over text tokens and raw image patches. H Company's materials describe 32 by 32 pixel patches, a 16,384-token context, and no separate pretrained vision tower or causal language model in the retrieval path. The 260M retriever card lists 263M parameters, 1,024 hidden size, 1,024-dimensional dense embeddings with Matryoshka dimensions, and 128-dimensional multi-vector embeddings per text token or image patch. The 800M card lists 800M parameters, 1,792 hidden size, and the same 128-dimensional multi-vector route.

The dense head is the candidate-search workhorse. It creates a compact representation that can be searched with cosine similarity in familiar vector infrastructure. This is usually where production teams start because it fits ANN indexing, metadata filters, access controls, and retrieval logs.

The late-interaction head keeps a more detailed token or patch lattice. The model cards describe L2-normalized multi-vector embeddings scored with MeanMaxSim. That style of scoring can preserve local evidence better than a single dense vector when the query depends on a table cell, caption, diagram label, or a small region of a page. The catch is that details matter. Padding, masking, normalization, query precision, document precision, and the exact scoring function can change quality and cost.

There are also separate Sentence Transformers checkpoints. The ST dense model only generates dense embeddings. The ST late-interaction model only generates multi-vector embeddings. That is not the same operational path as the default Transformers model, which returns dense and multi-vector embeddings together through NeoMMEForRetrieval. If an evaluation mixes these artifacts, the report should say so in plain language.

The candidate-recall bottleneck: what reranking cannot recover

The core lesson is blunt: dense recall@K sets the ceiling for downstream reranking. Late interaction can reorder retained pages, but it cannot rescue a relevant page that never reaches the shortlist.

Take a hypothetical annual report search. A user asks for a specific risk disclosure. The answer appears once, in small print inside an appendix table. A dense page embedding might retrieve the narrative risk section and several visually similar pages, while missing the appendix page at K equals 20. The late-interaction reranker can refine those 20 pages, but the table page remains invisible. Raising K might surface it. That also adds reranking work, memory movement, and perhaps more pages for a downstream VLM to inspect.

That is why teams should separate candidate recall@K, reranker nDCG@10, answer correctness, and citation quality. Collapsing them into one RAG score hides the failure mode. If answer accuracy improves after raising K, the gain may come from candidate recall rather than a better generator. If answer quality worsens after compression, the regression might be in retrieval, ranking, or answer synthesis.

Candidate-set sizing is a real design variable. Low K can keep systems fast while quietly capping recall. High K improves the chance that the right page reaches the reranker, but it can increase late-interaction latency, index reads, GPU or CPU pressure, and downstream context noise. The right value depends on the corpus. Language mix, layout density, scan quality, table frequency, and small print all matter. Optijara's Vaani evaluation map uses a similar habit: keep slices visible instead of letting aggregate scores hide the uncomfortable cases.

The Page-to-Candidate Recall Map: a practical evaluation workflow

The recommended framework for NeoMME-style visual retrieval is a Page-to-Candidate Recall Map. Its job is simple. It maps each query to the pages that should be reachable, the stage that finds or misses them, and the answer checks that depend on retrieval.

Start with artifacts. Pin the exact NeoMME checkpoint, processor configuration, library revision, rendering tool, image size, page identifiers, and storage format. Do not assume a later stable library release behaves like the current documentation page. Build relevance judgments at page level, then stratify them. A clean born-digital English PDF, a multilingual table, a rotated scan, and a dense appendix page should not be averaged together before anyone reviews the misses. Keep a holdout set and inspect failures manually. Small print, near-duplicate layouts, headers, footers, and long tables are where polished demos often differ from production behavior.

Then compare routes. OCR-text retrieval remains a serious baseline for clean PDFs, exact terms, identifiers, and compliance review. Dense-only NeoMME retrieval tests broad candidate recall. Late-only evaluation, where feasible, shows the cost and quality of detailed matching. Dense shortlist plus late reranking tests the likely hybrid route. Measure candidate recall@K and nDCG@10 before answer generation. Then add per-stage latency, index bytes, preprocessing time, retrieved-page-to-answer correctness, and citation faithfulness.

What the published measurements say, and what they do not include

H Company's release reports ViDoRe v3 nDCG@10 of 0.523 for NeoMME Retriever 260M and 0.556 for NeoMME Retriever 800M. The linked paper page repeats those author-reported values and states that the 260M model outperforms evaluated models strictly below 800M parameters on that benchmark. Older ViDoRe v1 and v2 figures use nDCG@5 in the model-card tables, so they should not be compared with v3 nDCG@10 as if the metric and setting were identical.

The release also reports about 51 pages per second for the 260M setup at matched 2048 by 2048 image input size on an NVIDIA L40S GPU. Read that narrowly. It refers to preprocessed tensors with calibrated batch settings. It does not include PDF rendering, upload, indexing, query handling, downstream generation, application orchestration, observability, or human review.

Compression claims need the same care. The blog describes hierarchical token pooling and asymmetric quantization reducing late-interaction index storage from roughly 1.5 MB to about 6 kB per page, 255 times smaller, while retaining more than 95 percent of baseline nDCG@10 on ViDoRe v3. That is a specific averaged late-interaction embedding result, not a total vector-store footprint. Dense vectors, metadata, page images, index overhead, backups, access logs, and monitoring data still count. The same materials discuss a different pooling and int8 configuration around 39 kB per page with more than 99 percent retained quality. Treat those as separate tradeoffs, not default settings.

Route tradeoffs: dense, late interaction, and hybrid visual retrieval

Dense-only may be enough for coarse navigation, duplicate-like pages, or broad topic retrieval. Late interaction earns its cost when the relevant evidence is local, visual, tabular, or easily blurred inside one dense vector. Hybrid retrieval is attractive because one NeoMME encoding pass can produce both representations. Still, the hybrid route only works when the first-stage shortlist is broad enough for the reranker to see the right pages.

Resource budgeting should be explicit. If a team is deciding whether to run retrieval locally, use hosted inference, or split workloads, the questions resemble the ones in Optijara's small-model deployment reality test: exact artifact, exact runtime, exact hardware, exact workload, and clear rollback conditions.

Implementation checklist, common mistakes, and caveats

The common mistakes are ordinary, which is exactly why they keep happening. Teams treat reranker nDCG as answer accuracy. They shrink K before measuring candidate recall. They confuse embedding compression with total storage cost. They assume native multilingual support means balanced performance in every language. They skip OCR-text baselines. They ignore rendering time because the benchmark starts after preprocessing. They forget that Apache 2.0 model weights do not settle document permissions.

Operational caveats belong in the design review. Query distribution can drift. Cached embeddings can become stale after document updates. Private documents may require stricter storage, retention, and access controls. Evaluation sets can overrepresent clean pages. A downstream VLM can hallucinate even when retrieval is good, or cite the wrong page even when the right page is present. Native multilingual capability does not prove Arabic, low-resource, handwritten, or OCR-free answer quality for a given corpus.

{
  "model_family": "NeoMME Retriever",
  "retrieval_routes": ["ocr_text_baseline", "dense_only", "late_interaction", "dense_shortlist_plus_rerank"],
  "must_measure": ["candidate_recall_at_k", "ndcg_at_10", "per_stage_latency", "index_bytes", "answer_correctness", "citation_faithfulness"],
  "excluded_costs_to_add_back": ["pdf_rendering", "upload", "indexing", "metadata", "page_images", "generation", "observability"],
  "deployment_caveats": ["dense_recall_caps_reranking", "compression_is_workload_specific", "retrieval_is_not_answer_generation"]
}

A useful next experiment is small and disciplined: choose representative documents, pin artifacts, build page-level judgments, sweep K, test compression separately, and then connect retrieved pages to answer and citation checks. If your team needs help, Optijara can help build that evaluation map before architecture choices turn into production cost.