Infinity-Parser2 Technical Report

Abstract

We present Infinity-Parser2 , a large multimodal model that couples a controllable data-synthesis pipeline with multi-task reinforcement learning for end-to-end document parsing, addressing the persistent scarcity of faithfully annotated parsing corpora. Our contributions are threefold. First, we build a scalable synthesis engine, pairing a controllable rendering framework with an iterative refinement loop, and use it to construct and open-source Infinity-Doc2-5M : a 5-million-sample bilingual (Chinese/English) corpus spanning diverse document types, annotated with element bounding boxes, canonical content forms (Markdown, HTML, LaTeX, SMILES, structured charts), and full-page reading order. Second, we introduce a verifiable, multi-task reward system that enables Joint Reinforcement Learning across eight co-trained objectives (document parsing, layout analysis, table parsing, math formula parsing, chart parsing, chemical formula parsing, document VQA, and general multimodal understanding), unifying perception, structure, and reasoning in a single optimization signal. Third, we release two variants under a shared architecture: Infinity-Parser2-Flash , optimized for low-latency inference with a 3.68 × \times throughput gain over Infinity-Parser-7B, and Infinity-Parser2-Pro , engineered for precision-critical settings. Infinity-Parser2-Pro reaches state-of-the-art 87.6% on olmOCR-Bench and 74.3% on ParseBench, surpassing DeepSeek-OCR-2, PaddleOCR-VL-1.5, and MinerU2.5, with strong generalization to charts, chemical formulas, and document VQA.

Models  Dataset  Code  Demo

Contents

2.1 Pipeline-based Methods

3.1 Data Iteration Flywheel

4.1 Overall Training Strategy

5.1 Implementation Details

A.1 Evaluation Benchmarks

1 Introduction

Document parsing has emerged as a pivotal frontier in multimodal understanding, serving as the structural bridge that transforms unstructured visual content into machine-readable, semantically grounded representations. As Large Language Models (LLMs) evolve from passive responders into autonomous agents that ingest, reason over, and act upon real-world artifacts, the ability to faithfully interpret heterogeneous documents (from scientific papers and financial filings to invoices, slides, and chemistry notes) has become a prerequisite for downstream reasoning and decision-making. Consequently, the scope of the field has long outgrown classical Optical Character Recognition (OCR): it now demands holistic, page-level understanding that jointly handles layout analysis, fine-grained element parsing, globally consistent reading order, and structured information extraction. Yet attaining reliable, near-human accuracy across this stack remains formidable, owing to the combinatorial diversity of layouts, the density of cross-modal cues, and the inherently structured nature of the targets.

Despite rapid progress, two structural bottlenecks continue to constrain the field. First, on the data side, prevailing Supervised Fine-Tuning (SFT) recipes built on Vision-Language Models (VLMs)  [1, 2, 3] are bounded by the scarcity of large-scale, faithfully annotated parsing corpora that simultaneously cover heterogeneous layouts, fine-grained element semantics, and globally consistent reading order. As a result, SFT models that perform competitively in-distribution often degrade sharply on out-of-domain (OOD) templates, low-resource languages, or specialized verticals such as charts and chemical formulas. Second, on the optimization side, document parsing is intrinsically a multi-task problem (text recognition, layout grounding, table and formula structuring, chart decoding, and document-level reasoning are tightly coupled), yet existing pipelines typically treat these objectives as isolated heads or sequential stages. Even recent Reinforcement Learning (RL) approaches  [4, 5, 6] predominantly optimize a single, narrow textual signal, leaving structural fidelity, spatial alignment, and cross-task transfer under-exploited.

To surmount these limitations, we introduce Infinity-Parser2 , a large multimodal model that couples a controllable data-synthesis pipeline with multi-task reinforcement learning to deliver comprehensive, end-to-end document parsing. To break the data bottleneck, we build a scalable synthesis engine featuring a controllable rendering framework paired with an iterative refinement loop, and use it to construct Infinity-Doc2-5M , a 5-million-sample bilingual (Chinese/English) corpus spanning a broad spectrum of document types and layouts (including academic papers, research reports and financial reports, newspapers, textbooks, exam papers, and magazines), covering single- and multi-column layouts, mixed text–table–figure compositions, and embedded mathematical, tabular, and molecular content. Each sample is richly annotated with element bounding boxes, canonical content forms (Markdown, HTML, LaTeX, SMILES, structured charts), and full-page reading order, providing the structural priors that downstream training requires. To consolidate the fragmented optimization landscape, we further advance the post-training recipe with a verifiable, multi-task reward system that enables Joint Reinforcement Learning across eight co-trained objectives (document parsing, layout analysis, table parsing, math formula parsing, chart parsing, chemical formula parsing, document VQA, and general multimodal understanding), unifying perception, structure, and reasoning under a single, end-to-end optimization signal. Finally, to meet the disparate demands of real-world deployment, we release two model...

Extensive experiments across a broad suite of public benchmarks validate the effectiveness of Infinity-Parser2 (Figure 1). Our system establishes new state-of-the-art results on end-to-end document parsing, with Infinity-Parser2-Pro reaching 87.6% on olmOCR-Bench and 74.3% on ParseBench, surpassing strong contemporary systems including DeepSeek-OCR-2, PaddleOCR-VL-1.5, and MinerU2.5. The Flash variant delivers a 3.68 × \times throughput improvement (441 → \rightarrow 1,624 tokens/s) over our prior Infinity-Parser-7B while remaining competitive in accuracy, enabling efficient large-scale deployment. Beyond document-centric metrics, Infinity-Parser2 generalizes strongly to charts, chemical formulas, and document VQA, and retains robust general multimodal reasoning, evidencing the breadth of capabilities unlocked by joint multi-task RL.

Our primary contributions are summarized as follows:

  • •

We design a controllable rendering framework coupled with an iterative refinement pipeline to overcome the parsing-data bottleneck, and use it to construct and open-source Infinity-Doc2-5M , a 5-million-sample bilingual corpus spanning academic papers, research reports and financial reports, newspapers, textbooks, exam papers, and magazines, annotated with element-level bounding boxes, canonical content forms (Markdown, HTML, LaTeX, SMILES, structured charts), and full-page reading order.

  • •

We propose a verifiable, multi-task reward system that drives Joint Reinforcement Learning across eight co-trained objectives (document parsing, layout analysis, table parsing, math formula parsing, chart parsing, chemical formula parsing, document VQA, and general multimodal understanding), unifying perception, structure, and reasoning within a single optimization signal.

  • •

We release two deployment-aware variants under a shared architecture: Infinity-Parser2-Flash , achieving a 3.68 × \times throughput improvement over Infinity-Parser-7B, and Infinity-Parser2-Pro , attaining 87.6% on olmOCR-Bench and 74.3% on ParseBench, surpassing DeepSeek-OCR-2, PaddleOCR-VL-1.5, and MinerU2.5.

  • •

To foster community innovation, we release the complete suite of assets, including the Infinity-Doc2-5M dataset, the source code, and both model variants (Infinity-Parser2-Flash and Infinity-Parser2-Pro).

Document parsing has rapidly evolved into a core research direction at the intersection of vision, language, and structure understanding, with the goal of converting heterogeneous digital and scanned pages into faithful, machine-readable representations. Based on how the parsing capability is trained and assembled, prevailing approaches can be broadly grouped into three families: pipeline-based methods, end-to-end methods, and reinforcement learning (RL)-based methods.

2.1 Pipeline-based Methods

A long-standing line of work follows a pipeline-based paradigm. Systems such as MinerU2.5  [1] and PaddleOCR-VL  [2] decompose document parsing into a sequence of specialized stages: a layout analyzer first detects and classifies semantic regions (text blocks, tables, formulas, and figures), after which task-specific expert models recognize the content of each region in parallel, and an optional reading-order module reassembles the page into a structured format such as Markdown or HTML. This modular design offers favorable engineering ergonomics, supports targeted optimization of individual components, and enables high-throughput inference on standardized layouts. However, the very staging that grants modularity also introduces error accumulation along the pipeline: imperfect region proposals or misclassified blocks propagate to downstream recognizers and reading-order predictors, where they are difficult to recover. The reliance on hand-engineered interfaces between stages further limits adaptability to documents whose layouts deviate from the assumptions baked into each module.

2.2 End-to-end Methods

End-to-end approaches instead seek to subsume the full parsing stack within a single model. Representative systems such as the dots.ocr series  [7, 8] and the DeepSeek-OCR series  [3, 9] fine-tune large Vision-Language Models (VLMs) via supervised fine-tuning (SFT) to map a document image directly to its structured representation, jointly handling text recognition, layout grounding, table and formula structuring, and reading-order reconstruction. By learning holistic visual–textual representations, these models avoid hand-crafted inter-stage interfaces and achieve strong in-distribution accuracy. Their effectiveness, however, is tightly coupled to the diversity and fidelity of the training corpus: the scarcity of large-scale, faithfully annotated parsing data (covering heterogeneous layouts, fine-grained element semantics, and globally consistent reading order) remains a fundamental bottleneck, and SFT-only models often degrade noticeably on out-of-domain (OOD) templates, low-resource languages, or specialized verticals such as charts and chemical formulas.

2.3 RL-based Methods

To move beyond the limits of token-level imitation, a growing body of work explores reinforcement learning as a post-training strategy for document parsing. Approaches such as  [4, 5, 6, 10] optimize VLMs against outcome-oriented rewards that reflect the quality of the parsed output, and have demonstrated encouraging gains across document types. Nevertheless, most existing RL pipelines remain narrowly scoped: they optimize a single, predominantly textual reward signal and treat related sub-capabilities (element parsing, table and chart decoding, chemical formula recognition, and document-level question answering) as isolated heads or sequential stages, leaving structural fidelity, spatial alignment, and cross-task transfer under-exploited. In contrast, our Infinity-Parser2 introduces a verifiable, multi-task reward system that drives Joint Reinforcement Learning across eight co-trained objectives (document parsing, layout analysis, table parsing, math formula parsing, chart parsing, chemical formula parsing, document VQA, and general multimodal understanding), unifying perception, structure, and reasoning within a single end-to-end optimization signal and thereby promoting robust generalization across heterogeneous documents and downstream tasks.

3 Data Curation

Our method centers on three pillars that jointly drive the construction of Infinity-Parser2. First, we propose a data iteration flywheel (Sec. 3.1), a closed-loop methodology in which model evaluation, bad-case mining, data construction, and fine-tuning continuously reinforce one another. Second, to support the data construction step at scale, we develop a dedicated document data synthesis engine (Sec. 3.2) that produces layout-faithful documents through a three-stage pipeline. Third, we describe the resulting task-specific datasets (Sec. 3.3) used to train the model across structure analysis, element recognition, and document reasoning tasks. In short, Sec. 3.1 formalizes how data evolves with the model, Sec. 3.2 details what tool produces this data at scale, and Sec. 3.3 describes what data the model is ultimately trained on.

3.1 Data Iteration Flywheel

Overview. Data quality and data construction strategy play a critical role in multimodal document understanding. While raw document data are abundant, transforming them into high-quality, task-relevant training data under limited annotation and compute budgets remains a major challenge. Static data pipelines, which fix the dataset before training begins, cannot respond to the model’s evolving weaknesses and therefore tend to over-invest in already-mastered patterns while under-covering rare or difficult cases.

To address this, we propose a model-driven data iteration flywheel that tightly couples data construction with model behavior. Rather than statically defining sample importance, the flywheel dynamically estimates sample value based on the current model’s inference results and iteratively refines the dataset.

As illustrated in Figure 2, each turn of the flywheel comprises four stages: (1) Model Evaluation and Bad-case Analysis, (2) Data Collection and Mining, (3) Model Annotation and Data Synthesis, and (4) Model Fine-tuning and Iteration. We describe the details of each stage in the following sections.

Stage 1: Model Evaluation and Bad-case Analysis. At each flywheel iteration, the current parsing model  1 1 1 The first iteration is bootstrapped from off-the-shelf open-source Qwen3.5-2B and Qwen3.5-35B-A3B. Subsequent iterations use the most recently fine-tuned model as the new baseline. is evaluated on a multi-task benchmark suite covering end-to-end document parsing, layout analysis, and element-level parsing. Each benchmark is scored by its official script at two levels: a per-sample accuracy on each page and a per-subcategory accuracy aggregated along axes such as document type, layout complexity, and element class. The subcategory breakdown is essential because a single overall score can hide systematic failures on narrow slices, e.g., a model with strong average accuracy may still collapse on financial reports or multi-column layouts.

Bad cases are then mined through a ranked diagnostic pipeline. Samples and sub-categories are first sorted by accuracy, with the lowest-scoring tail retained as the candidate pool so that diagnostic effort focuses on the model’s empirically weakest regions. These candidates are inspected via a visualization workflow that overlays predictions onto the source page, surfacing document-level errors such as handwriting misrecognition, scan-induced degradation, and mis-handled domain-specific notation, element-level errors such as missed glyphs, fragmented formulas, and mis-parsed tables, along with layout-level errors such as reading-order flips, omissions in dense text, and mis-grouped regions. Diagnosed cases are consolidated into a three-axis weakness taxonomy along document type, element type, and layout pattern, with every case tagged on all three axes to yield a compact multi-label description of current weaknesses.

We emphasize that this pool serves a strictly diagnostic purpose: because its samples originate from held-out evaluation benchmarks, they are never reused as training data, which would conflate measurement with optimization and contaminate subsequent evaluations. Instead, the accumulated weakness tags act as a demand signal that flows into Stage 2, where targeted acquisition is performed on disjoint, training-safe sources matching the same weakness profile.

Stage 2: Data Collection and Mining. Given the weakness tags from Stage 1, this stage samples raw, unlabeled documents from source domains according to those tags. Each tag is first translated into one or more acquisition queries: document-type tags map to domain and source-channel filters, element-type tags to content-density signals that surface documents densely populated by the target structure, and layout tags to page-level structural filters. The resulting query set is dispatched in parallel across three complementary sources (targeted web crawling, public document datasets, and internal proprietary corpora), with each tag routed to whichever subset best matches its profile. After harvesting, the pool is re-projected onto the Stage 1 taxonomy. Tags falling below a target count trigger additional harvesting, and persistent gaps escalate to the next flywheel iteration.

This stage serves a strictly sourcing purpose, with no annotation or training-time use occurring here, decoupling raw-data supply from labeling so that each can be tuned independently. In the initialization round, roughly 2,000 samples are collected per tag. Subsequent rounds adaptively rebalance the budget toward tags that remain under-resolved.

Stage 3: Model Annotation and Data Synthesis. Given the raw, unlabeled pool from Stage 2, this stage converts it into training-ready supervision through two complementary paths: one anchored on harvested real documents, the other on targeted synthesis. The dual design reflects two constraints: per-sample human annotation does not scale to the flywheel cadence, and real-document harvesting alone leaves residual gaps on rare cells of the weakness taxonomy such as low-resource glyphs, atypical layouts, and structurally complex elements.

On the real-data path, each page first undergoes layout analysis into semantically coherent regions (text blocks, formulas, and tables), which are then routed to domain-specific expert models. We use dots.ocr  [7] for layout analysis, PaddleOCR-VL  [2] for text recognition, MinerU2.5  [1] for formula parsing, and Infinity-Parser  [4] for table parsing. Label quality is ensured by low-confidence-score filtering and handcrafted rule-based filtering.

On the synthesis path, we generate targeted synthetic documents for the rare layouts, low-resource languages, and under-represented elements that real-data mining cannot economically supply. The synthesis engine (whose corpus selection, layout templates, and rendering parameters are conditioned directly on the Stage 1 tags) is detailed in Sec. 3.2. Outputs from both paths converge into a newly annotated batch, which is mixed with historical training data accumulated from prior iterations and forwarded to Stage 4.

Stage 4: Model Fine-tuning and Iteration. Given the merged training corpus from Stage 3, this stage fine-tunes the current parsing model on the expanded dataset. Each round’s newly annotated data is appended to the existing training corpus rather than replacing it, so that fine-tuning at every iteration still includes the targeted examples from all prior rounds. This prevents the model from regressing on long-tail weaknesses that earlier iterations have already addressed. The fine-tuned model is then handed back to Stage 1 of the subsequent iteration as the new baseline, where the diagnostic cycle resumes on a fresh evaluation of the same multi-task benchmark suite.

Because the benchmark suite remains fixed across iterations, per-iteration scores are directly comparable and serve as the convergence signal. Looping through Stages 1–4 round after round, the flywheel progressively converts each iteration’s residual weaknesses into the next iteration’s targeted training data, tightening coverage on the long tail of document types, element classes, and layout patterns flagged in earlier rounds. Iterations continue until benchmark gains across consecutive rounds become negligible, at which point we consider the flywheel converged.

3.2 DOM-Based Document Synthesis Engine

Human annotation is prohibitively expensive, while labels produced by expert models inevitably introduce noise. Data synthesis thus offers a promising alternative for generating document data at scale. However, no existing synthesis paradigm fully satisfies the requirements of document data generation. Pixel-level, diffusion-based generation  [11, 12] is well suited to scene-text images yet fails to provide reliable structural labels for documents. Block-level engines  [13, 14, 15] synthesize individual elements (such as tables, charts, and chemical formulas) but cannot assemble complete page-level documents. LaTeX-source synthesis  [16, 17] is structurally rich but offers no fine-grained bounding-box annotations. HTML-based generation  [4, 6] yields page-level images but relies on rigid, hand-authored templates with limited layout diversity. These gaps call for a synthesis engine that produces diverse, page-level documents while preserving fine-grained structural annotations.

We therefore design a novel DOM-based (Document Object Model) synthesis engine that meets both requirements. As illustrated in Figure 3, the engine operates in three stages: corpus acquisition, configuration, and DOM file generation. Its core idea is to adopt the DOM as a unified intermediate representation that binds content, hierarchical structure, and geometric coordinates within a single tree. Conditioned on the acquired corpus and a sampled template, the DOM file generation stage instantiates documents along two complementary paths: a fixed-layout path that reproduces the layout of real document exemplars, and a flexible-layout path that composes diverse, content-elastic page layouts. Because the coordinates are carried by the DOM itself, the engine extracts fine-grained, multi-task labels automatically at rendering time, without any post-hoc annotation.

Corpus Acquisition. This stage gathers the raw content (plain text, structured elements, and visual assets) that populates the synthetic documents. General text comprises multilingual paragraphs drawn from public sources  [18] , internal databases, and model distillation, filtered by length, language (English, Simplified Chinese, Traditional Chinese, etc.), and domain (academic, financial, educational, etc.) to match downstream document types. Structured elements include tables, mathematical formulas, charts, and chemical formulas, represented as HTML, LaTeX, CSV, and SMILES strings collected from arXiv and public datasets  [19, 20, 15] . Visual assets consist of figures and images  [21] embedded as document illustrations.

Configuration. A template defines the global appearance of a document through two complementary groups of parameters:

  • •

Page-level parameters govern the canvas and its functional regions: page size and margins, rendering resolution, the writing mode (horizontal or vertical) and text direction (left-to-right or right-to-left), the layout mode of the main body (single-column, multi-column, and grid layout), and the placement of auxiliary regions such as headers, footers, page numbers, and rotated side bars.

  • •

Element-level parameters specify the visual style of each content type (body text, multi-level headings, ordered and unordered lists, tables, figures, inline and block equations, code, algorithms, and references), covering typography (font family and size, color, line height, alignment, indentation, and spacing) together with a block-flow attribute that determines how each block wraps and breaks across columns and pages.

Rather than fixing these parameters, each template stores a preset default value together with a small perturbation range for every parameter. At generation time, we jitter the defaults (font sizes, margins, line spacing, colors, and region geometry) within these bounds, so that a single template instantiates into a family of documents that share the same logical structure yet differ in fine-grained appearance. This turns every template into a source of controlled data augmentation, substantially enlarging visual diversity while keeping each layout typographically valid. Moreover, the perturbation ranges need not be uniform: when the engine is driven by our data flywheel (Sec. 3.1), they can be biased toward the layout and document-type weaknesses recorded in its weakness taxonomy, so that augmentation concentrates on the regions where the parser currently underperforms. Maintaining a library of such templates across common document types (academic papers, financial reports, books, letters, and the like) lets the engine cover a broad space of real-world layouts.

DOM File Generation. Given the acquired corpus and a sampled template, this stage instantiates documents along two complementary paths. The fixed-layout path targets document types whose visual form must closely match real specimens, such as financial reports and forms. It takes a real page as a layout exemplar, uses a vision-language model to recover its region bounding boxes and attributes, and injects sampled content and template styles into that layout, producing documents that preserve the exemplar’s structure while varying its content. The flexible-layout path instead composes each document from scratch, letting page count and element placement emerge from the content so as to cover a broad space of multi-column, multi-page layouts.

In the flexible-layout path, the engine first instantiates a complete DOM tree that serves as the document’s logical intermediate representation. Every node is typed against a single element schema (paragraph, multi-level heading, list, table, figure, inline or block formula, code, header, footer, and side bar), and this type later drives renderer dispatch. The schema is open, so new modalities such as charts and chemical formulas are introduced by registering a node type and its renderer rather than by modifying the pipeline. Each node carries three pieces of information:

  • •

its content , retained in the native source form of each modality (plain text, LaTeX for formulas, HTML for tables, SMILES for molecules, etc.), so that nothing is lost to a lossy intermediate encoding.

  • •

its structural role (heading level, reading order, and parent–child containment), which makes the document hierarchy explicit.

  • •

a style descriptor sampled from the template (typography, alignment, spacing), together with a block-flow constraint that declares how the node may be paginated: flexible (freely splittable, e.g. body text, headings, and code), continuous (splittable but order-preserving, e.g. tables and lists), or atomic (indivisible, e.g. a figure and its caption).

Crucially, the DOM emitted at this point is purely logical: it fixes content, hierarchy, and appearance but leaves all geometry unresolved, deferring coordinate computation to rendering, which repopulates the same tree with exact positions. The DOM is thus a single source of truth from which one traversal yields ground-truth annotations for multiple downstream tasks: reading order, element-type and hierarchy labels, and structured text (Markdown/LaTeX/HTML) reconstructed directly from node content, without any additional labeling step. This explicit encoding of structural relations is precisely why we adopt a DOM rather than a flatter intermediate such as a Markdown or LaTeX string, from which hierarchy, reading order, and element boundaries would otherwise have to be recovered post hoc.

The flexible path then resolves this logical DOM into a concrete, paginated document, rasterizes it, and reads every label directly off the laid-out tree. Rather than predicting positions, the engine delegates typesetting to a real browser layout engine and then reads back exact coordinates, so that all geometric labels are correct by construction. Three tightly coupled components carry out this rendering.

Layout planning. The page canvas is first divided into functional regions (a main body together with headers, footers, and side bars), and the body is partitioned into a configurable arrangement of columns, rows, or grid cells, with cell extents taken from template ratios or jittered for diversity. The resulting regions are linearized into a reading sequence consistent with the document’s writing mode (horizontal or vertical) and text direction (left-to-right or right-to-left), so that reading-order labels follow directly from the plan rather than being inferred from pixels.

Content-elastic pagination. Elements are streamed into the planned regions one at a time, so that the page count emerges from the content itself rather than being fixed in advance. Before being committed, each element is rendered off-screen on a hidden measurement canvas and measured at its true rendered size, so that placement never relies on size estimates. Comparing this measured extent against the space left in the current region yields one of three outcomes (Figure 3). An element that fits entirely is placed in position ( FULL_FIT ). An element that fits only partially is broken at a measured point and continues at the top of the next region, for instance the second column of the same page ( SPLIT ). An element that cannot be accommodated in the remaining space is deferred to the next region or page ( OVERFLOW ). How a block may be broken follows its block-flow constraint. Flexible blocks break line by line across regions, continuous blocks break while preserving order with table headers repeated on every continuation, and atomic blocks are never broken. When an atomic block overflows, the engine performs a backward reflow, either re-laying the most recently placed blocks to reclaim room on the current page or deferring the block to the next page, which removes orphaned fragments while preserving the global reading order.

Element pre-rendering. Complex elements pass through dedicated sub-renderers before measurement, so that their true footprint, rather than an approximation, drives layout. Formulas are typeset to SVG by a LaTeX engine and synchronized on web-font readiness, with display equations assigned document-global numbering. Tables are rendered from HTML/CSS inside an isolated shadow DOM, their logical cell grid (including row and column spans) recovered, and cross-page cut points located by binary search over measured geometry within the boundary cell, under typographic safeguards that forbid mid-word breaks, line-leading punctuation, and orphaned sub/superscripts. Images and custom fonts (CJK, Latin, and mixed scripts) are fully loaded before measurement to prevent reflow. Dispatch is keyed on element type and routed through a shared library manager, so the engine extends to additional notation renderers, such as charts and chemical formulas, by registering a renderer rather than altering the pipeline.

Finally, each laid-out page is rasterized at high resolution by device-pixel supersampling (optionally composited over template-controlled background textures for visual variability), and, in lockstep with the image, per-task labels are exported by traversing the laid-out DOM: character-, line-, and cell-level bounding boxes obtained from the browser’s native geometry queries and offset into absolute page coordinates, alongside reading-order, element-type, hierarchy, and structured-text labels. Because the pixels and the annotations are read from one and the same DOM, they are exact and mutually consistent by construction, requiring no post-hoc detection or OCR.

3.3 Dataset Preparation

This section describes the final training data composition of Infinity-Parser2, assembled after the data iteration flywheel (Sec. 3.1) has converged. Each task draws from three sources: public datasets, real documents mined by the flywheel, and synthetic documents produced by the synthesis engine (Sec. 3.2). For each task we report its definition, data sources, scale, annotation or synthesis strategy, and output format. A consolidated overview is given in Table 1.

Document Structure Tasks. These tasks operate on the document as a whole and produce structural representations.

  • •

Document Parsing. Given an input document image, the model transforms it into a structured representation by identifying and ordering document elements according to the natural reading order, annotating each detected region with its element type, content, and spatial location, thereby converting unstructured pages into structured data. For the doc2json task, the training data combine real documents mined by the flywheel with synthetic documents produced by the synthesis engine: we mine 776K documents from the web spanning nine categories (exam papers, slides, academic papers, books, textbooks, magazines, notes, newspapers, and financial reports) and obtain high-quality pseudo labels for them through Stage 3 of our flywheel, manually annotate 2.5K complex newspaper images, and synthesize an additional 251K documents with our synthesis engine. For the doc2md task, we sample 57K high-quality examples from Infinity-Doc-400K  [4] and mine 62K scanned handwritten image–text pairs from the Library of Congress digital archives.

  • •

Layout Analysis. Layout analysis detects document elements in reading order and predicts their types and bounding boxes without recognizing their content. The training data comprise 4K samples from the publicly available M 6 Doc dataset  [22] , supplemented with 30K samples from our synthesis engine, for which layout labels are obtained directly from the DOM at near-zero cost. The outputs follow a standard layout-element schema covering titles, paragraphs, formulas, tables, figures, captions, headers, and footers.

Element-level Parsing Tasks. These tasks operate on individual document elements and produce element-specific content.

  • •

Table Parsing. Table parsing focuses on recognizing the content of table elements and producing structured sequences such as HTML and Markdown. For the table2html task, we collect 676K samples from PubTabNet  [19] , FinTabNet  [23] , and MMTab-HTML  [24] . For the table2md task, we obtain 27K samples from MMTab-MD  [24] and generate an additional 309K samples with our synthesis engine.

  • •

Math Formula Parsing. Math formula parsing recognizes the content of mathematical formula elements and produces structured LaTeX sequences. We collect 1.1M samples from im2latex  [25] , UniMER  [20] , HME  [26] , and CROHME  [27] .

  • •

Chart Parsing. Chart parsing converts visual charts (bar, line, pie, scatter, etc.) into structured data records and Python code. For the chart2table task, we collect 1.5M samples from ChartSFT  [28] , Chart-MoE  [29] , and UniChart  [30] . For the chart2json task, we obtain 900K samples from Chart-MoE  [29] and ChartQA  [31] . For the chart2code task, we collect 1.2M samples from ChartGen  [32] , Chart2Code  [33] , and Chart-MoE  [29] .

  • •

Chemical Formula Parsing. Chemical formula parsing recognizes 2D chemical formula diagrams and outputs their SMILES representations. Because real-world annotated data are scarce, this task relies heavily on our synthesis engine, which renders chemical formulas from SMILES corpora into diagram images with controlled visual styles. Using four rendering backends (CDK  [49] , RDKit  [50] , OpenChemLib  [51] , and Indigo  [52] ) and a corpus drawn from DECIMER  [15] , we generate 5.9M samples.

Reasoning and Generalization Tasks. These tasks go beyond per-element recognition and require document-level understanding or cross-domain generalization.

  • •

Document VQA. Document VQA requires answering natural-language questions grounded in a document image. We collect 1.6M samples from DocVQA  [34] , ChartQA  [31] , AI2D  [35] , DocReason25K  [36] , InfoVQA  [37] , DT-VQA  [38] , and TinyChart  [39] .

  • •

General Multimodal Understanding. To preserve the open-domain vision-language alignment of the underlying VLM and mitigate catastrophic forgetting during document-heavy fine-tuning, we additionally include general image–text pairs and instruction data. In total, we collect 3.2M samples from public multimodal corpora, including M4-Instruct  [40] , LLaVA-v1.5  [41] , ShareGPT-4V  [42] , ShareGPT-4o  [43] , CogVLM  [44] , ALLaVA-Instruct  [45] , LVIS-Instruct  [46] , TextVQA  [47] , OCRVQA  [48] , and AnyWord-3M  [12] .

Blank-page Handling. Beyond the task-specific data above, we additionally curate 5K blank-page samples spanning varying resolutions, background colors, and background textures. For inputs containing no valid content, the model is trained to emit task-specific empty outputs (e.g., empty Markdown, empty JSON array). This explicitly suppresses hallucinations on degenerate inputs and improves system robustness.

Dataset Summary. Overall, the final training set (Table 1) combines public, flywheel-mined, and synthesis-engine data across structure-level, element-level, and reasoning tasks, comprising approximately 5M samples after balanced sampling. The sampling ratios are designed to ensure balanced coverage along the multi-attribute schema introduced in Sec. 3.1, and to remain consistent with the bad-case distribution observed at flywheel convergence. To foster further progress in document parsing, after removing business-sensitive and privacy-related samples, we open-source the Infinity-Doc2-5M dataset at https://huggingface.co/datasets/infly/Infinity-Doc2-5M.

4 Model Training

4.1 Overall Training Strategy

We formulate document parsing as a unified image-to-sequence generation problem. Given a document image I I and a task instruction c c that specifies the target task and output format, the model π θ \pi_{\theta} autoregressively generates a token sequence y = ( y 1 , … , y T ) ∼ π θ ( ⋅ ∣ I , c ) y=(y_{1},\dots,y_{T})\sim\pi_{\theta}(\cdot\mid I,c) that decodes into a structured representation [ e 1 , … , e N ] [e_{1},\dots,e_{N}] , an ordered list of document elements rendered in Markdown, HTML, LaTeX, JSON, etc., where each e i e_{i} carries the i i -th element’s type, content, and spatial location (where applicable). A single parameter set θ \theta thus serves the entire parsing stack (document-, element-, and reasoning-level tasks) under one autoregressive interface, with the task selected solely through c c .

As illustrated in Figure 4, training proceeds in two stages. We first perform supervised fine-tuning (SFT) to instill broad parsing capability through next-token prediction, and then conduct joint reinforcement learning with verifiable rewards (RLVR) to align the model with task-native evaluation metrics and to close the gap between teacher-forced training and autoregressive inference. This SFT-then-RLVR recipe follows recent document-parsing systems  [4, 6] . Our contribution lies in the design of the reward system that drives the RL stage rather than in the recipe itself. Concretely, our training design rests on three choices:

  • •

Unified multi-task RLVR. We co-train eight objectives (document parsing, layout analysis, table parsing, math formula parsing, chart parsing, chemical formula parsing, document VQA, and general multimodal understanding) within a single RL stage, rewarding each with a structure-aware metric native to that task (Sec. 4.3.2).

  • •

Disentangled spatial–textual reward. For structured-layout parsing, we introduce a reward that scores element localization and content fidelity separately, a signal absent from prior edit-distance- or unit-test-based rewards (Sec. 4.3.2).

  • •

Reward routing with normalization. All objectives are optimized jointly through per-task reward routing and cross-task reward normalization, so that a single policy is trained over heterogeneous tasks with comparable gradient scales (Sec. 4.3.3).

Under this recipe we train two deployable variants on a shared architecture (Infinity-Parser2-Flash for low-latency inference and Infinity-Parser2-Pro for precision-critical settings), whose configurations are detailed in our experimental setup.

4.2 Supervised Fine-Tuning

We adopt Qwen3.5  [53] as our foundation model, a vision–language model pre-trained on large-scale multimodal data with strong inherent document-understanding capability. We fine-tune it on Infinity-Doc2-5M (Sec. 3.3) with a supervised next-token-prediction objective:

where 𝒟 \mathcal{D} denotes the training corpus and each triple ( I , c , y ) (I,c,y) pairs an input image and task instruction with its target sequence. The SFT corpus spans all sub-tasks, including the full general-multimodal-understanding set, which preserves the open-domain vision–language alignment of the base model and mitigates catastrophic forgetting during document-heavy fine-tuning. This stage endows the model with broad multi-task parsing competence and provides a strong initialization for the subsequent RL stage.

4.3 Joint Reinforcement Learning with Verifiable Rewards

4.3.1 Motivation

The next-token-prediction objective of SFT, trained with teacher forcing, optimizes the model under a ground-truth-conditioned distribution that differs from the autoregressive distribution encountered at inference. This exposure bias encourages the model to overfit the sequential dependencies and annotation noise of the training data, limiting its generalization to unseen document types. To bridge this train–inference gap, we add a reinforcement learning stage in which the model generates full sequences (rollouts) and is optimized against task-specific reward signals, promoting robust generation strategies that transfer better to out-of-distribution document formats. The effectiveness of this stage hinges entirely on the reward signals, whose design we describe next.

4.3.2 Reward Design

All RL signals in Infinity-Parser2 are verifiable rewards: each is computed by a deterministic, reference-based metric rather than a learned reward model, which keeps the signal faithful to ground truth and structurally limits reward hacking. We adopt a single guiding principle, metric-as-reward, under which every task is rewarded by its own native evaluation metric, so that RL optimizes directly for the quantity by which the task is ultimately judged. All rewards are normalized to [ 0 , 1 ] [0,1] to keep gradient magnitudes comparable across co-trained tasks (Sec. 4.3.3). When an output jointly encodes multiple facets (most notably structured-layout parsing, which couples element localization with textual content), a single scalar metric cannot separate distinct error modes. We therefore factorize the reward into facet-specific terms, as detailed below.

Disentangled Spatial–Textual Reward for Structured Parsing.

The doc2json task produces a joint detection–recognition structure: an ordered list of elements E i = ( c i , b i , t i ) E_{i}=(c_{i},b_{i},t_{i}) , each carrying a category c i c_{i} , a bounding box b i b_{i} , and textual content t i t_{i} . Scoring such an output with a single edit distance over its serialized form is ill-suited on two counts: it conflates where an element is with what it contains, yielding an ambiguous learning signal, and it measures geometric error through coordinate digit strings, where a few-pixel shift can perturb the reward disproportionately. We therefore disentangle the reward into a spatial term and a textual term, each evaluated in its native space.

Spatial reward. Matching predicted boxes to references requires a fragile assignment step (e.g., Hungarian matching) under an IoU threshold, and it further breaks down when the two sides are annotated at inconsistent granularity: a single reference block may be predicted as several finer boxes (one-to-N), or several reference blocks may be covered by one prediction (N-to-one), leaving no unambiguous one-to-one correspondence. We therefore avoid box matching altogether and instead score localization at the level of per-category occupied area. For each category k k , we merge all predicted boxes of that category into a single region P k P^{k} by geometric union, and likewise form G k G^{k} from the references. The spatial reward is the mean area Intersection-over-Union (mIoU)  [54] over the categories that appear in the prediction or the reference, K = K pred ∪ K gt K=K_{\mathrm{pred}}\cup K_{\mathrm{gt}} :

where | ⋅ | \lvert\cdot\rvert denotes area and | K | \lvert K\rvert the number of categories. A category present on only one side has an empty intersection and thus contributes 0 . The unions and intersections are evaluated exactly from the box geometry, so the spatial reward requires no rasterization or resolution parameter. This region-overlap formulation is assignment-free and order-invariant at the box level, and remains robust to mis-counting: spurious predictions enlarge the union while missed regions shrink the intersection, both lowering the score without any explicit box matching.

Textual reward. The element contents are concatenated in the predicted reading order into a single Markdown string s pred s_{\mathrm{pred}} and scored against the reference s gt s_{\mathrm{gt}} by Edit Distance Similarity, r text = EDS ​ ( s pred , s gt ) r_{\mathrm{text}}=\mathrm{EDS}(s_{\mathrm{pred}},s_{\mathrm{gt}}) . Because the concatenation follows reading order, this term additionally penalizes reading-order errors, so localization, content fidelity, and reading order are jointly covered by the two terms without a dedicated ordering reward. Using EDS also keeps the content signal consistent with the doc2md task.

Combination. The two terms are linearly combined,

with both components in [ 0 , 1 ] [0,1] . We report the weight λ \lambda in our experimental setup. Predictions that cannot be parsed into the element list receive zero reward. Unlike prior RLVR-based parsers that reward only flat textual output (Infinity-Parser  [4] couples edit distance with paragraph-count and reading-order terms but supplies no localization signal, while olmOCR-2  [6] scores unit tests on rendered text rather than on coordinate-bearing structured output), this reward acts directly on the structured prediction, separately crediting where each element is and what it contains.

Reward Suite over the Co-trained Objectives.

We extend the same metric-as-reward principle across all eight co-trained objectives:

  • •

Document parsing. doc2md is rewarded by EDS and doc2json by the spatial–textual reward above.

  • •

Layout analysis. Layout analysis, which predicts typed boxes without content, is rewarded by the spatial term r spatial r_{\mathrm{spatial}} alone.

  • •

Table parsing. We use the Tree-Edit-Distance-based Similarity (TEDS)  [55] for both table2html and table2md, reflecting the hierarchical structure of tables.

  • •

Math formula parsing. We use the Character Detection Matching (CDM) metric  [56] for formula2latex, which compares rendered formulas and is therefore robust to syntactically different but visually equivalent LaTeX.

  • •

Chart parsing. We adopt the Relative Mapping Similarity F1 (RMS-F1)  [57] for chart2table, the mean Average Precision from the Structuring Chart-oriented Representation Metric (SCRM)  [58] for chart2json, and EDS for chart2code, treating the generated plotting code as a string.

  • •

Chemical formula parsing. For chem2smiles we use the Tanimoto similarity  [59] between the predicted and reference molecular fingerprints.

  • •

Document VQA. For docvqa we use the Average Normalized Levenshtein Similarity (ANLS).

  • •

General multimodal understanding. We apply RLVR only to the verifiable subset of this objective (samples that carry reference short answers), scored by ANLS. Open-ended samples are excluded from RL and retained only in SFT.

4.3.3 Optimization

We optimize the policy with Group Relative Policy Optimization (GRPO)  [60] , which dispenses with a learned value model. For each input ( I , c ) (I,c) , the current policy samples a group of G G rollouts { y ( i ) } i = 1 G \{y^{(i)}\}_{i=1}^{G} , each scored by its task reward r ( i ) ∈ [ 0 , 1 ] r^{(i)}\in[0,1] . The group-relative advantage is obtained by normalizing rewards within the group,

The policy is then updated by maximizing the standard PPO-style clipped surrogate with these advantages, regularized by a KL penalty toward a frozen reference policy. Because every reward is verifiable, the advantage reflects genuine task quality rather than a learned proxy.

To co-train heterogeneous tasks in a single run, a reward router dispatches each rollout to the reward function of its task, while the within-group normalization places all rewards on a common [ 0 , 1 ] [0,1] scale, keeping advantages comparable across tasks within a mixed batch. The RL training set is constructed by randomly sampling 5% of each task from Infinity-Doc2-5M, yielding roughly 220K examples. Its detailed composition and the optimization hyperparameters are reported in our experimental setup.

5 Experiments

5.1 Implementation Details

  • •

Experimental Setup. Our approach builds upon the Qwen3.5 architecture  [53] . We train Infinity-Parser2-Pro based on Qwen3.5-35B-A3B, and train Infinity-Parser2-Flash based on Qwen3.5-2B. All experiments are performed on a cluster of 64 NVIDIA H100 GPUs, utilizing PyTorch 2.10.0 and CUDA 12.8. We employ Megatron-LM-0.16.0  [61] for distributed model parallel training.

  • •

Supervised Fine-Tuning (SFT) Stage. Both variants are trained for one epoch with the ms-swift  [62] framework on its Megatron-LM backend. We use a maximum sequence length of 32,768 with multimodal sequence packing, encode each page image into at most 16,384 visual tokens, and keep all modules (the ViT encoder, the vision–language aligner, and the LLM backbone) trainable. We adopt the Adam optimizer with a micro batch size of 1 and a global batch size of 64. The learning rate follows a cosine schedule that warms up over 3 % 3\% of the steps to a peak of 1 × 10 − 5 1\times 10^{-5} and then decays to 1 × 10 − 6 1\times 10^{-6} . Training is conducted in non-thinking mode. The two variants differ only in parallelism. Infinity-Parser2-Flash uses tensor parallel size 2, pipeline parallel size 1, and context parallel size 1. The MoE-based Infinity-Parser2-Pro uses tensor parallel size 8, expert parallel size 4, pipeline parallel size 1, and context parallel size 1, with grouped-GEMM expert computation, shared-expert overlap, and an MoE auxiliary-loss coefficient of 1 × 10 − 6 1\times 10^{-6} . Sequence parallelism and full activation recomputation are enabled throughout.

  • •

Reinforcement Learning (RL) Stage. We refine both SFT checkpoints with GRPO (Sec. 4.3.3) using the VeRL  [63] framework on its Megatron-LM backend, for one epoch with a learning rate of 1 × 10 − 6 1\times 10^{-6} . We draw 8 8 rollouts per prompt with an asynchronous vLLM engine (generation tensor parallel size 4, GPU memory utilization 0.5 0.5 ), apply a KL loss (low-variance estimator, coefficient 0.01 0.01 ), and set the entropy coefficient to 0 . For the disentangled doc2json reward (Sec. 4.3.2), the spatial and textual terms are weighted by λ = 0.3 \lambda=0.3 and 1 − λ = 0.7 1-\lambda=0.7 , respectively. The two variants differ in batch size, sequence budget, and parallelism. Infinity-Parser2-Flash uses a rollout batch size and mini-batch size of 32, maximum prompt and response lengths of 8,192 and 16,384, and tensor parallel size 2 with pipeline and context parallel sizes of 1. Infinity-Parser2-Pro uses a rollout batch size and mini-batch size of 48, maximum prompt and response lengths of 8,192, and tensor parallel size 2, context parallel size 2, expert parallel size 8, and pipeline parallel size 1, with MoE auxiliary- and z-loss coefficients of 1 × 10 − 2 1\times 10^{-2} and 1 × 10 − 3 1\times 10^{-3} . Paramete...