In brief

Document AI has moved well beyond classical character recognition. Modern models can process visual documents as a whole and transform layout, text and parts of their structure into machine-readable representations.

For IncluLearn.AI, that is an important foundation — but not the end goal. Reconstructing a scientific PDF as markup does not mean that the subject semantics of an embedded circuit, UML diagram or complex plot have been understood.

Pix2Struct: reconstructing visual structure as text

Pix2Struct was published at ICML in 2023. Among other pretraining tasks, the model learns to reconstruct masked web-page screenshots as simplified HTML.

The underlying idea is important: a visual document is not treated as pixels alone. Its visible structure is translated into a textual, machine-readable representation.

For IncluLearn.AI, this is especially relevant at the document level: layout, sections and visually situated content can be considered together.

Nougat: converting scientific PDFs into markup

Nougat was published at ICLR in 2024 and targets scientific documents. It processes PDF pages visually and generates markup, including mathematical content.

This addresses an important part of our problem. Scientific and technical PDFs often lose exactly the semantic structure that matters for machine processing and accessibility when they are reduced to a visual final format.

Even here, however, reconstructing document markup is not the same as fully understanding all embedded subject artefacts.

The limit of general Document AI

A document may contain:

  • ordinary prose,
  • headings and lists,
  • tables,
  • mathematical formulae,
  • diagrams,
  • circuits,
  • plots,
  • tasks that relate text to graphics.

A general model therefore does not necessarily need to solve every artefact down to its domain semantics.

A modular pipeline is more plausible for IncluLearn.AI:

source document → document structure → artefact classification → specialised semantic analysis → validation → human review → approved semantics

For example, a general model could detect that a region contains a UML diagram. Reconstruction of classes, attributes and relations could then be delegated to an analyser specialised for that domain.

Why the intermediate representation matters

If a model generates a finished long description, audio version or tactile template directly from a PDF, it becomes difficult to determine where a subject error entered the process.

An explicit semantic intermediate representation provides a reviewable point:

  • what was detected in the document?
  • which objects and relations were extracted?
  • which parts are uncertain?
  • what did a human correct?
  • which outputs were generated from it?

This is particularly important for accessibility because plausible but false statements can distort subject content.

Models should remain replaceable

Document AI and multimodal models are evolving quickly. A strong model today may be replaced soon.

IncluLearn.AI should therefore avoid tying too much lasting value to one model architecture. More durable assets are:

  • reviewed reference data,
  • neutral semantic schemas,
  • reproducible evaluation,
  • domain-specific rules,
  • human review processes,
  • accessible output forms.

Consequence for IncluLearn.AI

Document AI is not a competitor to the semantic-first approach. It is an input stage.

The research question therefore shifts from “Can the model read the text from the PDF?” to “Can the entire pipeline reconstruct and review subject structure reliably enough to generate several accessible representations from it?”