Synthetic Document Generation for Document Attention Network

For many years, document image analysis and recognition were considered complex tasks decomposed in two independent sub-tasks: layout analysis and text recognition. Layout analysis aims at detecting every image components such as text blocs, tables, images, graphics, signatures... Text recognition is devoted to the recognition of text inside text blocs (paragraphs). Recently, deep architectures have improved the state-of-the-art performance for both sub-tasks. But training each task independently requires twice the amount of annotation: the position (bounding box) of each element in the image; the transcription of text blocs with text line bounding boxes. Moreover, by training the two components independently there is no guaranty that the sequential pipeline is optimum. In any case, the pipeline cumulates the errors of each component. In addition, when moving from one type of document to another, each component requires its specific optimization/adaptation phase.

To overcome these problems and ease the training and adaptation phase of the processing pipeline, we proposed the Document Attention Network that achieves the document processing pipeline end-to-end for the first time (https://arxiv.org/abs/2203.12273). The system, composed of a fully convolutional encoder and a transformer decoder, is trained without any segmentation ground truth. The document structure is encoded with specific TAGS so that the whole document transcription is an Xml-like document with layout TAGS (see figure below). The system is trained to read the document in a chosen reading order that is defined in the ground truth. The complex DAN architecture is pre-trained on synthetic documents first, and then trained on real data following a curriculum learning strategy. The DAN achieves state of the art performance but is very dependent of its pre-training phase, and of the quality of the synthetic training examples generated.

This internship will explore some possible strategies to generate efficient training samples for the DAN.