A Multiplexed Network for End-to-End, Multilingual OCR
Published in CVF/IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, 2021
Recommended citation: Jing Huang, Guan Pang, Rama Kovvuri, Mandy Toh, Kevin J Liang, Praveen Krishnan, Xi Yin, and Tal Hassner. A Multiplexed Network for End-to-End, Multilingual OCR. CVF/IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, 2021.
M3 TextSpotter. The proposed M3 TextSpotter shares the same detection and segmentation trunk with Mask TextSpotter v3 [Liao et al. ECCV’20], but incorporates a novel Language Prediction Network (LPN). The output of the LPN then determines which script’s recognition head the multiplexer selects.
Abstract
Recent advances in OCR have shown that an end-to-end (E2E) training pipeline that includes both detection and recognition leads to the best results. However, many existing methods focus primarily on Latin-alphabet languages, often even only case-insensitive English characters. In this paper, we propose an E2E approach, Multiplexed Multilingual Mask TextSpotter, that performs script identification at the word level and handles different scripts with different recognition heads, all while maintaining a unified loss that simultaneously optimizes script identification and multiple recognition heads. Experiments show that our method outperforms the single-head model with similar number of parameters in end-to-end recognition tasks, and achieves state-of-the-art results on MLT17 and MLT19 joint text detection and script identification benchmarks. We believe that our work is a step towards the end-to-end trainable and scalable multilingual multi-purpose OCR system. Our code and model will be released.