Analyzing the Potential of Zero-Shot Recognition for Document Image Classification

Document image classification is one of the most important components in business automation workflow. Therefore, a range of different supervised image classification methods have been proposed, which rely on a large amount of labeled data, which is rarely available in practice. Furthermore, retraining of these models is necessary upon the introduction of new classes. In this paper, we analyze the potential of zero-shot document image classification based on computing the agreement between the images and the textual embeddings of class names/descriptions. This enables the deployment of document image classification models without the availability of any training data at zero training cost, alongside providing seamless integration of new classes. Our results show that using zero-shot recognition achieves significantly better than chance performance on document image classification benchmarks (49.51% accuracy on Tobacco-3482 in contrast to 10% random classifier accuracy and 39.22% on RVL-CDIP dataset in contrast to 6.25% random classifier accuracy). We also show that the representation learned by a vision transformer using image-text pairs is competitive to CNNs by training a linear SVM on top of the pre-computed representations which achieves comparable performance to state-of-the-art convolutional networks (85.74% on Tobacco-3482 dataset in contrast to 71.67% from ImageNet pretrained ResNet-50). Even though the initial results look encouraging, there is still a large gap to cover for zero-shot recognition within the domain of document images in contrast to natural scene image classification, which achieves comparable performance to fully supervised baselines. Our preliminary findings pave the way for the deployment of zero-shot document image classification in production settings.