Search

The Privacy Preserving Federated Learning Document VQA (PFL-DocVQA) competition challenged the community to develop provably private and communication-efficient solutions in a federated setting for a real-life use case: invoice processing. The competition introduced a dataset of real invoice documents, along with associated questions and answers requiring information extraction and reasoning over the document images. Thereby, it brings together researchers and expertise from the document analysis, privacy, and federated learning communities. Participants fine-tuned a pre-trained, state-of-the-art Document Visual Question Answering model provided by the organizers for this new domain, mimicking a typical federated invoice processing setup. The base model is a multi-modal generative language model, and sensitive information could be exposed through either the visual or textual input modality. Participants proposed elegant solutions to reduce communication costs while maintaining a minimum utility threshold in track 1 and to protect all information from each document provider using differential privacy in track 2. The competition served as a new testbed for developing and testing private federated learning methods, simultaneously raising awareness about privacy within the document image analysis and recognition community. Ultimately, the competition analysis provides best practices and recommendations for successfully running privacy-focused federated learning challenges in the future., Comment: 27 pages, 6 figures

Fonts are integral to creative endeavors, design processes, and artistic productions. The appropriate selection of a font can significantly enhance artwork and endow advertisements with a higher level of expressivity. Despite the availability of numerous diverse font designs online, traditional retrieval-based methods for font selection are increasingly being supplanted by generation-based approaches. These newer methods offer enhanced flexibility, catering to specific user preferences and capturing unique stylistic impressions. However, current impression font techniques based on Generative Adversarial Networks (GANs) necessitate the utilization of multiple auxiliary losses to provide guidance during generation. Furthermore, these methods commonly employ weighted summation for the fusion of impression-related keywords. This leads to generic vectors with the addition of more impression keywords, ultimately lacking in detail generation capacity. In this paper, we introduce a diffusion-based method, termed \ourmethod, to generate fonts that vividly embody specific impressions, utilizing an input consisting of a single letter and a set of descriptive impression keywords. The core innovation of \ourmethod lies in the development of dual cross-attention modules, which process the characteristics of the letters and impression keywords independently but synergistically, ensuring effective integration of both types of information. Our experimental results, conducted on the MyFonts dataset, affirm that this method is capable of producing realistic, vibrant, and high-fidelity fonts that are closely aligned with user specifications. This confirms the potential of our approach to revolutionize font generation by accommodating a broad spectrum of user-driven design requirements. Our code is publicly available at \url{https://github.com/leitro/GRIF-DM}., Comment: Accepted to ECAI2024

Document understanding models have recently demonstrated remarkable performance by leveraging extensive collections of user documents. However, since documents often contain large amounts of personal data, their usage can pose a threat to user privacy and weaken the bonds of trust between humans and AI services. In response to these concerns, legislation advocating ``the right to be forgotten" has recently been proposed, allowing users to request the removal of private information from computer systems and neural network models. A novel approach, known as machine unlearning, has emerged to make AI models forget about a particular class of data. In our research, we explore machine unlearning for document classification problems, representing, to the best of our knowledge, the first investigation into this area. Specifically, we consider a realistic scenario where a remote server houses a well-trained model and possesses only a small portion of training data. This setup is designed for efficient forgetting manipulation. This work represents a pioneering step towards the development of machine unlearning methods aimed at addressing privacy concerns in document analysis applications. Our code is publicly available at \url{https://github.com/leitro/MachineUnlearning-DocClassification}., Comment: Accepted to ICDAR2024

Documents are 2-dimensional carriers of written communication, and as such their interpretation requires a multi-modal approach where textual and visual information are efficiently combined. Document Visual Question Answering (Document VQA), due to this multi-modal nature, has garnered significant interest from both the document understanding and natural language processing communities. The state-of-the-art single-page Document VQA methods show impressive performance, yet in multi-page scenarios, these methods struggle. They have to concatenate all pages into one large page for processing, demanding substantial GPU resources, even for evaluation. In this work, we propose a novel method and efficient training strategy for multi-page Document VQA tasks. In particular, we employ a visual-only document representation, leveraging the encoder from a document understanding model, Pix2Struct. Our approach utilizes a self-attention scoring mechanism to generate relevance scores for each document page, enabling the retrieval of pertinent pages. This adaptation allows us to extend single-page Document VQA models to multi-page scenarios without constraints on the number of pages during evaluation, all with minimal demand for GPU resources. Our extensive experiments demonstrate not only achieving state-of-the-art performance without the need for Optical Character Recognition (OCR), but also sustained performance in scenarios extending to documents of nearly 800 pages compared to a maximum of 20 pages in the MP-DocVQA dataset. Our code is publicly available at \url{https://github.com/leitro/SelfAttnScoring-MPDocVQA}., Comment: Accepted to ICDAR2024

Document Visual Question Answering (DocVQA) has quickly grown into a central task of document understanding. But despite the fact that documents contain sensitive or copyrighted information, none of the current DocVQA methods offers strong privacy guarantees. In this work, we explore privacy in the domain of DocVQA for the first time, highlighting privacy issues in state of the art multi-modal LLM models used for DocVQA, and explore possible solutions. Specifically, we focus on invoice processing as a realistic document understanding scenario, and propose a large scale DocVQA dataset comprising invoice documents and associated questions and answers. We employ a federated learning scheme, that reflects the real-life distribution of documents in different businesses, and we explore the use case where the data of the invoice provider is the sensitive information to be protected. We demonstrate that non-private models tend to memorise, a behaviour that can lead to exposing private information. We then evaluate baseline training schemes employing federated learning and differential privacy in this multi-modal scenario, where the sensitive information might be exposed through either or both of the two input modalities: vision (document image) or language (OCR tokens). Finally, we design attacks exploiting the memorisation effect of the model, and demonstrate their effectiveness in probing a representative DocVQA models., Comment: 35 pages, 12 figures, accepted for publication at the 18th International Conference on Document Analysis and Recognition, ICDAR 2024

Speech Emotion Recognition (SER) is to recognize human emotions in a natural verbal interaction scenario with machines, which is considered as a challenging problem due to the ambiguous human emotions. Despite the recent progress in SER, state-of-the-art models struggle to achieve a satisfactory performance. We propose a self-attention based method with combined use of label-adaptive mixup and center loss. By adapting label probabilities in mixup and fitting center loss to the mixup training scheme, our proposed method achieves a superior performance to the state-of-the-art methods., Comment: Accepted to ICASSP 2023

Realization of giant and continuously tunable second-order photocurrent is desired for many nonlinear optical (NLO) and optoelectronic applications, which remains to be a great challenge. Here, based on a simple two-band model, we propose a concept of bulk electro-photovoltaic effect, that is, an out-of-plane external electric-field ($E_{ext}$) can continuously tune in-plane shift current along with its sign flip in a hetero-nodal-line (HNL) system. While strong linear optical transition around the nodal-loop may potentially generate giant shift current, an $E_{ext}$ can effectively control the radius of nodal-loop, which can continuously modulate the shift-vector components inside and outside nodal-loop holding opposite signs. This concept has been demonstrated in the HNL HSnN/MoS$_2$ system using first-principles calculations. The HSnN/MoS$_2$ hetero-bilayer not only can produce giant shift current with 1~2 magnitude order larger than other reported systems, but also can realize a giant bulk electro-photovoltaic effect. Our finding opens new routes to create and manipulate giant NLO responses in 2D materials.

One of the core concepts from the non-Hermitian skin effect is the extended complex wavevectors (CW) in the generalized Brillouin zone (GBZ), while the origin of CW remains elusive, and further experimental demonstration of GBZ is still lacking. We show that the bulk states of an open quantum system dynamically governed by the Lindblad master equation exhibit non-Bloch evolution which results in CW. Experimentally, we present temporal topolectrical circuits to serve as simulators for the dynamics of an open system. By reconstructing the correspondence between the bulk states of an open system and circuit voltage modes through gauge scale potentials in the circuit, the non-Bloch evolution is demonstrated. Facilitated by the simulators and proper approach to characterize the non-Bloch band proposed here, the GBZ is confirmed. Our work may advance the investigation of the dissipative topological modes and provide a versatile platform for exploring the unique evolution and topology for both closed and open systems., Comment: 5 figures

Handwritten Text Recognition has achieved an impressive performance in public benchmarks. However, due to the high inter- and intra-class variability between handwriting styles, such recognizers need to be trained using huge volumes of manually labeled training data. To alleviate this labor-consuming problem, synthetic data produced with TrueType fonts has been often used in the training loop to gain volume and augment the handwriting style variability. However, there is a significant style bias between synthetic and real data which hinders the improvement of recognition performance. To deal with such limitations, we propose a generative method for handwritten text-line images, which is conditioned on both visual appearance and textual content. Our method is able to produce long text-line samples with diverse handwriting styles. Once properly trained, our method can also be adapted to new target data by only accessing unlabeled text-line images to mimic handwritten styles and produce images with any textual content. Extensive experiments have been done on making use of the generated samples to boost Handwritten Text Recognition performance. Both qualitative and quantitative results demonstrate that the proposed approach outperforms the current state of the art., Comment: Accepted to TPAMI

Little is known about the role of weak interlayer coupling in the second harmonic generation (SHG) effects of two-dimensional van der Waals (vdW) systems. In this article, taking homo-bilayer $MoS_2/MoS_2$ and hetero-bilayer $MoS_2/MoSe_2$ as typical examples, we have systemically investigated their SHG susceptibilities as a function of interlayer hopping strength using first-principles calculations. For the SHG at zero frequency limit of both $MoS_2/MoS_2$ and $MoS_2/MoSe_2$, although the increase of tint can increase the intensities of interlayer optical transitions (IOT), the increased band repulsion around G point can eventually decrease their SHG values; the larger the tint, the smaller the SHG response. For the SHG spectra of $MoS_2/MoSe_2$ in the low photon-energy region, opposite to the $MoS_2/MoS_2$, their peak values are very sensitive to the variable tint, due to the strong $t_{int}$-dependent IOT dominating in the band edge; the larger the tint, the larger the SHG. For the SHG of $MoS_2/MoS_2$ in the high photon-energy region, comparing to the $MoS_2/MoSe_2$, their peak values will decrease in a much more noticeable way as the $t_{int}$ increases, due to the larger reduction of band nesting effect. Our study not only can successfully explain the puzzling experimental observations for the different SHG responses in different bilayer transition-metal dichalcogenides under variable tint, but also may provide a general understanding for designing controllable the SHG effects in the vdW systems.