Search

The Privacy Preserving Federated Learning Document VQA (PFL-DocVQA) competition challenged the community to develop provably private and communication-efficient solutions in a federated setting for a real-life use case: invoice processing. The competition introduced a dataset of real invoice documents, along with associated questions and answers requiring information extraction and reasoning over the document images. Thereby, it brings together researchers and expertise from the document analysis, privacy, and federated learning communities. Participants fine-tuned a pre-trained, state-of-the-art Document Visual Question Answering model provided by the organizers for this new domain, mimicking a typical federated invoice processing setup. The base model is a multi-modal generative language model, and sensitive information could be exposed through either the visual or textual input modality. Participants proposed elegant solutions to reduce communication costs while maintaining a minimum utility threshold in track 1 and to protect all information from each document provider using differential privacy in track 2. The competition served as a new testbed for developing and testing private federated learning methods, simultaneously raising awareness about privacy within the document image analysis and recognition community. Ultimately, the competition analysis provides best practices and recommendations for successfully running privacy-focused federated learning challenges in the future., Comment: 27 pages, 6 figures

Fonts are integral to creative endeavors, design processes, and artistic productions. The appropriate selection of a font can significantly enhance artwork and endow advertisements with a higher level of expressivity. Despite the availability of numerous diverse font designs online, traditional retrieval-based methods for font selection are increasingly being supplanted by generation-based approaches. These newer methods offer enhanced flexibility, catering to specific user preferences and capturing unique stylistic impressions. However, current impression font techniques based on Generative Adversarial Networks (GANs) necessitate the utilization of multiple auxiliary losses to provide guidance during generation. Furthermore, these methods commonly employ weighted summation for the fusion of impression-related keywords. This leads to generic vectors with the addition of more impression keywords, ultimately lacking in detail generation capacity. In this paper, we introduce a diffusion-based method, termed \ourmethod, to generate fonts that vividly embody specific impressions, utilizing an input consisting of a single letter and a set of descriptive impression keywords. The core innovation of \ourmethod lies in the development of dual cross-attention modules, which process the characteristics of the letters and impression keywords independently but synergistically, ensuring effective integration of both types of information. Our experimental results, conducted on the MyFonts dataset, affirm that this method is capable of producing realistic, vibrant, and high-fidelity fonts that are closely aligned with user specifications. This confirms the potential of our approach to revolutionize font generation by accommodating a broad spectrum of user-driven design requirements. Our code is publicly available at \url{https://github.com/leitro/GRIF-DM}., Comment: Accepted to ECAI2024

Document understanding models have recently demonstrated remarkable performance by leveraging extensive collections of user documents. However, since documents often contain large amounts of personal data, their usage can pose a threat to user privacy and weaken the bonds of trust between humans and AI services. In response to these concerns, legislation advocating ``the right to be forgotten" has recently been proposed, allowing users to request the removal of private information from computer systems and neural network models. A novel approach, known as machine unlearning, has emerged to make AI models forget about a particular class of data. In our research, we explore machine unlearning for document classification problems, representing, to the best of our knowledge, the first investigation into this area. Specifically, we consider a realistic scenario where a remote server houses a well-trained model and possesses only a small portion of training data. This setup is designed for efficient forgetting manipulation. This work represents a pioneering step towards the development of machine unlearning methods aimed at addressing privacy concerns in document analysis applications. Our code is publicly available at \url{https://github.com/leitro/MachineUnlearning-DocClassification}., Comment: Accepted to ICDAR2024

Documents are 2-dimensional carriers of written communication, and as such their interpretation requires a multi-modal approach where textual and visual information are efficiently combined. Document Visual Question Answering (Document VQA), due to this multi-modal nature, has garnered significant interest from both the document understanding and natural language processing communities. The state-of-the-art single-page Document VQA methods show impressive performance, yet in multi-page scenarios, these methods struggle. They have to concatenate all pages into one large page for processing, demanding substantial GPU resources, even for evaluation. In this work, we propose a novel method and efficient training strategy for multi-page Document VQA tasks. In particular, we employ a visual-only document representation, leveraging the encoder from a document understanding model, Pix2Struct. Our approach utilizes a self-attention scoring mechanism to generate relevance scores for each document page, enabling the retrieval of pertinent pages. This adaptation allows us to extend single-page Document VQA models to multi-page scenarios without constraints on the number of pages during evaluation, all with minimal demand for GPU resources. Our extensive experiments demonstrate not only achieving state-of-the-art performance without the need for Optical Character Recognition (OCR), but also sustained performance in scenarios extending to documents of nearly 800 pages compared to a maximum of 20 pages in the MP-DocVQA dataset. Our code is publicly available at \url{https://github.com/leitro/SelfAttnScoring-MPDocVQA}., Comment: Accepted to ICDAR2024

Document Visual Question Answering (DocVQA) has quickly grown into a central task of document understanding. But despite the fact that documents contain sensitive or copyrighted information, none of the current DocVQA methods offers strong privacy guarantees. In this work, we explore privacy in the domain of DocVQA for the first time, highlighting privacy issues in state of the art multi-modal LLM models used for DocVQA, and explore possible solutions. Specifically, we focus on invoice processing as a realistic document understanding scenario, and propose a large scale DocVQA dataset comprising invoice documents and associated questions and answers. We employ a federated learning scheme, that reflects the real-life distribution of documents in different businesses, and we explore the use case where the data of the invoice provider is the sensitive information to be protected. We demonstrate that non-private models tend to memorise, a behaviour that can lead to exposing private information. We then evaluate baseline training schemes employing federated learning and differential privacy in this multi-modal scenario, where the sensitive information might be exposed through either or both of the two input modalities: vision (document image) or language (OCR tokens). Finally, we design attacks exploiting the memorisation effect of the model, and demonstrate their effectiveness in probing a representative DocVQA models., Comment: 35 pages, 12 figures, accepted for publication at the 18th International Conference on Document Analysis and Recognition, ICDAR 2024

Speech Emotion Recognition (SER) is to recognize human emotions in a natural verbal interaction scenario with machines, which is considered as a challenging problem due to the ambiguous human emotions. Despite the recent progress in SER, state-of-the-art models struggle to achieve a satisfactory performance. We propose a self-attention based method with combined use of label-adaptive mixup and center loss. By adapting label probabilities in mixup and fitting center loss to the mixup training scheme, our proposed method achieves a superior performance to the state-of-the-art methods., Comment: Accepted to ICASSP 2023

Realization of giant and continuously tunable second-order photocurrent is desired for many nonlinear optical (NLO) and optoelectronic applications, which remains to be a great challenge. Here, based on a simple two-band model, we propose a concept of bulk electro-photovoltaic effect, that is, an out-of-plane external electric-field ($E_{ext}$) can continuously tune in-plane shift current along with its sign flip in a hetero-nodal-line (HNL) system. While strong linear optical transition around the nodal-loop may potentially generate giant shift current, an $E_{ext}$ can effectively control the radius of nodal-loop, which can continuously modulate the shift-vector components inside and outside nodal-loop holding opposite signs. This concept has been demonstrated in the HNL HSnN/MoS$_2$ system using first-principles calculations. The HSnN/MoS$_2$ hetero-bilayer not only can produce giant shift current with 1~2 magnitude order larger than other reported systems, but also can realize a giant bulk electro-photovoltaic effect. Our finding opens new routes to create and manipulate giant NLO responses in 2D materials.

One of the core concepts from the non-Hermitian skin effect is the extended complex wavevectors (CW) in the generalized Brillouin zone (GBZ), while the origin of CW remains elusive, and further experimental demonstration of GBZ is still lacking. We show that the bulk states of an open quantum system dynamically governed by the Lindblad master equation exhibit non-Bloch evolution which results in CW. Experimentally, we present temporal topolectrical circuits to serve as simulators for the dynamics of an open system. By reconstructing the correspondence between the bulk states of an open system and circuit voltage modes through gauge scale potentials in the circuit, the non-Bloch evolution is demonstrated. Facilitated by the simulators and proper approach to characterize the non-Bloch band proposed here, the GBZ is confirmed. Our work may advance the investigation of the dissipative topological modes and provide a versatile platform for exploring the unique evolution and topology for both closed and open systems., Comment: 5 figures

Handwritten Text Recognition has achieved an impressive performance in public benchmarks. However, due to the high inter- and intra-class variability between handwriting styles, such recognizers need to be trained using huge volumes of manually labeled training data. To alleviate this labor-consuming problem, synthetic data produced with TrueType fonts has been often used in the training loop to gain volume and augment the handwriting style variability. However, there is a significant style bias between synthetic and real data which hinders the improvement of recognition performance. To deal with such limitations, we propose a generative method for handwritten text-line images, which is conditioned on both visual appearance and textual content. Our method is able to produce long text-line samples with diverse handwriting styles. Once properly trained, our method can also be adapted to new target data by only accessing unlabeled text-line images to mimic handwritten styles and produce images with any textual content. Extensive experiments have been done on making use of the generated samples to boost Handwritten Text Recognition performance. Both qualitative and quantitative results demonstrate that the proposed approach outperforms the current state of the art., Comment: Accepted to TPAMI

Little is known about the role of weak interlayer coupling in the second harmonic generation (SHG) effects of two-dimensional van der Waals (vdW) systems. In this article, taking homo-bilayer $MoS_2/MoS_2$ and hetero-bilayer $MoS_2/MoSe_2$ as typical examples, we have systemically investigated their SHG susceptibilities as a function of interlayer hopping strength using first-principles calculations. For the SHG at zero frequency limit of both $MoS_2/MoS_2$ and $MoS_2/MoSe_2$, although the increase of tint can increase the intensities of interlayer optical transitions (IOT), the increased band repulsion around G point can eventually decrease their SHG values; the larger the tint, the smaller the SHG response. For the SHG spectra of $MoS_2/MoSe_2$ in the low photon-energy region, opposite to the $MoS_2/MoS_2$, their peak values are very sensitive to the variable tint, due to the strong $t_{int}$-dependent IOT dominating in the band edge; the larger the tint, the larger the SHG. For the SHG of $MoS_2/MoS_2$ in the high photon-energy region, comparing to the $MoS_2/MoSe_2$, their peak values will decrease in a much more noticeable way as the $t_{int}$ increases, due to the larger reduction of band nesting effect. Our study not only can successfully explain the puzzling experimental observations for the different SHG responses in different bilayer transition-metal dichalcogenides under variable tint, but also may provide a general understanding for designing controllable the SHG effects in the vdW systems.

Following our ab initio nonlinear optical (NLO) materials design guidelines, in this Letter, we discovered a novel type of structure to realize potential deep-ultraviolet (DUV) NLO performance in the classical beryllium borate system. By densely stacking the NLO-active layered frameworks, the key design scheme for the structural evolution from the (Be2BO3F2) layers in KBe2BO3F2 (KBBF) to the novel (Be2BO5H3) layers in berborite is illustrated. Based on available experimental results and systematical theoretical evaluation from first principles, the NLO properties of berborite are further obtained as comparable as the only pratical DUV NLO crystal KBBF. It is demonstrated that berborite can achieve available DUV phase-matched output with strong NLO effect for the practically important 177.3 nm and 193.7 nm lasers. Once obtained with sizable single crystal, it can be applied as a promising DUV NLO crystal., Comment: 10 pages, 3 figures

The recently discovered two-dimensional (2D) layered semiconductor MoSi2N4 has aroused great interest due to its unique 2D material characteristics. In this Letter, we found that differences in the structural details for MoSi2N4 may lead to differences in the intensity of second harmonic generation (SHG) and its response to strain. Accordingly, SHG can be used as a simple technique to identify the structural details of this system. We further calculated the SHG effects of MoSi2N4 derivatives and investigated their strain-regulation mechanism, especially including the anomalous SHG responses under strain for MoSi2P4 and MoGe2P4, differing from other known 2D materials. The studies may have forward-looking significance for the research of nonlinear optics and optoelectronics in this novel 2D material system., Comment: 7 pages, 4 figures

Although silicon oxynitrides are important semiconductors for many practical applications, their potential second-order nonlinear optical (NLO) applications, regardless of balanced or controllable performance, have never been systemically explored. Using the first-principles calculations, in this article, we discover that the sinoite (i.e., typical silicon oxynitride Si2N2O) can simultaneously exhibit wide optical bandgap, strong second-harmonic generation (SHG) effect, and large birefringence, which are further confirmed by our preliminary experimental data. Importantly, we propose that alloying engineering can be further applied to control the balanced NLO properties in the Si2N2O system. Combining first-principles calculations and cluster expansion theory, we demonstrate that alloying Ge into Si2N2O can easily form low formation energy Si2(1-x)Ge2xN2O alloys, which can in turn achieve controllable phase-matching harmonic output with high SHG efficiency at different energy ranges. Therefore, alloy engineering could provide a unique approach to effectively control the balanced NLO performance of Si2(1-x)Ge2xN2O, making this polar alloy system holding potential applications in tunable laser frequency conversion and controllable all-optical devices., Comment: 18 pages, 5 figures. Inorganic Chemistry, 2021

Two-dimensional (2D) layered semiconductors with both ultrawide bandgap and strong second harmonic generation (SHG) are essential for expanding the nonlinear optical (NLO) applications to deep-ultraviolet (DUV) region in nanoscale. Unfortunately, these materials are rare in nature and have not been discovered until now. In this Letter, we predict the B2S2O9 (BSO), an existing layered oxide, can exhibit both DUV bandgap and strong SHG effects, comparable to the best known DUV NLO bulks. The strong SHG intensities in BSO, originated from the ordered arrangement of polar SO4 and BO4 tetrahedra forming planar structure, are linearly tunable by the layer thickness. Surprisingly, the spontaneous rotations of rigid tetrahedra under strains can induce the (nearly) zero Poisson's ratios in BSO, which simultaneously result in the robust SHG effects against large strains, fundamentally differing from other known 2D NLO semiconductors. The discovery of BSO may provide an unprecedented opportunity to explore DUV NLO physics and applications in 2D limit.

The advent of recurrent neural networks for handwriting recognition marked an important milestone reaching impressive recognition accuracies despite the great variability that we observe across different writing styles. Sequential architectures are a perfect fit to model text lines, not only because of the inherent temporal aspect of text, but also to learn probability distributions over sequences of characters and words. However, using such recurrent paradigms comes at a cost at training stage, since their sequential pipelines prevent parallelization. In this work, we introduce a non-recurrent approach to recognize handwritten text by the use of transformer models. We propose a novel method that bypasses any recurrence. By using multi-head self-attention layers both at the visual and textual stages, we are able to tackle character recognition as well as to learn language-related dependencies of the character sequences to be decoded. Our model is unconstrained to any predefined vocabulary, being able to recognize out-of-vocabulary words, i.e. words that do not appear in the training vocabulary. We significantly advance over prior art and demonstrate that satisfactory recognition accuracies are yielded even in few-shot learning scenarios.

Although current image generation methods have reached impressive quality levels, they are still unable to produce plausible yet diverse images of handwritten words. On the contrary, when writing by hand, a great variability is observed across different writers, and even when analyzing words scribbled by the same individual, involuntary variations are conspicuous. In this work, we take a step closer to producing realistic and varied artificially rendered handwritten words. We propose a novel method that is able to produce credible handwritten word images by conditioning the generative process with both calligraphic style features and textual content. Our generator is guided by three complementary learning objectives: to produce realistic images, to imitate a certain handwriting style and to convey a specific textual content. Our model is unconstrained to any predefined vocabulary, being able to render whatever input word. Given a sample writer, it is also able to mimic its calligraphic features in a few-shot setup. We significantly advance over prior art and demonstrate with qualitative, quantitative and human-based evaluations the realistic aspect of our synthetically produced images., Comment: Accepted to ECCV2020

Sequence-to-sequence models have recently become very popular for tackling handwritten word recognition problems. However, how to effectively integrate an external language model into such recognizer is still a challenging problem. The main challenge faced when training a language model is to deal with the language model corpus which is usually different to the one used for training the handwritten word recognition system. Thus, the bias between both word corpora leads to incorrectness on the transcriptions, providing similar or even worse performances on the recognition task. In this work, we introduce Candidate Fusion, a novel way to integrate an external language model to a sequence-to-sequence architecture. Moreover, it provides suggestions from an external language knowledge, as a new input to the sequence-to-sequence recognizer. Hence, Candidate Fusion provides two improvements. On the one hand, the sequence-to-sequence recognizer has the flexibility not only to combine the information from itself and the language model, but also to choose the importance of the information provided by the language model. On the other hand, the external language model has the ability to adapt itself to the training corpus and even learn the most commonly errors produced from the recognizer. Finally, by conducting comprehensive experiments, the Candidate Fusion proves to outperform the state-of-the-art language models for handwritten word recognition tasks.