Search

Implicit graph neural networks have gained popularity in recent years as they capture long-range dependencies while improving predictive performance in static graphs. Despite the tussle between performance degradation due to the oversmoothing of learned embeddings and long-range dependency being more pronounced in dynamic graphs, as features are aggregated both across neighborhood and time, no prior work has proposed an implicit graph neural model in a dynamic setting. In this paper, we present Implicit Dynamic Graph Neural Network (IDGNN) a novel implicit neural network for dynamic graphs which is the first of its kind. A key characteristic of IDGNN is that it demonstrably is well-posed, i.e., it is theoretically guaranteed to have a fixed-point representation. We then demonstrate that the standard iterative algorithm often used to train implicit models is computationally expensive in our dynamic setting as it involves computing gradients, which themselves have to be estimated in an iterative manner. To overcome this, we pose an equivalent bilevel optimization problem and propose an efficient single-loop training algorithm that avoids iterative computation by maintaining moving averages of key components of the gradients. We conduct extensive experiments on real-world datasets on both classification and regression tasks to demonstrate the superiority of our approach over the state-of-the-art baselines. We also demonstrate that our bi-level optimization framework maintains the performance of the expensive iterative algorithm while obtaining up to \textbf{1600x} speed-up.

We study the T-system of type $A_\infty$, also known as the octahedron recurrence/equation, viewed as a 2+1-dimensional discrete evolution equation. Generalizing the study of [P. Di Francesco and R. Soto-Garrido. Arctic curves of the octahedron equation. J. Phys. A, 47(28):285204, 34, 2014], we consider initial data along parallel ``slanted" planes perpendicular to an arbitrary admissible direction $(r,s,t)\in {\mathbb Z}_+^3$. The solution of the T-system is interpreted as the partition function of a dimer model on some suitable ``pinecone" graph introduced in [M. Bousquet-M\'elou, J. Propp, and J. West. Perfect matchings for the three-term Gale-Robinson sequences. Electron. J. Combin., 16(1):Research Paper 125, 37, 2009]. The T-system formulation and some exact solutions in uniform or periodic cases allow us to explore the thermodynamic limit of the corresponding dimer models and to derive exact arctic curves separating the various phases of the system., Comment: 69 pages, 42 figures

We propose GNNInfer, the first automatic property inference technique for GNNs. To tackle the challenge of varying input structures in GNNs, GNNInfer first identifies a set of representative influential structures that contribute significantly towards the prediction of a GNN. Using these structures, GNNInfer converts each pair of an influential structure and the GNN to their equivalent FNN and then leverages existing property inference techniques to effectively capture properties of the GNN that are specific to the influential structures. GNNINfer then generalizes the captured properties to any input graphs that contain the influential structures. Finally, GNNInfer improves the correctness of the inferred properties by building a model (either a decision tree or linear regression) that estimates the deviation of GNN output from the inferred properties given full input graphs. The learned model helps GNNInfer extend the inferred properties with constraints to the input and output of the GNN, obtaining stronger properties that hold on full input graphs. Our experiments show that GNNInfer is effective in inferring likely properties of popular real-world GNNs, and more importantly, these inferred properties help effectively defend against GNNs' backdoor attacks. In particular, out of the 13 ground truth properties, GNNInfer re-discovered 8 correct properties and discovered likely correct properties that approximate the remaining 5 ground truth properties. Using properties inferred by GNNInfer to defend against the state-of-the-art backdoor attack technique on GNNs, namely UGBA, experiments show that GNNInfer's defense success rate is up to 30 times better than existing baselines., Comment: 20 pages main paper, 10 pages for appendix

Successful Artificial Intelligence systems often require numerous labeled data to extract information from document images. In this paper, we investigate the problem of improving the performance of Artificial Intelligence systems in understanding document images, especially in cases where training data is limited. We address the problem by proposing a novel finetuning method using reinforcement learning. Our approach treats the Information Extraction model as a policy network and uses policy gradient training to update the model to maximize combined reward functions that complement the traditional cross-entropy losses. Our experiments on four datasets using labels and expert feedback demonstrate that our finetuning mechanism consistently improves the performance of a state-of-the-art information extractor, especially in the small training data regime., Comment: Accepted to ICONIP 2022

This paper introduces a new information extraction model for business documents. Different from prior studies which only base on span extraction or sequence labeling, the model takes into account advantage of both span extraction and sequence labeling. The combination allows the model to deal with long documents with sparse information (the small amount of extracted information). The model is trained end-to-end to jointly optimize the two tasks in a unified manner. Experimental results on four business datasets in English and Japanese show that the model achieves promising results and is significantly faster than the normal span-based extraction method. The code is also available., Comment: Accepted to IJCNN 2022

Principal component analysis is a simple yet useful dimensionality reduction technique in modern machine learning pipelines. In consequential domains such as college admission, healthcare and credit approval, it is imperative to take into account emerging criteria such as the fairness and the robustness of the learned projection. In this paper, we propose a distributionally robust optimization problem for principal component analysis which internalizes a fairness criterion in the objective function. The learned projection thus balances the trade-off between the total reconstruction error and the reconstruction error gap between subgroups, taken in the min-max sense over all distributions in a moment-based ambiguity set. The resulting optimization problem over the Stiefel manifold can be efficiently solved by a Riemannian subgradient descent algorithm with a sub-linear convergence rate. Our experimental results on real-world datasets show the merits of our proposed method over state-of-the-art baselines., Comment: International Conference on Learning Representations (ICLR) 2022

Cementless implants have become widely used for total hip replacement surgery. The long-term stability of these implants is achieved by bone growing around and into the porous surface of the implant, a process called osseointegration. However, debonding of the bone-implant interface can still occur due to aseptic implant loosening and insufficient osseointegration, which may have dramatic consequences. The aim of this work is to describe a new 3D finite element frictional contact formulation for the debonding of partially osseointegrated implants. The contact model is based on a modified Coulomb's friction law (Immel et al. 2020, Biomech. Model. Mechanobiol.) that takes into account the tangential debonding of the bone-implant interface. This model is extended in the direction normal to the bone-implant interface by considering a cohesive zone model, to account for adhesion phenomena in the normal direction and for adhesive friction of partially bonded interfaces. The model is applied to simulate the debonding of an acetabular cup implant. The influence of partial osseointegration and adhesive effects on the long-term stability of the implant is assessed. The influence of different patient- and implant-specific parameters such as the friction coefficient, the trabecular Young's modulus and the interference fit is also analyzed, in order to determine the optimal stability for different configurations. Furthermore, this work provides guidelines for future experimental and computational studies, that are necessary for further parameter calibration., Comment: Biomech Model Mechanobiol (2022)

Information extraction (IE) for visually-rich documents (VRDs) has achieved SOTA performance recently thanks to the adaptation of Transformer-based language models, which shows the great potential of pre-training methods. In this paper, we present a new approach to improve the capability of language model pre-training on VRDs. Firstly, we introduce a new query-based IE model that employs span extraction instead of using the common sequence labeling approach. Secondly, to further extend the span extraction formulation, we propose a new training task that focuses on modelling the relationships among semantic entities within a document. This task enables target spans to be extracted recursively and can be used to pre-train the model or as an IE downstream task. Evaluation on three datasets of popular business documents (invoices, receipts) shows that our proposed method achieves significant improvements compared to existing models. The method also provides a mechanism for knowledge accumulation from multiple downstream IE tasks., Comment: Accepted to Document Images and Language Workshop at ICDAR 2021

Short and long-term stabilities of cementless implants are strongly determined by the interfacial load transfer between implants and bone tissue. Stress-shielding effects arise from shear stresses due to the difference of material properties between bone and the implant. It remains difficult to measure the stress field in periprosthetic bone tissue. This study proposes to investigate the dependence of the stress field in periprosthetic bone tissue on i) the implant surface roughness, ii) material properties of bone and of the implant, iii) the bone-implant contact ratio. To do so, a microscale 2-D finite element model of an osseointegrated boneimplant interface was developed where the surface roughness was modeled by a sinusoidal surface. The results show that the isostatic pressure is not affected by the presence of the bone-implant interface while shear stresses arise due to the combined effects of a geometrical singularity (for low surface roughness) and of shear stresses at the bone-implant interface (for high surface roughness). Stress-shielding effects are likely to be more important when the bone-implant contact ratio value is low, which corresponds to a case of relatively low implant stability. Shear stress reach a maximum value at a distance from the interface comprised between 0 and 0.1 time roughness wavelength $\lambda$ and tend to 0 at a distance from the implant surface higher than $\lambda$, independently from bone-implant contact ratio and waviness ratio. A comparison with an analytical model allows validating the numerical results. Future work should use the present approach to model osseointegration phenomena., Comment: Journal of Orthopaedic Research, Wiley, In press

FUNSD is one of the limited publicly available datasets for information extraction from document im-ages. The information in the FUNSD dataset is defined by text areas of four categories ("key", "value", "header", "other", and "background") and connectivity between areas as key-value relations. In-specting FUNSD, we found several inconsistency in labeling, which impeded its applicability to thekey-value extraction problem. In this report, we described some labeling issues in FUNSD and therevision we made to the dataset. We also reported our implementation of for key-value detection onFUNSD using a UNet model as baseline results and an improved UNet model with Channel-InvariantDeformable Convolution.

Cementless implants are widely used in orthopedic and oral surgery. However, debonding-related failure still occurs at the bone-implant interface. It remains difficult to predict such implant failure since the underlying osseointegration phenomena are still poorly understood. Especially in terms of friction and adhesion at the macro-scale, there is a lack of data and reliable models. The aim of this work is to present a new friction formulation that can model the tangential contact behavior between osseointegrated implants and bone tissue, with focus on debonding. The classical Coulomb's law is combined with a state variable friction law to model a displacement-dependent friction coefficient. A smooth state function, based on the sliding distance, is used to model implant debonding. The formulation is implemented in a 3D nonlinear finite element framework, and it is calibrated with experimental data and compared to an analytical model for mode III cleavage of a coin-shaped, titanium implant (Mathieu et al. 2012). Overall, the results show close agreement with the experimental data, especially the peak and the softening part of the torque curve with a relative error of less than 2.25 %. In addition, better estimates of the bone's shear modulus and the adhesion energy are obtained. The proposed model is particularly suitable to account for partial osseointegration, as is also shown.

Background Resonance frequency analyses and quantitative ultrasound methods have been suggested to assess dental implant primary stability. PurposeThe purpose of this study was to compare the results obtained using these two techniques applied to the same dental implants inserted in various bone phantoms. Materials and MethodsDifferent values of trabecular bone density and cortical thickness were considered to assess the effect of bone quality on the respective indicators (UI and ISQ). The effect of the implant insertion depth and of the final drill diameter was also investigated. ResultsISQ values increase and UI values decrease as a function of trabecular density, cortical thickness and the screwing of the implant. When the implant diameter varies, the UI values are significantly different for all final drill diameters (except for two), while the ISQ values are similar for all final drill diameters lower than 3.2 mm and higher than 3.3 mm. The error on the estimation of parameters with the QUS device is between 4 and 8 times lower compared to that made with the RFA technique.ConclusionsThe results show that ultrasound technique provides a better estimation of different parameters related to the implant stability compared to the RFA technique., Comment: Clinical Implant Dentistry and Related Research, Wiley, 2018

Quantitative ultrasound can be used to characterize the evolution of the bone-implant interface (BII), which is a complex system due to the implant surface roughness and to partial contact between bone and the implant. The aim of this study is to derive the main determinants of the ultrasonic response of the BII during osseointegration phenomena. The influence of i) the 25 surface roughness parameters and ii) the thickness W of a soft tissue layer on the reflection coefficient r of the BII was investigated using a two-dimensional finite element model. When W increases from 0 to 150 $\mu$m, r increases from values in the range [0.45; 0.55] to values in the range [0.75; 0.88] according to the roughness parameters. An optimization method was developed to determine the sinusoidal roughness profile leading to the most similar ultrasonic 30 response for all values of W compared to the original profile. The results show that the difference between the ultrasonic responses of the optimal sinusoidal profile and of the original profile was lower to typical experimental errors. This approach provides a better understanding of the ultrasonic response of the BII, which may be used in future numerical simulation realized at the scale of an implant., Comment: Journal of the Acoustical Society of America, Acoustical Society of America, In press

Degenerative discopathy is a common pathology which may require spine surgery. A metallic cylindrical pin is inserted into the vertebral body to maintain soft tissues and may be used as a reflector of ultrasonic wave to estimate bone density. The first aim of this paper is to validate a 3D model to simulate the ultrasonic propagation in a trabecular bone sample in which a metallic pin has been inserted. We also aim at determining the effect of changes of bone volume fraction (BV/TV) and of positioning errors on the quantitative ultrasound (QUS) parameters in this specific configuration. The approach consists in coupling finite difference time domain simulation with X-ray microcomputed tomography. The correlation coefficient between experimental and simulated speed of sound (SOS) (respectively broadband ultrasonic attenuation (BUA)) was equal to 0.90 (respectively 0.55). The results show a significant correlation of SOS with BV/TV (R = 0.82), while BUA values exhibit a non-linear behavior versus BV/TV. The orientation of the pin should be controlled with an accuracy of around 1{\textdegree} in order to obtain accurate results. The results indicate that using the ultrasonic wave reflected by a pin has a potential to estimate the bone density. SOS is more reliable than BUA due its lower sensitivity to the tilt angle.