Search

Despite the empirical success in various domains, it has been revealed that deep neural networks are vulnerable to maliciously perturbed input data that can dramatically degrade their performance. These are known as adversarial attacks. To counter adversarial attacks, adversarial training formulated as a form of robust optimization has been demonstrated to be effective. However, conducting adversarial training brings much computational overhead compared with standard training. In order to reduce the computational cost, we propose an annealing mechanism, annealing mechanism for adversarial training acceleration (Amata), to reduce the overhead associated with adversarial training. The proposed Amata is provably convergent, well-motivated from the lens of optimal control theory, and can be combined with existing acceleration methods to further enhance performance. It is demonstrated that, on standard datasets, Amata can achieve similar or better robustness with around 1/3-1/2 the computational time compared with traditional methods. In addition, Amata can be incorporated into other adversarial training acceleration algorithms (e.g., YOPO, Free, Fast, and ATTA), which leads to a further reduction in computational time on large-scale problems.

In this perspective, the authors challenge the status quo of polymer innovation. The authors first explore how research in polymer design is conducted today, which is both time consuming and unable to capture the multi-scale complexities of polymers. The authors discuss strategies that could be employed in bringing together machine learning, data curation, high-throughput experimentation, and simulations, to build a system that can accurately predict polymer properties from their descriptors and enable inverse design that is capable of designing polymers based on desired properties.

Though machine learning algorithms are able to achieve pattern recognition from the correlation between data and labels, the presence of spurious features in the data decreases the robustness of these learned relationships with respect to varied testing environments. This is known as out-of-distribution (OoD) generalization problem. Recently, invariant risk minimization (IRM) attempts to tackle this issue by penalizing predictions based on the unstable spurious features in the data collected from different environments. However, similar to domain adaptation or domain generalization, a prevalent non-trivial limitation in these works is that the environment information is assigned by human specialists, i.e. a priori, or determined heuristically. However, an inappropriate group partitioning can dramatically deteriorate the OoD generalization and this process is expensive and time-consuming. To deal with this issue, we propose a novel theoretically principled min-max framework to iteratively construct a worst-case splitting, i.e. creating the most challenging environment splittings for the backbone learning paradigm (e.g. IRM) to learn the robust feature representation. We also design a differentiable training strategy to facilitate the feasible gradient- based computation. Numerical experiments show that our algorithmic framework has achieved superior and stable performance in various datasets, such as Colored MNIST and Punctuated Stanford sentiment treebank (SST). Furthermore, we also find our algorithm to be robust even to a strong data poisoning attack. To the best of our knowledge, this is one of the first to adopt differentiable environment splitting method to enable stable predictions across environments without environment index information, which achieves the state-of-the-art performance on datasets with strong spurious correlation, such as Colored MNIST.

In materials science, the discovery of recipes that yield nanomaterials with defined optical properties is costly and time-consuming. In this study, we present a two-step framework for a machine learning-driven high-throughput microfluidic platform to rapidly produce silver nanoparticles with the desired absorbance spectrum. Combining a Gaussian process-based Bayesian optimization (BO) with a deep neural network (DNN), the algorithmic framework is able to converge towards the target spectrum after sampling 120 conditions. Once the dataset is large enough to train the DNN with sufficient accuracy in the region of the target spectrum, the DNN is used to predict the colour palette accessible with the reaction synthesis. While remaining interpretable by humans, the proposed framework efficiently optimizes the nanomaterial synthesis and can extract fundamental knowledge of the relationship between chemical composition and optical properties, such as the role of each reactant on the shape and amplitude of the absorbance spectrum. Agency for Science, Technology and Research (A*STAR) Published version We would like to thank Swee Liang Wong, Lim Yee-Fun, Xu Yang, Jatin Kumar, Liu Xiali and Li Jiali for equipment support and helpful discussions. Support was provided by the Accelerated Materials Development for Manufacturing Program at A*STAR via the AME Programmatic Fund by the Agency for Science, Technology and Research under Grant no. A1898b0043, (F.M.B., Z.R., T.H., W.K.W., F.Z., J.X., S.J., Z.M., D.B., K.H., S.A.K., Q.L., and X.W.) and Singapore’s National Research Foundation through the Singapore MIT Alliance for Research and Technology’s Low energy electronic systems (LEES) IRG (Z.R., I.P.S.T., and T.B.).

Combining high-throughput experiments with machine learning allows quick optimization of parameter spaces towards achieving target properties. In this study, we demonstrate that machine learning, combined with multi-labeled datasets, can additionally be used for scientific understanding and hypothesis testing. We introduce an automated flow system with high-throughput drop-casting for thin film preparation, followed by fast characterization of optical and electrical properties, with the capability to complete one cycle of learning of fully labeled ~160 samples in a single day. We combine regio-regular poly-3-hexylthiophene with various carbon nanotubes to achieve electrical conductivities as high as 1200 S/cm. Interestingly, a non-intuitive local optimum emerges when 10% of double-walled carbon nanotubes are added with long single wall carbon nanotubes, where the conductivity is seen to be as high as 700 S/cm, which we subsequently explain with high fidelity optical characterization. Employing dataset resampling strategies and graph-based regressions allows us to account for experimental cost and uncertainty estimation of correlated multi-outputs, and supports the proving of the hypothesis linking charge delocalization to electrical conductivity. We therefore present a robust machine-learning driven high-throughput experimental scheme that can be applied to optimize and understand properties of composites, or hybrid organic-inorganic materials., Comment: 10 pages, 5 figures, includes Supplementary Information

Study of diffusion in solids is of fundamental importance in understanding the materials properties such as phase transformations, segregation, and corrosion in processing and operations. Reliable diffusion data is crucial in the successful design of materials and manufacturing processes. In this work, we use gradient boosting with decision tree regression, a robust machine learning method, to develop the model for predicting interstitial diffusion activation energies of the light element X (B, C, O, and N) in 54 metals with bcc, fcc and hcp lattices. With the introduction of elastic strain energy, a dominant concept of interstitial diffusion theory, we are able to train the machine learning model achieving high accuracy of R-squared 0.9 without being over-fitted by using 94 impurity – host binary systems reported in the literature. Furthermore, our study reveals that apart from elastic strain energy, the electronic interaction between the impurity and the host (represented by two fundamental parameters in the well-known Miedema model) also plays an important role. By accurately predicting the diffusion activation energies, we report their trends across 4th, 5th and 6th period and Lanthanides series of the host atoms. More importantly, our predicted data can be used to approximate the transport rates in materials for which little or no diffusion data are available, thus accelerating the design of a wide range of new application-specific materials.

We consider distributed optimization with degenerate loss functions, where the optimal sets of local loss functions have a non-empty intersection. This regime often arises in optimizing large-scale multi-agent AI systems (e.g., deep learning systems), where the number of trainable weights far exceeds the number of training samples, leading to highly degenerate loss surfaces. Under appropriate conditions, we prove that distributed gradient descent in this case converges even when communication is arbitrarily less frequent, which is not the case for non-degenerate loss functions. Moreover, we quantitatively analyze the convergence rate, as well as the communication and computation trade-off, providing insights into designing efficient distributed optimization algorithms. Our theoretical findings are confirmed by both distributed convex optimization and deep learning experiments.

Recent work linking deep neural networks and dynamical systems opened up new avenues to analyze deep learning. In particular, it is observed that new insights can be obtained by recasting deep learning as an optimal control problem on difference or differential equations. However, the mathematical aspects of such a formulation have not been systematically explored. This paper introduces the mathematical formulation of the population risk minimization problem in deep learning as a mean-field optimal control problem. Mirroring the development of classical optimal control, we state and prove optimality conditions of both the Hamilton-Jacobi-Bellman type and the Pontryagin type. These mean-field results reflect the probabilistic nature of the learning problem. In addition, by appealing to the mean-field Pontryagin's maximum principle, we establish some quantitative relationships between population and empirical learning problems. This serves to establish a mathematical foundation for investigating the algorithmic and theoretical connections between optimal control and deep learning., Comment: 44 pages

The committor function is a central object of study in understanding transitions between metastable states in complex systems. However, computing the committor function for realistic systems at low temperatures is a challenging task due to the curse of dimensionality and the scarcity of transition data. In this paper, we introduce a computational approach that overcomes these issues and achieves good performance on complex benchmark problems with rough energy landscapes or in high dimensions. The new approach combines deep learning, data sampling, and feature engineering techniques. This establishes an alternative practical method for studying rare transition events between metastable states in complex, high dimensional systems.