Search

Ensemble learning is a widespread technique to improve the prediction performance of neural networks. However, it comes at the price of increased memory and inference time. In this work we propose a novel model fusion technique called \emph{Neuron Transplantation (NT)} in which we fuse an ensemble of models by transplanting important neurons from all ensemble members into the vacant space obtained by pruning insignificant neurons. An initial loss in performance post-transplantation can be quickly recovered via fine-tuning, consistently outperforming individual ensemble members of the same model capacity and architecture. Furthermore, NT enables all the ensemble members to be jointly pruned and jointly trained in a combined model. Comparing it to alignment-based averaging (like Optimal-Transport-fusion), it requires less fine-tuning than the corresponding OT-fused model, the fusion itself is faster and requires less memory, while the resulting model performance is comparable or better. The code is available under the following link: https://github.com/masterbaer/neuron-transplantation., Comment: 18 pages, 7 figures, conference: ECML-PKDD 2024

The current landscape in time-series forecasting is dominated by Transformer-based models. Their high parameter count and corresponding demand in computational resources pose a challenge to real-world deployment, especially for commercial and scientific applications with low-power embedded devices. Pruning is an established approach to reduce neural network parameter count and save compute. However, the implications and benefits of pruning Transformer-based models for time series forecasting are largely unknown. To close this gap, we provide a comparative benchmark study by evaluating unstructured and structured pruning on various state-of-the-art multivariate time series models. We study the effects of these pruning strategies on model predictive performance and computational aspects like model size, operations, and inference time. Our results show that certain models can be pruned even up to high sparsity levels, outperforming their dense counterpart. However, fine-tuning pruned models is necessary. Furthermore, we demonstrate that even with corresponding hardware and software support, structured pruning is unable to provide significant time savings., Comment: 16 pages, 5 figures, submitted to ACM Transactions on Intelligent Systems and Technology

The FAIR principles are globally accepted guidelines for improved data management practices with the potential to align data spaces on a global scale. In practice, this is only marginally achieved through the different ways in which organizations interpret and implement these principles. The concept of FAIR Digital Objects provides a way to realize a domain-independent abstraction layer that could solve this problem, but its specifications are currently diverse, contradictory, and restricted to semantic models. In this work, we introduce a rigorously formalized data model with a set of assertions using formal expressions to provide a common baseline for the implementation of FAIR Digital Objects. The model defines how these objects enable machine-actionable decisions based on the principles of abstraction, encapsulation, and entity relationship to fulfill FAIR criteria for the digital resources they represent. We provide implementation examples in the context of two use cases and explain how our model can facilitate the (re)use of data across domains. We also compare how our model assertions are met by FAIR Digital Objects as they have been described in other projects. Finally, we discuss our results' adoption criteria, limitations, and perspectives in the big data context. Overall, our work represents an important milestone for various communities working towards globally aligned data spaces through FAIRification., Comment: Accepted at the 2024 IEEE International Conference on Big Data (IEEE BigData 2024)

The greedoid is a significant abstraction of the matroid allowing for a more flexible analysis of structures in which the greedy algorithm "works." However, their diverse structure imposes difficulties towards their application in combinatorial optimization [Sze21]. In response, we revisit the polymatroid greedoid [KL85a] to characterize it by properties approximating those of matroids, by using the submodularity of its polymatroid representation in particular. Towards doing so, our main contribution is a full description of this class. Specifically, we show that a greedoid is a polymatroid greedoid if and only if it is an optimistic interval greedoid whose kernels are closed under intersection. This constitutes the first necessary and sufficient characterization of the polymatroid greedoid in terms of its combinatorial attributes, thereby resolving a central open question of Korte and Lov\'asz [KL85a]. Here, we introduce the optimism property to approximate properties of a matroid's continuations which are implied by the closure axioms of its span, which no longer hold for greedoids. And, because the kernels of an interval greedoid are in many ways an extension of a matroid's closed sets, our direction of necessity is a direct generalization of Birkhoff and Edmond's characterization of the meet in the lattice of a matroid's closed sets [Bir35, Edm03]. Towards achieving this result, our main technical insights arise from relating the lattice of flats of a polymatroid greedoid to that of the closed sets of its representation through order preserving mappings. Specifically, we will show the novel insight that the notion of polymatroid representation considered in [KL85a] is equivalent to the existence of a certain Galois connection. As a consequence, the representation of a greedoid via a polymatroid is an order theoretic concept in disguise., Comment: 26 pages, 4 figures. In versions 1 and 2 there is an error in the proof of the main claim. This has been noted on the first page, and the authors are currently revising the manuscript to resolve this issue

The gradients used to train neural networks are typically computed using backpropagation. While an efficient way to obtain exact gradients, backpropagation is computationally expensive, hinders parallelization, and is biologically implausible. Forward gradients are an approach to approximate the gradients from directional derivatives along random tangents computed by forward-mode automatic differentiation. So far, research has focused on using a single tangent per step. This paper provides an in-depth analysis of multi-tangent forward gradients and introduces an improved approach to combining the forward gradients from multiple tangents based on orthogonal projections. We demonstrate that increasing the number of tangents improves both approximation quality and optimization performance across various tasks.

Much energy has been devoted to developing a matroid's computational properties, yet parallel algorithm design for matroid optimization seems less understood. Specifically, the current state of the art is a folklore reduction from optimization to the search based on methods originating in [KUW88]. However, while this reduction adds only constant overhead in terms of \emph{adaptive complexity}, it imposes a high cost in \emph{query complexity}. In response, we present a new reduction from optimization to search within the class of \emph{binary matroids} which, when $n$ and $r$ take the size of the ground set and matroid rank respectively, implies a novel optimization algorithm terminating in $\mathcal{O}(\sqrt{n}\cdot\log r)$ parallel rounds using only $\mathcal{O}(rn\cdot\log r)$ independence queries. This is a significant improvement in query complexity when the matroid is sparse, meaning $r \ll n$, while trading off only a logarithmic factor of the rank in the adaptive complexity. At a technical level, our method begins by observing that a basis is optimal if and only if it is the set of points of minimum weight in any cocircuit. Importantly, this certificate reveals that simultaneous tests for \emph{local optimality} in cocircuits is a general paradigm for parallel matroid optimization. By combining this idea with connections between bases and cocircuits we obtain our reduction, whose efficiency follows by analyzing the lattice of flats. A primary goal of our study is initiating a finer understanding of parallel matroid optimization. And so, since many of our techniques begin with observations about general matroids and their flats, we hope that our efforts aid the future design of parallel matroid algorithms and applications of lattice theory thereof., Comment: 35 pages, 7 figures, 4 algorithms, 1 table

Transformers have recently gained prominence in long time series forecasting by elevating accuracies in a variety of use cases. Regrettably, in the race for better predictive performance the overhead of model architectures has grown onerous, leading to models with computational demand infeasible for most practical applications. To bridge the gap between high method complexity and realistic computational resources, we introduce the Residual Cyclic Transformer, ReCycle. ReCycle utilizes primary cycle compression to address the computational complexity of the attention mechanism in long time series. By learning residuals from refined smoothing average techniques, ReCycle surpasses state-of-the-art accuracy in a variety of application use cases. The reliable and explainable fallback behavior ensured by simple, yet robust, smoothing average techniques additionally lowers the barrier for user acceptance. At the same time, our approach reduces the run time and energy consumption by more than an order of magnitude, making both training and inference feasible on low-performance, low-power and edge computing devices. Code is available at https://github.com/Helmholtz-AI-Energy/ReCycle, Comment: 9 pages, 3 figures, to be published at IEEE CAI 2024, Associated code available at https://github.com/Helmholtz-AI-Energy/ReCycle

[Context and motivation]: Understanding and interpreting regulatory norms and inferring software requirements from them is a critical step towards regulatory compliance, a matter of significant importance in various industrial sectors. [Question/ problem]: However, interpreting regulations still largely depends on individual legal expertise and experience within the respective domain, with little to no systematic methodologies and supportive tools to guide this practice. In fact, research in this area is too often detached from practitioners' experiences, rendering the proposed solutions not transferable to industrial practice. As we argue, one reason is that we still lack a profound understanding of industry- and domain-specific practices and challenges. [Principal ideas/ results]: We aim to close this gap and provide such an investigation at the example of the banking and insurance domain. We conduct an industrial multi-case study as part of a long-term academia-industry collaboration with a medium-sized software development and renovation company. We explore contemporary industrial practices and challenges when inferring requirements from regulations to support more problem-driven research. Our study investigates the complexities of requirement engineering in regulatory contexts, pinpointing various issues and discussing them in detail. We highlight the gathered insights and the practical challenges encountered and suggest avenues for future research. [Contribution]: Our contribution is a comprehensive case study focused on the FinTech domain, offering a detailed understanding of the specific needs within this sector. We have identified key practices for managing regulatory requirements in software development, and have pinpointed several challenges. We conclude by offering a set of recommendations for future problem-driven research directions.

Communication bottlenecks severely hinder the scalability of distributed neural network training, particularly in high-performance computing (HPC) environments. We introduce AB-training, a novel data-parallel method that leverages low-rank representations and independent training groups to significantly reduce communication overhead. Our experiments demonstrate an average reduction in network traffic of approximately 70.31\% across various scaling scenarios, increasing the training potential of communication-constrained systems and accelerating convergence at scale. AB-training also exhibits a pronounced regularization effect at smaller scales, leading to improved generalization while maintaining or even reducing training time. We achieve a remarkable 44.14 : 1 compression ratio on VGG16 trained on CIFAR-10 with minimal accuracy loss, and outperform traditional data parallel training by 1.55\% on ResNet-50 trained on ImageNet-2012. While AB-training is promising, our findings also reveal that large batch effects persist even in low-rank regimes, underscoring the need for further research into optimized update mechanisms for massively distributed training.

Predicting the performance of various infrastructure design options in complex federated infrastructures with computing sites distributed over a wide area network that support a plethora of users and workflows, such as the Worldwide LHC Computing Grid (WLCG), is not trivial. Due to the complexity and size of these infrastructures, it is not feasible to deploy experimental test-beds at large scales merely for the purpose of comparing and evaluating alternate designs. An alternative is to study the behaviours of these systems using simulation. This approach has been used successfully in the past to identify efficient and practical infrastructure designs for High Energy Physics (HEP). A prominent example is the Monarc simulation framework, which was used to study the initial structure of the WLCG. New simulation capabilities are needed to simulate large-scale heterogeneous computing systems with complex networks, data access and caching patterns. A modern tool to simulate HEP workloads that execute on distributed computing infrastructures based on the SimGrid and WRENCH simulation frameworks is outlined. Studies of its accuracy and scalability are presented using HEP as a case-study. Hypothetical adjustments to prevailing computing architectures in HEP are studied providing insights into the dynamics of a part of the WLCG and candidates for improvements., Comment: 26th International Conference on Computing in High Energy and Nuclear Physics (CHEP 2023)

This study explores the learning dynamics of neural networks by analyzing the singular value decomposition (SVD) of their weights throughout training. Our investigation reveals that an orthogonal basis within each multidimensional weight's SVD representation stabilizes during training. Building upon this, we introduce Orthogonality-Informed Adaptive Low-Rank (OIALR) training, a novel training method exploiting the intrinsic orthogonality of neural networks. OIALR seamlessly integrates into existing training workflows with minimal accuracy loss, as demonstrated by benchmarking on various datasets and well-established network architectures. With appropriate hyperparameter tuning, OIALR can surpass conventional training setups, including those of state-of-the-art models.

The graph polynomial for the number of independent sets of size $k$ in a general undirected graph is shown to be equal to an elementary symmetric polynomial of the vertex monomials, which are determined by the edges incident at the vertices. The edge variables that comprise the vertex monomials are shown to be nilpotent of degree two. The index of nilpotency of the algebra generated by the graph's vertex monomials is the order of a maximum independent set of the graph. Graph polynomials for the number of cliques and vertex covers are derived from the graph polynomial for independent sets. The graph polynomial for the number of bipartite cuts is given as a coefficient of a multivariate Laurent polynomial.

We define a new class of predicates called equilevel predicates on a distributive lattice which eases the analysis of parallel algorithms. Many combinatorial problems such as the vertex cover problem, the bipartite matching problem, and the minimum spanning tree problem can be modeled as detecting an equilevel predicate. The problem of detecting an equilevel problem is NP-complete, but equilevel predicates with the helpful property can be detected in polynomial time in an online manner. An equilevel predicate has the helpful property with a polynomial time algorithm if the algorithm can return a nonempty set of indices such that advancing on any of them can be used to detect the predicate. Furthermore, the refined independently helpful property allows online parallel detection of such predicates in NC. When the independently helpful property holds, advancing on all the specified indices in parallel can be used to detect the predicate in polylogarithmic time. We also define a special class of equilevel predicates called solitary predicates. Unless NP = RP, this class of predicate also does not admit efficient algorithms. Earlier work has shown that solitary predicates with the efficient advancement can be detected in polynomial time. We introduce two properties called the antimonotone advancement and the efficient rejection which yield the detection of solitary predicates in NC. Finally, we identify the minimum spanning tree, the shortest path, and the conjunctive predicate detection as problems satisfying such properties, giving alternative certifications of their NC memberships as a result., Comment: To appear in ICDCN 2024

A large body of literature within the additive manufacturing (AM) community has focused on successfully creating stable tungsten (W) microstructures due to significant interest in its application for extreme environments. However, solidification cracking and additional embrittling features at grain boundaries have resulted in poorly performing microstructures, stymying the application of AM as a manufacturing technique for W. Several alloying strategies, such as ceramic particles and ductile elements, have emerged with the promise to eliminate solidification cracking while simultaneously enhancing stability against recrystallization. In this work, we provide new insights regarding the defects and microstructural features that result from the introduction of ZrC for grain refinement and NiFe as a ductile reinforcement phase - in addition to the resulting thermophysical and mechanical properties. ZrC is shown to promote microstructural stability with increased hardness due to the formation of ZrO2 dispersoids. Conversely, NiFe forms into micron-scale FCC phase regions within a BCC W matrix, producing enhanced toughness relative to pure AM W. A combination of these effects is realized in the WNiFe+ZrC system and demonstrates that complex chemical environments coupled with the tuning of AM microstructures provides an effective pathway for enabling laser AM W materials with enhanced stability and performance.

Given the urgency to reduce fossil fuel energy production to make climate tipping points less likely, we call for resource-aware knowledge gain in the research areas on Universe and Matter with emphasis on the digital transformation. A portfolio of measures is described in detail and then summarized according to the timescales required for their implementation. The measures will both contribute to sustainable research and accelerate scientific progress through increased awareness of resource usage. This work is based on a three-days workshop on sustainability in digital transformation held in May 2023., Comment: 20 pages, 2 figures, publication following workshop 'Sustainability in the Digital Transformation of Basic Research on Universe & Matter', 30 May to 2 June 2023, Meinerzhagen, Germany, https://indico.desy.de/event/37480

Visualization onboarding supports users in reading, interpreting, and extracting information from visual data representations. General-purpose onboarding tools and libraries are applicable for explaining a wide range of graphical user interfaces but cannot handle specific visualization requirements. This paper describes a first step towards developing an onboarding library called VisAhoi, which is easy to integrate, extend, semi-automate, reuse, and customize. VisAhoi supports the creation of onboarding elements for different visualization types and datasets. We demonstrate how to extract and describe onboarding instructions using three well-known high-level descriptive visualization grammars - Vega-Lite, Plotly.js, and ECharts. We show the applicability of our library by performing two usage scenarios that describe the integration of VisAhoi into a VA tool for the analysis of high-throughput screening (HTS) data and, second, into a Flourish template to provide an authoring tool for data journalists for a treemap visualization. We provide a supplementary website that demonstrates the applicability of VisAhoi to various visualizations, including a bar chart, a horizon graph, a change matrix or heatmap, a scatterplot, and a treemap visualization.

Segregation engineering has emerged as a promising pathway towards designing thermally stable nanocrystalline alloys with enhanced mechanical properties. However, the compositional and processing space for solute stabilized microstructures is vast, thus the application of high-throughput techniques to accelerate optimal material development is increasingly attractive. In this work, combinatorial synthesis is combined with high-throughput characterization techniques to explore microstructural transitions through annealing of a nanocrystalline ternary Al-base alloy containing a transition metal (TM=Ni) and rare earth dopant (RE=Y). A down-selected optimal composition with the highest thermal stability is annealed through in situ transmission electron microscopy, revealing that the removal of the RE species is correlated to a reduction in the microstructural stability at high temperatures as a result of variations in intermetallic phase formation. Results demonstrate the benefits of co-segregation for enhancing mechanical hardness and delaying the onset of microstructural instability., Comment: 12 pages, 4 figures

Backpropagation has long been criticized for being biologically implausible, relying on concepts that are not viable in natural learning processes. This paper proposes an alternative approach to solve two core issues, i.e., weight transport and update locking, for biological plausibility and computational efficiency. We introduce Feed-Forward with delayed Feedback (F$^3$), which improves upon prior work by utilizing delayed error information as a sample-wise scaling factor to approximate gradients more accurately. We find that F$^3$ reduces the gap in predictive performance between biologically plausible training algorithms and backpropagation by up to 96%. This demonstrates the applicability of biologically plausible training and opens up promising new avenues for low-energy training and parallelization.

We present Propulate, an evolutionary optimization algorithm and software package for global optimization and in particular hyperparameter search. For efficient use of HPC resources, Propulate omits the synchronization after each generation as done in conventional genetic algorithms. Instead, it steers the search with the complete population present at time of breeding new individuals. We provide an MPI-based implementation of our algorithm, which features variants of selection, mutation, crossover, and migration and is easy to extend with custom functionality. We compare Propulate to the established optimization tool Optuna. We find that Propulate is up to three orders of magnitude faster without sacrificing solution accuracy, demonstrating the efficiency and efficacy of our lazy synchronization approach. Code and documentation are available at https://github.com/Helmholtz-AI-Energy/propulate, Comment: 18 pages, 5 figures submitted to ISC High Performance 2023

There are a multitude of Blockchain-based physical infrastructure systems, operating on a crypto-currency enabled token economy, where infrastructure suppliers are rewarded with tokens for enabling, validating, managing and/or securing the system. However, today's token economies are largely designed without infrastructure systems in mind, and often operate with a fixed token supply (e.g., Bitcoin). This paper argues that token economies for infrastructure networks should be structured differently - they should continually incentivize new suppliers to join the network to provide services and support to the ecosystem. As such, the associated token rewards should gracefully scale with the size of the decentralized system, but should be carefully balanced with consumer demand to manage inflation and be designed to ultimately reach an equilibrium. To achieve such an equilibrium, the decentralized token economy should be adaptable and controllable so that it maximizes the total utility of all users, such as achieving stable (overall non-inflationary) token economies. Our main contribution is to model infrastructure token economies as dynamical systems - the circulating token supply, price, and consumer demand change as a function of the payment to nodes and costs to consumers for infrastructure services. Crucially, this dynamical systems view enables us to leverage tools from mathematical control theory to optimize the overall decentralized network's performance. Moreover, our model extends easily to a Stackelberg game between the controller and the nodes, which we use for robust, strategic pricing. In short, we develop predictive, optimization-based controllers that outperform traditional algorithmic stablecoin heuristics by up to $2.4 \times$ in simulations based on real demand data from existing decentralized wireless networks.

ParaDime is a framework for parametric dimensionality reduction (DR). In parametric DR, neural networks are trained to embed high-dimensional data items in a low-dimensional space while minimizing an objective function. ParaDime builds on the idea that the objective functions of several modern DR techniques result from transformed inter-item relationships. It provides a common interface for specifying these relations and transformations and for defining how they are used within the losses that govern the training process. Through this interface, ParaDime unifies parametric versions of DR techniques such as metric MDS, t-SNE, and UMAP. It allows users to fully customize all aspects of the DR process. We show how this ease of customization makes ParaDime suitable for experimenting with interesting techniques such as hybrid classification/embedding models and supervised DR. This way, ParaDime opens up new possibilities for visualizing high-dimensional data., Comment: accepted at EuroVis 2023

In this paper, we present a hybrid quantum/classical algorithm to solve an NP-hard combinatorial problem called the multiple target data association (MTDA) and tracking problem. We use diabatic quantum annealing (DQA) to enumerate the low energy, or high probability, feasible assignments, and we use a classical computer to find the Bayesian expected mean track estimate by summing over these assignments. We demonstrate our hybrid quantum/classical approach on a simple example. This may be the first demonstration of a Bayesian hybrid quantum-classical multiple target tracking filter. We contrast our DQA method with the adiabatic quantum computing (AQC) approach to MTDA. We give a theoretical overview of DQA and characterize some of the technical limitations of using quantum annealers in this novel diabatic modality., Comment: 9 pages, 6 figures

In this work, we present a neural approach to reconstructing rooted tree graphs describing hierarchical interactions, using a novel representation we term the Lowest Common Ancestor Generations (LCAG) matrix. This compact formulation is equivalent to the adjacency matrix, but enables learning a tree's structure from its leaves alone without the prior assumptions required if using the adjacency matrix directly. Employing the LCAG therefore enables the first end-to-end trainable solution which learns the hierarchical structure of varying tree sizes directly, using only the terminal tree leaves to do so. In the case of high-energy particle physics, a particle decay forms a hierarchical tree structure of which only the final products can be observed experimentally, and the large combinatorial space of possible trees makes an analytic solution intractable. We demonstrate the use of the LCAG as a target in the task of predicting simulated particle physics decay structures using both a Transformer encoder and a Neural Relational Inference encoder Graph Neural Network. With this approach, we are able to correctly predict the LCAG purely from leaf features for a maximum tree-depth of $8$ in $92.5\%$ of cases for trees up to $6$ leaves (including) and $59.7\%$ for trees up to $10$ in our simulated dataset., Comment: 14 pages, 6 figures, accepted in Machine Learning: Science and Technology

In current blockchain systems, full nodes that perform all of the available functionalities need to store the entire blockchain. In addition to the blockchain, full nodes also store a blockchain-summary, called the \emph{state}, which is used to efficiently verify transactions. With the size of popular blockchains and their states growing rapidly, full nodes require massive storage resources in order to keep up with the scaling. This leads to a tug-of-war between scaling and decentralization since fewer entities can afford expensive resources. We present \emph{hybrid nodes} for proof-of-work (PoW) cryptocurrencies which can validate transactions, validate blocks, validate states, mine, select the main chain, bootstrap new hybrid nodes, and verify payment proofs. With the use of a protocol called \emph{trimming}, hybrid nodes only retain polylogarithmic number of blocks in the chain length in order to represent the proof-of-work of the blockchain. Hybrid nodes are also optimized for the storage of the state with the use of \emph{stateless blockchain} protocols. The lowered storage requirements should enable more entities to join as hybrid nodes and improve the decentralization of the system. We define novel theoretical security models for hybrid nodes and show that they are provably secure. We also show that the storage requirement of hybrid nodes is near-optimal with respect to our security definitions.