Search

Multi-Task Learning (MTL) is a learning paradigm in machine learning and its aim is to leverage useful information contained in multiple related tasks to help improve the generalization performance of all the tasks. In this paper, we give a survey for MTL from the perspective of algorithmic modeling, applications and theoretical analyses. For algorithmic modeling, we give a definition of MTL and then classify different MTL algorithms into five categories, including feature learning approach, low-rank approach, task clustering approach, task relation learning approach and decomposition approach as well as discussing the characteristics of each approach. In order to improve the performance of learning tasks further, MTL can be combined with other learning paradigms including semi-supervised learning, active learning, unsupervised learning, reinforcement learning, multi-view learning and graphical models. When the number of tasks is large or the data dimensionality is high, we review online, parallel and distributed MTL models as well as dimensionality reduction and feature hashing to reveal their computational and storage advantages. Many real-world applications use MTL to boost their performance and we review representative works in this paper. Finally, we present theoretical analyses and discuss several future directions for MTL. [ABSTRACT FROM AUTHOR]

With the proliferation of user-generated online videos, Multimodal Sentiment Analysis (MSA) has attracted increasing attention recently. Despite significant progress, there are still two major challenges on the way towards robust MSA: 1) inefficiency when modeling cross-modal interactions in unaligned multimodal data; and 2) vulnerability to random modality feature missing which typically occurs in realistic settings. In this paper, we propose a generic and unified framework to address them, named Efficient Multimodal Transformer with Dual-Level Feature Restoration (EMT-DLFR). Concretely, EMT employs utterance-level representations from each modality as the global multimodal context to interact with local unimodal features and mutually promote each other. It not only avoids the quadratic scaling cost of previous local-local cross-modal interaction methods but also leads to better performance. To improve model robustness in the incomplete modality setting, on the one hand, DLFR performs low-level feature reconstruction to implicitly encourage the model to learn semantic information from incomplete data. On the other hand, it innovatively regards complete and incomplete data as two different views of one sample and utilizes siamese representation learning to explicitly attract their high-level representations. Comprehensive experiments on three popular datasets demonstrate that our method achieves superior performance in both complete and incomplete modality settings. [ABSTRACT FROM AUTHOR]

Pretrained deep models hold their learnt knowledge in the form of model parameters. These parameters act as “memory” for the trained models and help them generalize well on unseen data. However, in absence of training data, the utility of a trained model is merely limited to either inference or better initialization towards a target task. In this paper, we go further and extract synthetic data by leveraging the learnt model parameters. We dub them Data Impressions, which act as proxy to the training data and can be used to realize a variety of tasks. These are useful in scenarios where only the pretrained models are available and the training data is not shared (e.g., due to privacy or sensitivity concerns). We show the applicability of data impressions in solving several computer vision tasks such as unsupervised domain adaptation, continual learning as well as knowledge distillation. We also study the adversarial robustness of lightweight models trained via knowledge distillation using these data impressions. Further, we demonstrate the efficacy of data impressions in generating data-free Universal Adversarial Perturbations (UAPs) with better fooling rates. Extensive experiments performed on benchmark datasets demonstrate competitive performance achieved using data impressions in absence of original training data. [ABSTRACT FROM AUTHOR]

As deep learning models are usually massive and complex, distributed learning is essential for increasing training efficiency. Moreover, in many real-world application scenarios like healthcare, distributed learning can also keep the data local and protect privacy. Recently, the asynchronous decentralized parallel stochastic gradient descent (ADPSGD) algorithm has been proposed and demonstrated to be an efficient and practical strategy where there is no central server, so that each computing node only communicates with its neighbors. Although no raw data will be transmitted across different local nodes, there is still a risk of information leak during the communication process for malicious participants to make attacks. In this paper, we present a differentially private version of asynchronous decentralized parallel SGD framework, or A(DP) $^2$ 2 SGD for short, which maintains communication efficiency of ADPSGD and prevents the inference from malicious participants. Specifically, Rényi differential privacy is used to provide tighter privacy analysis for our composite Gaussian mechanisms while the convergence rate is consistent with the non-private version. Theoretical analysis shows A(DP) $^2$ 2 SGD also converges at the optimal $\mathcal {O}(1/\sqrt{T})$ O (1 / T) rate as SGD. Empirically, A(DP) $^2$ 2 SGD achieves comparable model accuracy as the differentially private version of Synchronous SGD (SSGD) but runs much faster than SSGD in heterogeneous computing environments. [ABSTRACT FROM AUTHOR]

Time series classification models have been garnering significant importance in the research community. However, not much research has been done on generating adversarial samples for these models. These adversarial samples can become a security concern. In this paper, we propose utilizing an adversarial transformation network (ATN) on a distilled model to attack various time series classification models. The proposed attack on the classification model utilizes a distilled model as a surrogate that mimics the behavior of the attacked classical time series classification models. Our proposed methodology is applied onto 1-nearest neighbor dynamic time warping (1-NN DTW) and a fully convolutional network (FCN), all of which are trained on 42 University of California Riverside (UCR) datasets. In this paper, we show both models were susceptible to attacks on all 42 datasets. When compared to Fast Gradient Sign Method, the proposed attack generates a larger faction of successful adversarial black-box attacks. A simple defense mechanism is successfully devised to reduce the fraction of successful adversarial samples. Finally, we recommend future researchers that develop time series classification models to incorporating adversarial data samples into their training data sets to improve resilience on adversarial samples. [ABSTRACT FROM AUTHOR]

Federated learning (FL), as a distributed machine learning setting, has received considerable attention in recent years. To alleviate privacy concerns, FL essentially promises that multiple parties jointly train the model by exchanging gradients rather than raw data. However, intrinsic privacy issue still exists in FL, e.g., user’s training samples could be revealed by solely inferring gradients. Moreover, the emerging poisoning attack also poses a crucial security threat to FL. In particular, due to the distributed nature of FL, malicious users may submit crafted gradients during the training process to undermine the integrity and availability of the model. Furthermore, there exists a contradiction in simultaneously addressing two issues, that is, privacy-preserving FL solutions are dedicated to ensuring gradients indistinguishability, whereas the defenses against poisoning attacks tend to remove outliers based on their similarity. To solve such a dilemma, in this paper, we aim to build a bridge between the two issues. Specifically, we present a privacy-enhanced FL (PEFL) framework that adopts homomorphic encryption as the underlying technology and provides the server with a channel to punish poisoners via the effective gradient data extraction of the logarithmic function. To the best of our knowledge, the PEFL is the first effort to efficiently detect the poisoning behaviors in FL under ciphertext. Detailed theoretical analyses illustrate the security and convergence properties of the scheme. Moreover, the experiments conducted on real-world datasets show that the PEFL can effectively defend against label-flipping and backdoor attacks, two representative poisoning attacks in FL. [ABSTRACT FROM AUTHOR]

Recent studies have shown that robustness to adversarial attacks can be transferred across deep neural networks. In other words, we can make a weak model more robust with the help of a strong teacher model. In this paper, we ask if models can “learn together” and “teach each other” to achieve better robustness instead of learning from a static teacher. We study how interactions among models enhance robustness via knowledge distillation. We propose mutual adversarial training (MAT), in which multiple models are trained together and share the knowledge of adversarial examples to achieve improved robustness. MAT allows robust models to explore a larger space of adversarial samples and find more robust feature spaces and decision boundaries. Through extensive experiments on the CIFAR-10, CIFAR-100, and mini-ImageNet datasets, we demonstrate that MAT can effectively improve model robustness and outperform state-of-the-art methods under white-box attacks. In addition, we show that MAT can also mitigate the robustness trade-off among different perturbation types. Specially, we train specialist models that learn to defend a specific perturbation type and a generalist model that learns to defend multiple perturbation types by learning from the specialists, which brings as much as 13.4% accuracy gain to AT baselines against the union of $l_{\infty} $ , $l_{2}$ , and $l_{1}$ attacks. Our results show the effectiveness of the proposed method and demonstrate that collaborative learning is an effective strategy for designing robust models. [ABSTRACT FROM AUTHOR]

We give an overview of recent developments in the modeling of radiowave propagation, based on machine learning (ML) algorithms. We identify the input and output specification and the architecture of the model as the main challenges associated with ML-driven propagation models. Relevant papers are discussed and categorized based on their approach to each of these challenges. Emphasis is given on presenting the prospects and open problems in this promising and rapidly evolving area. [ABSTRACT FROM AUTHOR]

A deep belief network (DBN) is effective to create a powerful generative model by using training data. However, it is difficult to fast determine its optimal structure given specific applications. In this paper, a growing DBN with transfer learning (TL-GDBN) is proposed to automatically decide its structure size, which can accelerate its learning process and improve model accuracy. First, a basic DBN structure with single hidden layer is initialized and then pretrained, and the learned weight parameters are frozen. Second, TL-GDBN uses TL to transfer the knowledge from the learned weight parameters to newly added neurons and hidden layers, which can achieve a growing structure until the stopping criterion for pretraining is satisfied. Third, the weight parameters derived from pretraining of TL-GDBN are further fine-tuned by using layer-by-layer partial least square regression from top to bottom, which can avoid many problems of traditional backpropagation algorithm-based fine-tuning. Moreover, the convergence analysis of the TL-GDBN is presented. Finally, TL-GDBN is tested on two benchmark data sets and a practical wastewater treatment system. The simulation results show that it has better modeling performance, faster learning speed, and more robust structure than existing models. Note to Practitioners—Transfer learning (TL) aims to improve training effectiveness by transferring knowledge from a source domain to target domain. This paper presents a growing deep belief network (DBN) with TL to improve the training effectiveness and determine the optimal model size. Facing a complex process and real-world workflow, DBN tends to require long time for its successful training. The proposed growing DBN with TL (TL-GDBN) accelerates the learning process by instantaneously transferring the knowledge from a source domain to each new deeper or wider substructure. The experimental results show that the proposed TL-GDBN model has a great potential to deal with complex system, especially the systems with high nonlinearity. As a result, it can be readily applicable to some industrial nonlinear systems. [ABSTRACT FROM AUTHOR]

Mobile crowdsensing (MCS) combined with federated learning, as an emerging data collection and intelligent process paradigm, has received lots of attention in social networks and mobile Internet-of-Things, etc. However, as the openness and transparent of mobile crowdsensing tasks, federated learning model and training samples for crowdsensing data still face enormous privacy revealing risks, and it will reduce the willingness of people or nodes to actively participate and provide data in MCS. In this paper, we present a Privacy-Enhanced Aggregation for Federated Learning in MCS, namely PrivacyEAFL, to implement the training of federated learning under mobile crowdsensing system in terms of privacy protection of all participants. Firstly, considering that the crowdsensing server might share information with some participants to obtain and leak some local models, we design a collusion-resistant data aggregation approach by combining homomorphic cryptosystem and hashed Diffie-Hellman key exchange protocol. Secondly, we design a data encoding and aggregating method with data packing which can reduce the computation cost and communication overhead for the system. Thirdly, as the number of participants’ samples are dynamically changeable in MCS, we design a sample number protection method that can implement the security and privacy of the number of training samples owned by participants. Finally, we provide the experimental results on real-world datasets (i.e, MNIST and Car Evaluation) with crowdsensing devices under Raspberry-Pi 4B and Redmi-K30 Pro, respectively, and the results demonstrate that our scheme is more efficient and practical in secure and privacy-enhanced model aggregation for federated learning in mobile crowdsensing. [ABSTRACT FROM AUTHOR]

Federated learning (FL) is a privacy-preserving machine learning paradigm that enables multiple clients to train a unified model without disclosing their private data. However, susceptibility to membership inference attacks (MIAs) arises due to the natural inclination of FL models to overfit on the training data during the training process, thereby enabling MIAs to exploit the subtle differences in the FL model’s parameters, activations, or predictions between the training and testing data to infer membership information. It is worth noting that most if not all existing MIAs against FL require access to the model’s internal information or modification of the training process, yielding them unlikely to be performed in practice. In this paper, we present with TEAR the first evidence that it is possible for an honest-but-curious federated client to perform MIA against an FL system, by exploring the Temporal Evolution of the Adversarial Robustness between the training and non-training data. We design a novel adversarial example generation method to quantify the target sample’s adversarial robustness, which can be utilized to obtain the membership features to train the inference model in a supervised manner. Extensive experiment results on five realistic datasets demonstrate that TEAR can achieve a strong inference performance compared with two existing MIAs, and is able to escape from the protection of two representative defenses. [ABSTRACT FROM AUTHOR]

How should affect be appropriately annotated and how should machine learning best be employed to map manifestations of affect to affect annotations? What is the use of ratings of affect for the study of affective computing and how should we treat them? These are the key questions this paper attempts to address by investigating the impact of dissimilar representations of annotated affect on the efficacy of affect modelling. In particular, we compare several different binary-class and pairwise preference representations for automatically learning from ratings of affect. The representations are compared and tested on three datasets: one synthetic dataset (testing “in vitro ”) and two affective datasets (testing “in vivo”). The synthetic dataset couples a number of attributes with generated rating values. The two affective datasets contain physiological and contextual user attributes, and speech attributes, respectively; these attributes are coupled with ratings of various affective and cognitive states. The main results of the paper suggest that ratings (when used) should be naturally transformed to ordinal (ranked) representations for obtaining more reliable and generalisable models of affect. The findings of this paper have a direct impact on affect annotation and modelling research but, most importantly, challenge the traditional state-of-practice in affective computing and psychometrics at large. [ABSTRACT FROM PUBLISHER]

Generalization ability, which reflects the prediction ability of a learned model, is an important property in genetic programming (GP) for symbolic regression. Structural risk minimization (SRM) is a framework providing a reliable estimation of the generalization performance of prediction models. Introducing the framework into GP has the potential to drive the evolutionary process toward models with good generalization performance. However, this is tough due to the difficulty in obtaining the Vapnik–Chervonenkis (VC) dimension of nonlinear models. To address this difficulty, this paper proposes an SRM-driven GP approach, which uses an experimental method (instead of theoretical estimation) to measure the VC dimension of a mixture of linear and nonlinear regression models for the first time. The experimental method has been conducted using uniform and nonuniform settings. The results show that our method has impressive generalization gains over standard GP and GP with the 0.632 bootstrap, and that the proposed method using the nonuniform setting has further improvement than its counterpart using the uniform setting. Further analyzes reveal that the proposed method can evolve more compact models, and that the behavioral difference between these compact models and the target models is much smaller than their counterparts evolved by the other GP methods. [ABSTRACT FROM AUTHOR]

We present a novel approach to deformable object manipulation that does not rely on highly accurate modeling. The key contribution of this paper is to formulate the task as a multiarmed bandit problem, with each arm representing a model of the deformable object. To “pull” an arm and evaluate its utility, we use the arm’s model to generate a velocity command for the gripper(s) holding the object and execute it. As the task proceeds and the object deforms, the utility of each model can change. Our framework estimates these changes and balances the exploration of the model set with exploitation of high-utility models. We also propose an approach based on Kalman filtering for nonstationary multiarmed normal bandits to leverage the coupling between the models to learn more from each arm pull. We demonstrate that our method outperforms previous methods on synthetic trials and performs competitively on several manipulation tasks in simulation. Note to Practitioners—This paper is motivated by the problem of how to choose appropriate parameters for models used in deformable object manipulation. Existing approaches use time-consuming data collection and analysis methods to generate models for a given object and/or task. In contrast, our approach can be applied to multiple deformable objects or tasks without this time-consuming model generation step, learning what models are useful during the execution of the task itself. This enables the faster development of systems for a wide range of tasks, including those where a priori data collection is not possible. In this paper, we are only considering the modeling and control aspects of deformable object manipulation. These ideas can then be integrated with sensing and planning systems for commercial/industrial applications. We briefly discuss some methods and considerations for such an integration at the end of this paper. [ABSTRACT FROM AUTHOR]

There is an increasingly higher number of mixed-signal circuits within microprocessors. A significant portion of them corresponds to high-speed input/output (HSIO) links. Post-silicon validation of HSIO links is critical to provide a release qualification decision. One of the major challenges in HSIO electrical validation is the physical layer (PHY) tuning process, where equalization techniques are typically used to cancel any undesired effect. Current industrial practices for PHY tuning in HSIO links are very time-consuming, since they require massive laboratory measurements. On the other hand, surrogate modeling techniques allow to develop an approximation of a system response within a design space of interest. In this paper, we analyze several surrogate modeling methods and design of experiments techniques to identify the best approach to efficiently optimize a receiver equalizer. We evaluate the models performance by comparing with actual measured responses on a real server HSIO link. We then perform a surrogate-based optimization on the best model to obtain the optimal PHY tuning settings of an HSIO link. Our methodology is validated by measuring the real functional eye diagram of the physical system using the optimal surrogate model solution. [ABSTRACT FROM AUTHOR]

To exploit massive amounts of data generated at mobile edge networks, federated learning (FL) has been proposed as an attractive substitute for centralized machine learning (ML). By collaboratively training a shared learning model at edge devices, FL avoids direct data transmission and thus overcomes high communication latency and privacy issues as compared to centralized ML. To improve the communication efficiency in FL model aggregation, over-the-air computation has been introduced to support a large number of simultaneous local model uploading by exploiting the inherent superposition property of wireless channels. However, due to the heterogeneity of communication capacities among edge devices, over-the-air FL suffers from the straggler issue in which the device with the weakest channel acts as a bottleneck of the model aggregation performance. This issue can be alleviated by device selection to some extent, but the latter still suffers from a tradeoff between data exploitation and model communication. In this paper, we leverage the reconfigurable intelligent surface (RIS) technology to relieve the straggler issue in over-the-air FL. Specifically, we develop a learning analysis framework to quantitatively characterize the impact of device selection and model aggregation error on the convergence of over-the-air FL. Then, we formulate a unified communication-learning optimization problem to jointly optimize device selection, over-the-air transceiver design, and RIS configuration. Numerical experiments show that the proposed design achieves substantial learning accuracy improvement compared with the state-of-the-art approaches, especially when channel conditions vary dramatically across edge devices. [ABSTRACT FROM AUTHOR]

Automated machine learning (AutoML) has achieved remarkable progress on various tasks, which is attributed to its minimal involvement of manual feature and model designs. However, most of existing AutoML pipelines only touch parts of the full machine learning pipeline, e.g., neural architecture search or optimizer selection. This leaves potentially important components such as data cleaning and model ensemble out of the optimization, and still results in considerable human involvement and suboptimal performance. The main challenges lie in the huge search space assembling all possibilities over all components, as well as the generalization ability over different tasks like image, text, and tabular etc. In this paper, we present a first-of-its-kind fully AutoML pipeline, to comprehensively automate data preprocessing, feature engineering, model generation/selection/training and ensemble for an arbitrary dataset and evaluation metric. Our innovation lies in the comprehensive scope of a learning pipeline, with a novel “life-long” knowledge anchor design to fundamentally accelerate the search over the full search space. Such knowledge anchors record detailed information of pipelines and integrates them with an evolutionary algorithm for joint optimization across components. Experiments demonstrate that the result pipeline achieves state-of-the-art performance on multiple datasets and modalities. Specifically, the proposed framework was extensively evaluated in the NeurIPS 2019 AutoDL challenge, and won the only champion with a significant gap against other approaches, on all the image, video, speech, text and tabular tracks. [ABSTRACT FROM AUTHOR]

Deep learning has aroused a lot of attention and has been used successfully in many domains, such as accurate image recognition and medical diagnosis. Generally, the training of models requires large, representative datasets, which may be collected from a large number of users and contain sensitive information (e.g., users’ photos and medical information). The collected data would be stored and computed by service providers (SPs) or delegated to an untrusted cloud. The users can neither control how it will be used, nor realize what will be learned from it, which make the privacy issues prominent and severe. To solve the privacy issues, one of the most popular approaches is to encrypt users’ data with their public keys. However, this technique inevitably leads to another challenge that how to train the model based on multi-key encrypted data. In this paper, we propose a novel privacy-preserving deep learning model, namely PDLM, to apply deep learning over the encrypted data under multiple keys. In PDLM, lots of users contribute their encrypted data to SP to learn a specific model. We adopt an effective privacy-preserving calculation toolkit to achieve the training process based on stochastic gradient descent (SGD) in a privacy-preserving manner. We also prove that our PDLM can achieve users’ privacy preservation and analyze the efficiency of PDLM in theory. Finally, we conduct an experiment to evaluate PDLM over two real-world datasets and empirical results demonstrate that our PDLM can effectively and efficiently train the model in a privacy-preserving way. [ABSTRACT FROM AUTHOR]

High-dimensional and sparse (HiDS) data with non-negativity constraints are commonly seen in industrial applications, such as recommender systems. They can be modeled into an HiDS matrix, from which non-negative latent factor analysis (NLFA) is highly effective in extracting useful features. Preforming NLFA on an HiDS matrix is ill-posed, desiring an effective regularization scheme for avoiding overfitting. Current models mostly adopt a standard ${L} _{2}$ scheme, which does not consider the imbalanced distribution of known data in an HiDS matrix. From this point of view, this paper proposes an instance-frequency-weighted regularization (IR) scheme for NLFA on HiDS data. It specifies the regularization effects on each latent factors with its relevant instance count, i.e., instance-frequency, which clearly describes the known data distribution of an HiDS matrix. By doing so, it achieves finely grained modeling of regularization effects. The experimental results on HiDS matrices from industrial applications demonstrate that compared with an ${L} _{2}$ scheme, an IR scheme enables a resultant model to achieve higher accuracy in missing data estimation of an HiDS matrix. [ABSTRACT FROM AUTHOR]

Recent adversarial attack works attempt to improve the transferability by applying various differentiable transformations on input images. Considering the differentiable transformations and the original model together as a new model, these methods can be regarded as model augmentation that effectively derives an ensemble of models from the single original model. Despite their impressive performance, the model augmentation policies used in these methods are manually designed by experimental attempts, leaving the design of model augmentation policy an open question. In this paper, we propose an Automatic Model Augmentation (AutoMA) approach to find a strong model augmentation policy for transferable adversarial attacks. Specifically, we design a discrete search space that contains various diffierentiable transformations with different parameters and adopt reinforcement learning to search for the strong augmentation policy. The sampled augmentation policies together with the rewards they obtain during the searching process reveal several valuable observations for designing more powerful attacks using model augmentation policy: 1) Augmentation transformations on color space are less effective; 2) The transformation type diversity matters; and 3) Using small distortion for geometric transformations while larger distortion for intensity transformations. Extensive experiments show that the augmentation policy found by AutoMA achieves superior performance than existing manually designed policies in a wide range of cases. [ABSTRACT FROM AUTHOR]

Clinically oriented deep learning algorithms, combined with large-scale medical datasets, have significantly promoted computer-aided diagnosis. To address increasing ethical and privacy issues, Federated Learning (FL) adopts a distributed paradigm to collaboratively train models, rather than collecting samples from multiple institutions for centralized training. Despite intensive research on FL, two major challenges are still existing when applying FL in the real-world medical scenarios, including the performance degradation (i.e., retrogress) after each communication and the intractable class imbalance. Thus, in this paper, we propose a novel personalized FL framework to tackle these two problems. For the retrogress problem, we first devise a Progressive Fourier Aggregation (PFA) at the server side to gradually integrate parameters of client models in the frequency domain. Then, at the client side, we design a Deputy-Enhanced Transfer (DET) to smoothly transfer global knowledge to the personalized local model. For the class imbalance problem, we propose the Conjoint Prototype-Aligned (CPA) loss to facilitate the balanced optimization of the FL framework. Considering the inaccessibility of private local data to other participants in FL, the CPA loss calculates the global conjoint objective based on global imbalance, and then adjusts the client-side local training through the prototype-aligned refinement to eliminate the imbalance gap with such a balanced goal. Extensive experiments are performed on real-world dermoscopic and prostate MRI FL datasets. The experimental results demonstrate the advantages of our FL framework in real-world medical scenarios, by outperforming state-of-the-art FL methods with a large margin. The source code is available at https://github.com/CityU-AIM-Group/PRR-Imbalance https://github.com/CityU-AIM-Group/PRR-Imbalance. [ABSTRACT FROM AUTHOR]

Federated learning (FL) is an emerging privacy-preserving paradigm that enables multiple participants collaboratively to train a global model without uploading raw data. Considering heterogeneous computing and communication capabilities of different participants, asynchronous FL can avoid the stragglers effect in synchronous FL and adapts to scenarios with vast participants. Both staleness and non-IID data in asynchronous FL would reduce the model utility. However, there exists an inherent contradiction between the solutions to the two problems. That is, mitigating the staleness requires to select less but consistent gradients while coping with non-IID data demands more comprehensive gradients. To address the dilemma, this paper proposes a two-stage weighted $K$ K asynchronous FL with adaptive learning rate (WKAFL). By selecting consistent gradients and adjusting learning rate adaptively, WKAFL utilizes stale gradients and mitigates the impact of non-IID data, which can achieve multifaceted enhancement in training speed, prediction accuracy and training stability. We also present the convergence analysis for WKAFL under the assumption of unbounded staleness to understand the impact of staleness and non-IID data. Experiments implemented on both benchmark and synthetic FL datasets show that WKAFL has better overall performance compared to existing algorithms. [ABSTRACT FROM AUTHOR]

Intrinsic image decomposition is the task of mapping image to albedo and shading. Classical approaches derive methods from spatial models. The modern literature stresses evaluation, by comparing predictions to human judgements (“lighter”, “same as”, “darker”). The best modern intrinsic image methods train a map from image to albedo using images rendered from computer graphics models and example human judgements. This approach yields practical methods, but obtaining rendered images can be inconvenient. Furthermore, the approach cannot explain how a one could learn to recover intrinsic images without geometric, surface and illumination models, as people and animals appear to do. This paper describes a method that learns intrinsic image decomposition without seeing human annotations, rendered data, or ground truth data. Instead, the method relies on paradigms – spatial models of albedo and of shading. Rather than finding the “best” albedo and shading for an image via optimization, our approach trains a neural network on synthetic images. The synthetic images are constructed by multiplying albedos and shading fields sampled from our models. The network is subject to a novel smoothing procedure that ensures good behavior at short scales on real images. An averaging procedure ensures that reported albedo and shading are largely equivariant – different crops and scalings of an image will report the same albedo and shading at shared points. This averaging procedure controls long scale error. The standard evaluation for an intrinsic image method is a WHDR score. Our method achieves WHDR scores competitive with those of strong recent methods allowed to see training WHDR annotations, rendered data, and ground truth data. Our method produces albedo and shading maps with attractive qualitative properties – for example, albedo fields do not suppress wood grain and represent narrow grooves in surfaces well. Because our method is unsupervised, we can compute estimates of the test/train variance of WHDR scores; these are quite large, and suggest is unsafe to rely small differences in reported WHDR. [ABSTRACT FROM AUTHOR]

Traditional Deep Neural Network (DNN) security is mostly related to the well-known adversarial input example attack. Recently, another dimension of adversarial attack, namely, attack on DNN weight parameters, has been shown to be very powerful. As a representative one, the Bit-Flip-based adversarial weight Attack (BFA) injects an extremely small amount of faults into weight parameters to hijack the executing DNN function. Prior works of BFA focus on un-targeted attack that can hack all inputs into a random output class by flipping a very small number of weight bits stored in computer memory. This paper proposes the first work of targeted BFA based (T-BFA) adversarial weight attack on DNNs, which can intentionally mislead selected inputs to a target output class. The objective is achieved by identifying the weight bits that are highly associated with classification of a targeted output through a class-dependent vulnerable weight bit searching algorithm. Our proposed T-BFA performance is successfully demonstrated on multiple DNN architectures for image classification tasks. For example, by merely flipping 27 out of 88 million weight bits of ResNet-18, our T-BFA can misclassify all the images from ’Hen’ class into ’Goose’ class (i.e., 100% attack success rate) in ImageNet dataset, while maintaining 59.35% validation accuracy. Moreover, we successfully demonstrate our T-BFA attack in a real computer prototype system running DNN computation, with Ivy Bridge-based Intel i7 CPU and 8GB DDR3 memory. [ABSTRACT FROM AUTHOR]

Accurate solar photovoltaic (PV) generation forecast is critical to the reliable and economic operation of a modern power system. In practice, due to various faulty issues in the sensor, communication, or database system, the historical and online measurement data may not be always complete, and the missing data could dramatically degrade the forecasting model's accuracy. To solve this problem, this paper proposes an integrated missing-data tolerant model for probabilistic PV power generation forecasting. Taking historical PV generations as input, this model is based on a recursive long short-term memory network (Rec-LSTM), which can provide multi-step ahead forecasting of the probability distribution of PV generation. The unobserved input data will be imputed recursively based on the model output at the previous time step. During the training process, the imputations and forecasting values are iteratively updated by the negative log-likelihood loss function. As a salient advantage, this method can deal with data missing scenarios at both offline and online stages. Numerical experiments are conducted on two one-year datasets from Australia and Singapore, respectively. Probabilistic forecasting for both large-scale and small-scale building-level PV power generation is tested at the time resolution of 15 mins. Testing results show the proposed method can achieve superior probabilistic prediction accuracy as well as strong robustness under various data missing scenarios, compared to other state-of-the-art methods. [ABSTRACT FROM AUTHOR]

As an emulous alternative to deep neural networks, Deep Forest emerges with features like low complexity, fewer hyper-parameters, and good robustness, which are predominantly desired in distributed computing applications and ecosystems. Recently, an efficient distributed Deep Forest system, named ForestLayer, was proposed, designing a fine-grained sub-Forest-based task-parallel algorithm to improve the parallel computing efficiency of Deep Forest. However, the sub-Forest splitting of ForestLayer is static and one-off without adaptability to the computing environment, nevertheless, the size of splitting granularity has a significant impact on the system performance. To further improve the computing efficiency and scalability of the distributed Deep Forest, in this paper, we propose a novel distributed Deep Forest algorithm, named BLB-gcForest (Bag of Little Bootstraps-gcForest), which augments the gcForest (multi-Grained Cascade Forest) approach for constructing Deep Forest. BLB-gcForest carries out parallel computation for each tree in sub-Forests at a finer parallel granularity and integrates with the Bag of Little Bootstraps (BLB) mechanism to reduce massive transmitted feature instances for Cascade Forest Layers, utterly improving both computation efficiency and communication efficiency. Moreover, to solve the problem of the forest splitting granularity, we further design an adaptive sub-Forest splitting algorithm to ensure the maximum resource utilization for parallel computation of each sub-Forest. Experimental results on four well-known large-scale datasets, namely YEAST, LETTER, MNIST, CIFAR10, show that the training efficiency of BLB-gcForest achieves up to 20.3x and 1.64x speedups compared with the state-of-the-art gcForest and ForestLayer, respectively while guaranteeing higher accuracy and better robustness [ABSTRACT FROM AUTHOR]

With the integration of Artificial Intelligence (AI) and Internet of Things (IoT), the Federated Edge Learning (FEL), a promising computing framework is developing. However, there are still unsolved issues on communication efficiency and data security due to the huge models and unreliable transmission links. To address these issues, this paper proposes a novel federated edge learning system, called LightFed, where the edge nodes upload only vital partial local models, and successfully achieve lightweight communication and model aggregation. First, a novel model aggregation method Model Splitting and Splicing (MSS) and a Selective Parameter Transmission (SPT) scheme are proposed. By detecting the updating gradients of local parameters and filtering significant parameters, selective rotated transmission and efficient aggregation of local models are achieved. Second, a Training Filling Model (TFM) is proposed to infer the total data distribution of edge nodes, and train a filling model to mitigate the unbalanced training data without violating the data privacy of individual users. Moreover, a blockchain-powered confusion transmission mechanism is proposed for defending the attacks from external adversaries and protecting the model information. Finally, extensive experimental results demonstrate that our LightFed significantly outperforms the existing FEL systems in terms of communication efficiency and privacy security. [ABSTRACT FROM AUTHOR]

The accuracy of artificial intelligence (AI) models is crucial for connected and autonomous vehicles (CAVs). However, in reality, model training under less frequent weather faces the problem of insufficient sampling. Also, in the real world, weather, sunlight, etc., can only change with the speed of the real-time clock, so the traditional sampling process is very slow. Moreover, currently, collective learning, which can make up the limited experience and computing power of a single vehicle, is always introduced to cases where the data from participants have the same structure, wasting massive heterogeneous data from vehicles of different brands. Therefore, in this paper, we propose a quantum collective learning and many-to-many matching game-based scheme in the metaverse for CAVs. The environment is simulated in the metaverse, which has its own time clock system, thereby expanding sample size and speeding up the sampling process. And we quantify the quality of intelligence in collective learning from the perspective of feature diversity. It is the cornerstone of collective learning between heterogeneous vehicles, facilitating maximum utilization of data with different structures. Then, we formulate the distributed vehicles selection problem as a many-to-many matching game and use Gale–Shapely algorithm to solve it. Also, we formulate the spectrum resource allocation problem as a discrete Markov decision process (MDP) and adopt a quantum-inspired reinforcement learning (QRL) algorithm to find the optimal policy to achieve the high revenue of the system. In simulations, the performance of the proposed scheme is compared with existing methods. [ABSTRACT FROM AUTHOR]

In the past few years, kernel-based one-class extreme learning machine (ELM) receives quite a lot of attention by researchers for offline/batch learning due to its noniterative and fast learning capability. This paper extends this concept for adaptive online learning with regularized kernel-based one-class ELM classifiers for detection of outliers, and are collectively referred to as ORK-OCELM. Two frameworks, viz., boundary and reconstruction, are presented to detect the target class in ORK-OCELM. The kernel hyperplane-based baseline one-class ELM model considers whole data in a single chunk, however, the proposed one-class classifiers are adapted in an online fashion from the stream of training samples. The performance of ORK-OCELM is evaluated on a standard benchmark as well as synthetic datasets for both types of environments, i.e., stationary and nonstationary. While evaluating on stationary datasets, these classifiers are compared against batch learning-based one-class classifiers. Similarly, while evaluating on nonstationary datasets, the comparison is done with incremental learning-based online one-class classifiers. The results indicate that the proposed classifiers yield better or similar outcomes for both. In the nonstationary dataset evaluation, adaptability of the proposed classifiers in a changing environment is also demonstrated. It is further shown that the proposed classifiers have large stream data handling capability even under limited system memory. Moreover, the proposed classifiers gain significant time improvement compared to traditional online one-class classifiers (in all aspects of training and testing). A faster learning ability of the proposed classifiers makes them more suitable for real-time anomaly detection. [ABSTRACT FROM AUTHOR]

Deep Learning and Machine Learning models have recently been shown to be effective in many real world applications. While these models achieve increasingly better predictive performance, their structures have also become much more complex. A common and difficult problem for complex models is overfitting. Regularization is used to penalize the complexity of the model in order to avoid overfitting. However, in most learning frameworks, regularization function is usually set with some hyper-parameters where the best setting is difficult to find. In this paper, we propose an adaptive regularization method, as part of a large end-to-end healthcare data analytics software stack, which effectively addresses the above difficulty. First, we propose a general adaptive regularization method based on Gaussian Mixture (GM) to learn the best regularization function according to the observed parameters. Second, we develop an effective update algorithm which integrates Expectation Maximization (EM) with Stochastic Gradient Descent (SGD). Third, we design a lazy update and sparse update algorithm to reduce the computational cost by 4x and 20x, respectively. The overall regularization framework is fast, adaptive, and easy-to-use. We validate the effectiveness of our regularization method through an extensive experimental study over 14 standard benchmark datasets and three kinds of deep learning/machine learning models. The results illustrate that our proposed adaptive regularization method achieves significant improvement over state-of-the-art regularization methods. [ABSTRACT FROM AUTHOR]

In order to retain more feature information of local areas on a point cloud, local grouping and subsampling are the necessary data structuring steps in most hierarchical deep learning models. Due to the disorder nature of the points in a point cloud, the significant time cost may be consumed when grouping and subsampling the points, which consequently results in poor scalability. This paper proposes a fast data structuring method called PSNet (Point Structuring Net). PSNet transforms the spatial features of the points and matches them to the features of local areas in a point cloud. PSNet achieves grouping and sampling at the same time while the existing methods process sampling and grouping in two separate steps (such as using FPS plus kNN). PSNet performs feature transformation pointwise while the existing methods uses the spatial relationship among the points as the reference for grouping. Thanks to these features, PSNet has two important advantages: 1) the grouping and sampling results obtained by PSNet is stable and permutation invariant; and 2) PSNet can be easily parallelized. PSNet can replace the data structuring methods in the mainstream point cloud deep learning models in a plug-and-play manner. We have conducted extensive experiments. The results show that PSNet can improve the training and inference speed significantly while maintaining the model accuracy. [ABSTRACT FROM AUTHOR]

With substantial amount of time, resources and human (team) efforts invested to explore and develop successful deep neural networks (DNN), there emerges an urgent need to protect these inventions from being illegally copied, redistributed, or abused without respecting the intellectual properties of legitimate owners. Following recent progresses along this line, we investigate a number of watermark-based DNN ownership verification methods in the face of ambiguity attacks, which aim to cast doubts on the ownership verification by forging counterfeit watermarks. It is shown that ambiguity attacks pose serious threats to existing DNN watermarking methods. As remedies to the above-mentioned loophole, this paper proposes novel passport-based DNN ownership verification schemes which are both robust to network modifications and resilient to ambiguity attacks. The gist of embedding digital passports is to design and train DNN models in a way such that, the DNN inference performance of an original task will be significantly deteriorated due to forged passports. In other words, genuine passports are not only verified by looking for the predefined signatures, but also reasserted by the unyielding DNN model inference performances. Extensive experimental results justify the effectiveness of the proposed passport-based DNN ownership verification schemes. Code is available at https://github.com/kamwoh/DeepIPR. [ABSTRACT FROM AUTHOR]

Deep Neural Networks (DNNs) often rely on vast datasets for training. Given the large size of such datasets, it is conceivable that they contain specific samples that either do not contribute or negatively impact the DNN’s optimization. Modifying the training distribution to exclude such samples could provide an effective solution to improve performance and reduce training time. This paper proposes to scale up ensemble Active Learning (AL) methods to perform acquisition at a large scale (10k to 500k samples at a time). We do this with ensembles of hundreds of models, obtained at a minimal computational cost by reusing intermediate training checkpoints. This allows us to automatically and efficiently perform a training data subset search for large labeled datasets. We observe that our approach obtains favorable subsets of training data, which can be used to train more accurate DNNs than training with the entire dataset. We perform an extensive experimental study of this phenomenon on three image classification benchmarks (CIFAR-10, CIFAR-100, and ImageNet), as well as an internal object detection benchmark for prototyping perception models for autonomous driving. Unlike existing studies, our experiments on object detection are at the scale required for production-ready autonomous driving systems. We provide insights on the impact of different initialization schemes, acquisition functions, and ensemble configurations at this scale. Our results provide strong empirical evidence that optimizing the training data distribution can significantly benefit large-scale vision tasks. [ABSTRACT FROM AUTHOR]

Traffic flow modeling for traffic state estimation is a vital component in many traffic management and operation systems. To leverage both machine learning (ML) methods and classical traffic flow models, the previous study has developed a hybrid framework for encoding traffic flow into multivariant Gaussian Process. However, the computational efficiency is low due to multiple inputs, outputs and equations. To improve the efficiency of the previous method, this paper presents a new modeling framework, named gradual physics regularized learning, to incrementally encode complex traffic flow models into the ML process. More specifically, the method starts with the involvement of traffic flow models from the lower-order version, such as the fundamental diagram and the kinetic wave models. Then the learned parameters and hyperparameters can be further fine-tuned with the high-order models. A field test based on real-world freeway measurements indicates the proposed model can leverage the additional physical equations to achieve better performance in estimation accuracy and robustness. Meanwhile, the gradual learning method can significantly reduce the computational efforts and further enables its application to scenarios with either larger datasets or more complex traffic flow models. [ABSTRACT FROM AUTHOR]

Accurate short-term traffic flow prediction plays an important role in the field of modern Intelligent Transportation Systems. Since various uncontrollable factors (e.g. weather, traffic jams or accidents), collected traffic data inevitably contain outliers. This makes it challenge to achieve satisfactory results for traffic flow prediction. Least Squares Twin Support Vector Regression (LSTSVR) has been shown to provide a powerful potential in nonlinear prediction problems. This is especially true when using appropriate heuristic algorithms to determine the parameters of nonlinear LSTSVR. In view of this, a novel LSTSVR model based on the robust $\text{L}_{2,\mathrm {p}}$ -norm ($0< p\le 2$) distance is proposed to alleviate the negative effect of traffic data with outliers, called PLSTSVR. An iterative algorithm is designed to solve the optimization problem of PLSTSVR, which has great potential for solving other relevant optimization problems. To search the parameters of constructed PLSTSVR, this paper constructs two traffic flow prediction models based on PLSTSVR and heuristic algorithms (Fruit Fly Optimization Algorithm and Particle Swarm Optimization), called PLSTSVR-FOA and PLSTSVR-PSO. Extensive experiments demonstrate that the constructed models are more effective and robust than other competing models in various experimental settings. [ABSTRACT FROM AUTHOR]

Wide models such as generalized linear models and factorization-based models have been extensively used in various predictive applications, e.g., recommendation, CTR prediction, and image recognition. Due to the memory bounded property of the models, the performance improvement on CPU is reaching the limitation. GPU is known to have many computation units and high memory bandwidth, and becomes a promising platform for training machine learning models. However, the GPU training for the wide models is far from optimal due to the sparsity and irregularity in wide models. The existing GPU-based wide models are even slower than the ones using CPU. The classical training schema of the wide models does not optimized for the GPU architecture, which suffers from large amount of random memory accesses and redundant read/write of intermediate values. In this paper, we propose an efficient GPU-training framework for the large-scale wide models, named cuWide. To fully benefit from the memory hierarchy of GPU, cuWide applies a new flow-based schema for training, which leverages the spatial and temporal locality of wide models to drastically reduce the amount of communication with GPU global memory. To do so, we adopt a bigraph computation model to efficiently realize the flow-based schema and exploit three flexible interfaces for programming. Further, we use the 2D partition of mini-batch (in sample and feature dimensions) with proposed graph abstraction to optimize GPU memory access for sparse data, and apply several spatial-temporal caching mechanisms (importance-based model caching and cross-stage accumulation caching mechanisms) to achieve a high performance kernel. To efficiently implement cuWide, we also propose several GPU-oriented optimizations, including feature-oriented data layout to enhance the data locality, replication mechanism to reduce update conflicts in shared memory, and multi-stream scheduling to overlap data transferring and kernel computing. We show that cuWide can be up to more than 20× faster than the state-of-the-art GPU solutions and multi-core CPU solutions. [ABSTRACT FROM AUTHOR]

The paper presents several novel residual recurrent neural network (RNN) models for digital predistortion of radio-frequency power amplifiers. Then, unstructured and structured neural network (NN) pruning algorithms are proposed to reduce the per-sample computational complexity and the required memory, without compromising their modeling performance. The mathematical equations of the complexity and the memory are derived to analyze the hardware cost and design low cost models. An experimental platform is developed to measure the modeling performance in terms of adjacent channel leakage ratio (ACLR), normalized mean square error, error vector magnitude (EVM) and bit error rate. The platform uses an envelope tracking PA excited by 80 MHz IEEE 802.11ac signals. Compared with the state-of-the-art RNN model in (Sun et al., 2019), the proposed models reduce the per-sample complexity and required memory by $94\hbox{--}99$ %, they also improve the ACLR and EVM by at least 3 dB and 1 dB at 0.45 dB and 3.15 dB gain compression, respectively. Furthermore, an observation path with time alignment is used to remove the transmit latency in the state-of-the-art approach. At 3.15 dB gain compression, the proposed models outperform the state-of-the-art feedforward and convolutional NN models by at least 1.3 dB in EVM, outperform the general memory polynomial model by at least 3.3 dB in EVM. The above evidences indicate that the proposed RNN models can be implemented with relatively low hardware cost and achieve the best EVM in high power efficiency scenario. [ABSTRACT FROM AUTHOR]

Federated Learning (FL) has been recognized as a privacy-preserving machine learning (ML) technology that enables collaborative training and learning of a global ML model based on the aggregation of distributed local model updates. However, security and privacy guarantees could be compromised due to malicious participants and the centralized aggregation manner. Possessing attractive features like decentralization, immutability and auditability, Blockchain is promising to enable a tamper-proof and trust-free framework to enhance performance and security in IoT based FL systems. However, directly integrating blockchains into the large scale IoT-based FL scenarios still faces many limitations, such as high computation and storage demands, low transactions throughput, poor scalability and challenges in privacy preservation. This paper proposes $\mu $ DFL, a novel hierarchical IoT network fabric for decentralized federated learning (DFL) atop of a lightweight blockchain called microchain. Following the hierarchical infrastructure of FL, participants in $\mu $ DFL are fragmented into multiple small scale microchains. Each microchain network relies on a hybrid Proof of Credit (PoC) block generation and Voting-based Chain Finality (VCF) consensus protocol to ensure efficiency and privacy-preservation at the network of edge. Meanwhile, microchains are federated vie a high-level inter-chain network, which adopts an efficient Byzantine Fault Tolerance (BFT) consensus protocol to achieve scalability and security. A proof-of-concept prototype is implemented, and the experimental results verify the feasibility of the proposed $\mu $ DFL solution in cross-devices FL settings with efficiency, security and privacy guarantees. [ABSTRACT FROM AUTHOR]

Linear estimation of signals is often based on covariance matrices estimated from training, which can perform poorly if the training data are limited and the estimated covariance matrices are ill-conditioned. Shrinking the covariance matrix toward a scaled identity matrix can improve the robustness against the model uncertainty provided the shrinkage factor is appropriately chosen. This paper introduces several cross-validation schemes for choosing the shrinkage factors in applications where the covariance matrices are replaced with sample covariance matrices or constructed from least squares estimates of the linear model parameters. For cases where the training and out-of-training data are identically distributed, we derive leave-one-out cross-validation (LOOCV) schemes that repeatedly split the training data with respect to time to determine the optimal shrinkage factors for either model fitting or signal estimation. For cases where they are distributed differently, we develop alternative LOOCV schemes that repeatedly split the out-of-training observations with respect to space. We derive computationally efficient implementations of those schemes and provide examples to demonstrate their performance. [ABSTRACT FROM AUTHOR]

Outlier detection is an important topic in machine learning and has been used in a wide range of applications. In this paper, we approach outlier detection as a binary-classification issue by sampling potential outliers from a uniform reference distribution. However, due to the sparsity of data in high-dimensional space, a limited number of potential outliers may fail to provide sufficient information to assist the classifier in describing a boundary that can separate outliers from normal data effectively. To address this, we propose a novel Single-Objective Generative Adversarial Active Learning (SO-GAAL) method for outlier detection, which can directly generate informative potential outliers based on the mini-max game between a generator and a discriminator. Moreover, to prevent the generator from falling into the mode collapsing problem, the stop node of training should be determined when SO-GAAL is able to provide sufficient information. But without any prior information, it is extremely difficult for SO-GAAL. Therefore, we expand the network structure of SO-GAAL from a single generator to multiple generators with different objectives (MO-GAAL), which can generate a reasonable reference distribution for the whole dataset. We empirically compare the proposed approach with several state-of-the-art outlier detection methods on both synthetic and real-world datasets. The results show that MO-GAAL outperforms its competitors in the majority of cases, especially for datasets with various cluster types or high irrelevant variable ratio. The experiment codes are available at: https://github.com/leibinghe/GAAL-based-outlier-detection. [ABSTRACT FROM AUTHOR]

Domain adaptation is to transfer the shared knowledge learned from the source domain to a new environment, i.e., target domain. One common practice is to train the model on both labeled source-domain data and unlabeled target-domain data. Yet the learned models are usually biased due to the strong supervision of the source domain. Most researchers adopt the early-stopping strategy to prevent over-fitting, but when to stop training remains a challenging problem since the lack of the target-domain validation set. In this paper, we propose one efficient bootstrapping method, called Adaboost Student, explicitly learning complementary models during training and liberating users from empirical early stopping. Adaboost Student combines deep model learning with the conventional training strategy, i.e., adaptive boosting, and enables interactions between learned models and the data sampler. We adopt one adaptive data sampler to progressively facilitate learning on hard samples and aggregate “weak” models to prevent over-fitting. Extensive experiments show that (1) Without the need to worry about the stopping time, AdaBoost Student provides one robust solution by efficient complementary model learning during training. (2) AdaBoost Student is orthogonal to most domain adaptation methods, which can be combined with existing approaches to further improve the state-of-the-art performance. We have achieved competitive results on three widely-used scene segmentation domain adaptation benchmarks. [ABSTRACT FROM AUTHOR]

Over the past years, the increasingly severe data island problem has spawned an emerging distributed deep learning framework—federated learning, in which the global model can be constructed over multiple participants without directly sharing their raw data. Despite its promising prospect, there are still many security challenges in federated learning, such as privacy preservation and integrity verification. Furthermore, federated learning is usually performed with the assistance of a center, which is prone to cause trust worries and communicational bottlenecks. To tackle these challenges, in this paper, we propose a privacy-preserving and verifiable decentralized federated learning framework, named PVD-FL, which can achieve secure deep learning model training under a decentralized architecture. Specifically, we first design an efficient and verifiable cipher-based matrix multiplication (EVCM) algorithm to execute the most basic calculation in deep learning. Then, by employing EVCM, we design a suite of decentralized algorithms to construct the PVD-FL framework, which ensures the confidentiality of both global model and local update and the verification of every training step. Detailed security analysis shows that PVD-FL can well protect privacy against various inference attacks and guarantee training integrity. In addition, the extensive experiments on real-world datasets also demonstrate that PVD-FL can achieve lossless accuracy and practical performance. [ABSTRACT FROM AUTHOR]

Model compression is a widely-used approach for reducing the size of deep learning models without much accuracy loss, enabling resource-hungry models to be compressed for use on resource-constrained devices. In this paper, we study the risk that model compression could provide an opportunity for adversaries to inject stealthy backdoors. In a backdoor attack on a machine learning model, an adversary produces a model that performs well on normal inputs but outputs targeted misclassifications on inputs containing a small trigger pattern. We design stealthy backdoor attacks such that the full-sized model released by adversaries appears to be free from backdoors (even when tested using state-of-the-art techniques), but when the model is compressed it exhibits a highly effective backdoor. We show this can be done for two common model compression techniques—model pruning and model quantization—even in settings where the adversary has limited knowledge of how the particular compression will be done. Our findings demonstrate the importance of performing security tests on the models that will actually be deployed not in their precompressed version. Our implementation is available at https://github.com/yulongtzzz/Stealthy-Backdoors-as-Compression-Artifacts. [ABSTRACT FROM AUTHOR]

In recent years, data has become the new oil that fuels various machine learning (ML) applications. Just as the oil refining, providing data to an ML model is a product of massive costs and expertise efforts. However, how to protect the intellectual property (IP) of the training data in ML remains largely open. In this paper, we present MeFA, a novel framework for detecting training data IP embezzlement via Membership Fingerprint Authentication, which is able to determine whether a suspect ML model is trained on the to be protected target data or not. The key observation is that a part of data has a similar influence on the prediction behavior of different ML models. On this basis, MeFA leverages membership inference techniques to extract these data as the fingerprints of the target data and constructs an authentication model to verify the data’s ownership by identifying the obtained membership fingerprints. MeFA has several salient features. It does not assume any knowledge of the suspect model except for its black-box prediction API, through which we can merely get the prediction output of a given input, and also does not require any modification to the dataset or the training process, since it takes advantage of the inherent membership property of the data. As a by-product, MeFA can also serve as a post-protection to verify the ownership of ML models, without modifying the training process of the model. Extensive experiments on three realistic datasets and seven types of ML models validate the effectiveness of MeFA, and demonstrate that it is also robust to scenarios when the training data is partially used or preprocessed with representative membership inference defenses. [ABSTRACT FROM AUTHOR]

An emerging computational paradigm, named federated edge learning (FEL), enables intelligent computing at the network edge with the feature of preserving data privacy for edge devices. Given their constrained resources, it becomes a great challenge to achieve high execution performance for FEL. Most of the state-of-the-arts concentrate on enhancing FEL from the perspective of system operation procedures, taking few precautions during the composition step of the FEL system. Though a few recent studies recognize the importance of FEL formation and propose server-centric device selection schemes, the impact of data sizes is largely overlooked. In this paper, we take advantage of game theory to depict the decision dilemma among edge devices regarding whether to participate in FEL or not given their heterogeneous sizes of local datasets. For realizing both the individual and global optimization, the server is employed to solve the participation dilemma, which requires accurate information collection for devices’ local datasets. Hence, we utilize mechanism design to enable truthful information solicitation. With the help of correlated equilibrium, we derive a decision making strategy for devices from the global perspective, which can achieve the long-term stability and efficacy of FEL. For scalability consideration, we optimize the computational complexity of the basic solution to the polynomial level. Lastly, extensive experiments based on both real and synthetic data are conducted to evaluate our proposed mechanisms, with experimental results demonstrating the performance advantages. [ABSTRACT FROM AUTHOR]

Actuated by the growing attention to personal healthcare and the pandemic, the popularity of E-health is proliferating. Nowadays, enhancement on medical diagnosis via machine learning models has been highly effective in many aspects of e-health analytics. Nevertheless, in the classic cloud-based/centralized e-health paradigms, all the data will be centrally stored on the server to facilitate model training, which inevitably incurs privacy concerns and high time delay. Distributed solutions like Decentralized Stochastic Gradient Descent (D-SGD) are proposed to provide safe and timely diagnostic results based on personal devices. However, methods like D-SGD are subject to the gradient vanishing issue and usually proceed slowly at the early training stage, thereby impeding the effectiveness and efficiency of training. In addition, existing methods are prone to learning models that are biased towards users with dense data, compromising the fairness when providing E-health analytics for minority groups. In this paper, we propose a Decentralized Block Coordinate Descent (D-BCD) learning framework that can better optimize deep neural network-based models distributed on decentralized devices for E-health analytics. As a gradient-free optimization method, Block Coordinate Descent (BCD) mitigates the gradient vanishing issue and converges faster at the early stage compared with the conventional gradient-based optimization. To overcome the potential data scarcity issues for users’ local data, we propose similarity-based model aggregation that allows each on-device model to leverage knowledge from similar neighbor models, so as to achieve both personalization and high accuracy for the learned models. Benchmarking experiments on three real-world datasets illustrate the effectiveness and practicality of our proposed D-BCD, where additional simulation study showcases the strong applicability of D-BCD in real-life E-health scenarios. [ABSTRACT FROM AUTHOR]

Communication limitation at the edge is widely recognized as a major bottleneck for federated learning (FL). Multi-hop wireless networking provides a cost-effective solution to enhance service coverage and spectrum efficiency at the edge, which could facilitate large-scale and efficient machine learning (ML) model aggregation. However, FL over multi-hop wireless networks has rarely been investigated. In this paper, we optimize FL over wireless mesh networks by taking into account the heterogeneity in communication and computing resources at mesh routers and clients. We present a framework that each intermediate router performs in-network model aggregation before sending the data to the next hop, so as to reduce the outgoing data traffic and hence aggregate more models under limited communication resources. To accelerate model training, we formulate our optimization problem by jointly considering model aggregation, routing, and spectrum allocation. Although the problem is a non-convex mixed-integer nonlinear programming, we transform it into a mixed-integer linear programming (MILP), and develop a coarse-grained fixing procedure to solve it efficiently. Simulation results demonstrate the effectiveness of the solution approach, and the superiority of the in-network aggregation scheme over the counterpart without in-network aggregation. [ABSTRACT FROM AUTHOR]

Machine learning (ML) has attracted a great research interest for physical layer design problems, such as channel estimation, thanks to its low complexity and robustness. Channel estimation via ML requires model training on a dataset, which usually includes the received pilot signals as input and channel data as output. In previous works, model training is mostly done via centralized learning (CL), where the whole training dataset is collected from the users at the base station (BS). This approach introduces huge communication overhead for data collection. In this paper, to address this challenge, we propose a federated learning (FL) framework for channel estimation. We design a convolutional neural network (CNN) trained on the local datasets of the users without sending them to the BS. We develop FL-based channel estimation schemes for both conventional and RIS (intelligent reflecting surface) assisted massive MIMO (multiple-input multiple-output) systems, where a single CNN is trained for two different datasets for both scenarios. We evaluate the performance for noisy and quantized model transmission and show that the proposed approach provides approximately 16 times lower overhead than CL, while maintaining satisfactory performance close to CL. Furthermore, the proposed architecture exhibits lower estimation error than the state-of-the-art ML-based schemes. [ABSTRACT FROM AUTHOR]