20 results on '"compressed learning"'
Number of results to display per page
Search Results
2. Deep Learning Models for Inference on Compressed Signals with Known or Unknown Measurement Matrix
- Author
-
Yu, Huiyuan, Cheng, Maggie, Goos, Gerhard, Series Editor, Hartmanis, Juris, Founding Editor, Bertino, Elisa, Editorial Board Member, Gao, Wen, Editorial Board Member, Steffen, Bernhard, Editorial Board Member, Yung, Moti, Editorial Board Member, Antonacopoulos, Apostolos, editor, Chaudhuri, Subhasis, editor, Chellappa, Rama, editor, Liu, Cheng-Lin, editor, Bhattacharya, Saumik, editor, and Pal, Umapada, editor
- Published
- 2025
- Full Text
- View/download PDF
3. Privacy-Preserving Federated Compressed Learning Against Data Reconstruction Attacks Based on Secure Data
- Author
-
Xiao, Di, Li, Jinkun, Li, Min, Filipe, Joaquim, Editorial Board Member, Ghosh, Ashish, Editorial Board Member, Prates, Raquel Oliveira, Editorial Board Member, Zhou, Lizhu, Editorial Board Member, Luo, Biao, editor, Cheng, Long, editor, Wu, Zheng-Guang, editor, Li, Hongyi, editor, and Li, Chaojie, editor
- Published
- 2024
- Full Text
- View/download PDF
4. Compressed Sensing-Based IoMT Applications
- Author
-
Lal, Bharat, Li, Qimeng, Gravina, Raffaele, Corsonello, Pasquale, Fortino, Giancarlo, Series Editor, Liotta, Antonio, Series Editor, Savaglio, Claudio, editor, Zhou, MengChu, editor, and Ma, Jianhua, editor
- Published
- 2024
- Full Text
- View/download PDF
5. Compressed Principal Component Regression (C–PCR) Algorithm and FPGA Validation.
- Author
-
Zamani, Hossein, Bahrami, Hamid Reza, Garris, Paul A., and Mohseni, Pedram
- Abstract
To address the hardware and/or software implementation issues of principal component regression (PCR), we propose a novel algorithm called compressed PCR (C–PCR). C–PCR projects the input data to a lower dimensional space first, and then applies the compressed data to a significantly smaller PCR engine. We show that C–PCR can lower the computational complexity of PCR with a factor of compression ratio (CR) squared, i.e., CR2. Moreover, the output signal of C–PCR follows that of PCR with a small error, which increases with CR, when the projections are random. Using datasets of prerecorded brain neurochemicals, we experimentally show that C–PCR can achieve CRs as high as ~ 10. As far as hardware implementation is concerned, the experimental results show that reduction rates of 32% to 45% in different FPGA resources can be achieved using C–PCR. [ABSTRACT FROM AUTHOR]
- Published
- 2020
- Full Text
- View/download PDF
6. Compressive Gaussian Mixture Estimation
- Author
-
Bourrier, Anthony, Gribonval, Rémi, Pérez, Patrick, Benedetto, John J., Series editor, Boche, Holger, editor, Calderbank, Robert, editor, Kutyniok, Gitta, editor, and Vybíral, Jan, editor
- Published
- 2015
- Full Text
- View/download PDF
7. Compressed Learning of Deep Neural Networks for OpenCL-Capable Embedded Systems.
- Author
-
Lee, Sangkyun and Lee, Jeonghyun
- Subjects
ARTIFICIAL neural networks ,DEEP learning ,SPARSE matrices ,DATA structures ,IMAGE compression ,IMAGE representation - Abstract
Deep neural networks (DNNs) have been quite successful in solving many complex learning problems. However, DNNs tend to have a large number of learning parameters, leading to a large memory and computation requirement. In this paper, we propose a model compression framework for efficient training and inference of deep neural networks on embedded systems. Our framework provides data structures and kernels for OpenCL-based parallel forward and backward computation in a compressed form. In particular, our method learns sparse representations of parameters using ℓ 1 -based sparse coding while training, storing them in compressed sparse matrices. Unlike the previous works, our method does not require a pre-trained model as an input and therefore can be more versatile for different application environments. Even though the use of ℓ 1 -based sparse coding for model compression is not new, we show that it can be far more effective than previously reported when we use proximal point algorithms and the technique of debiasing. Our experiments show that our method can produce minimal learning models suitable for small embedded devices. [ABSTRACT FROM AUTHOR]
- Published
- 2019
- Full Text
- View/download PDF
8. Dimension-Adaptive Bounds on Compressive FLD Classification
- Author
-
Kabán, Ata, Durrant, Robert J., Hutchison, David, editor, Kanade, Takeo, editor, Kittler, Josef, editor, Kleinberg, Jon M., editor, Mattern, Friedemann, editor, Mitchell, John C., editor, Naor, Moni, editor, Nierstrasz, Oscar, editor, Pandu Rangan, C., editor, Steffen, Bernhard, editor, Sudan, Madhu, editor, Terzopoulos, Demetri, editor, Tygar, Doug, editor, Vardi, Moshe Y., editor, Weikum, Gerhard, editor, Goebel, Randy, editor, Siekmann, Jörg, editor, Wahlster, Wolfgang, editor, Jain, Sanjay, editor, Munos, Rémi, editor, Stephan, Frank, editor, and Zeugmann, Thomas, editor
- Published
- 2013
- Full Text
- View/download PDF
9. Robust and Lightweight Ensemble Extreme Learning Machine Engine Based on Eigenspace Domain for Compressed Learning.
- Author
-
Li, Huai-Ting, Chou, Ching-Yao, Chen, Yi-Ta, Wang, Sheng-Hui, and Wu, An-Yeu
- Subjects
- *
MACHINE learning , *HARDWARE , *VERY large scale circuit integration , *ROBUST control - Abstract
Compressive sensing (CS) is applied to electrocardiography (ECG) telemonitoring system to address the energy constraint of signal acquisition in sensors. In addition, on-sensor-analysis transmitting only abnormal data further reduces the energy consumption. To combine both advantages, “On-CS-sensor-analysis” can be achieved by compressed learning (CL), which analyzes signals directly in compressed domain. Extreme learning machine (ELM) provides an effective solution to achieve the goal of low-complexity CL. However, single ELM model has limited accuracy and is sensitive to the quality of data. Furthermore, hardware non-idealities in CS sensors result in learning performance degradation. In this work, we propose the ensemble of sub-eigenspace-ELM (SE-ELM), including two novel approaches: 1) We develop the eigenspace transformation for compressed noisy data, and further utilize a subspace-based dictionary to remove the interferences, and 2) Hardware-friendly design for ensemble of ELM provides high accuracy while maintaining low complexity. The simulation results on ECG-based atrial fibrillation show the SE-ELM can achieve the highest accuracy with 61.9% savings of the required multiplications compared with conventional methods. Finally, we implement this engine in TSMC 90 nm technology. The postlayout results show the proposed CL engine can provide competitive area- and energy-efficiency compared to existing machine learning engines. [ABSTRACT FROM AUTHOR]
- Published
- 2019
- Full Text
- View/download PDF
10. Compressed Learning of Deep Neural Networks for OpenCL-Capable Embedded Systems
- Author
-
Sangkyun Lee and Jeonghyun Lee
- Subjects
compressed learning ,regularization ,proximal point algorithm ,debiasing ,embedded systems ,OpenCL ,Technology ,Engineering (General). Civil engineering (General) ,TA1-2040 ,Biology (General) ,QH301-705.5 ,Physics ,QC1-999 ,Chemistry ,QD1-999 - Abstract
Deep neural networks (DNNs) have been quite successful in solving many complex learning problems. However, DNNs tend to have a large number of learning parameters, leading to a large memory and computation requirement. In this paper, we propose a model compression framework for efficient training and inference of deep neural networks on embedded systems. Our framework provides data structures and kernels for OpenCL-based parallel forward and backward computation in a compressed form. In particular, our method learns sparse representations of parameters using ℓ 1 -based sparse coding while training, storing them in compressed sparse matrices. Unlike the previous works, our method does not require a pre-trained model as an input and therefore can be more versatile for different application environments. Even though the use of ℓ 1 -based sparse coding for model compression is not new, we show that it can be far more effective than previously reported when we use proximal point algorithms and the technique of debiasing. Our experiments show that our method can produce minimal learning models suitable for small embedded devices.
- Published
- 2019
- Full Text
- View/download PDF
11. Learning in the compressed data domain: Application to milk quality prediction.
- Author
-
Vimalajeewa, Dixon, Kulatunga, Chamil, and Berry, Donagh P.
- Subjects
- *
DAIRY farming , *MILK quality , *PRINCIPAL components analysis , *REGRESSION analysis , *RANDOM variables - Abstract
Smart dairy farming has become one of the most exciting and challenging area in cloud-based data analytics. Transfer of raw data from all farms to a central cloud is currently not feasible as applications are generating more data while internet connectivity is lacking in rural farms. As a solution, Fog computing has become a key factor to process data near the farm and derive farm insights by exchanging data between on-farm applications and transferring some data to the cloud. In this context, learning in the compressed data domain, where de-compression is not necessary, is highly desirable as it minimizes the energy used for communication/computation, reduces required memory/storage, and improves application latency. Mid-infrared spectroscopy (MIRS) is used globally to predict several milk quality parameters as well as deriving many animal-level phenotypes. Therefore, compressed learning on MIRS data is beneficial both in terms of data processing in the Fog, as well as storing large data sets in the cloud. In this paper, we used principal component analysis and wavelet transform as two techniques for compressed learning to convert MIRS data into a compressed data domain. The study derives near lossless compression parameters for both techniques to transform MIRS data without impacting the prediction accuracy for a selection of milk quality traits. [ABSTRACT FROM AUTHOR]
- Published
- 2018
- Full Text
- View/download PDF
12. Inference on compressive measurements
- Author
-
Fernández Berni, Jorge, Rodríguez Sakamoto, Riu, Universidad de Sevilla. Departamento de Electrónica y Electromagnetismo, Jiménez Gómez, Marina, Fernández Berni, Jorge, Rodríguez Sakamoto, Riu, Universidad de Sevilla. Departamento de Electrónica y Electromagnetismo, and Jiménez Gómez, Marina
- Abstract
Compressed sensing is a new paradigm capable of sampling and compressing signals in one step. Its original purpose was to compress sparse or compressible signals in a way that reconstruction from the measurements taken were possible. However, many applications do not require signal recovery. Therefore, a new branch of compressed sensing aims to directly perform inference using the information encoded in the samples, instead of recovering the complete signal and then solving inference problems over the reconstructed signal. In this thesis, the mathematical framework for detection, classi cation, estimation and ltering on compressed measurements was studied. Moreover, applications of inference mainly based on machine learning as an implementation tool were reviewed. Finally, machine-learning algorithms were tested on compressed measurements., El muestreo compresivo es un nuevo paradigma capaz de muestrear y comprimir señales en un solo paso. Su objetivo original era el de comprimir señales sparse o compresibles de tal modo que su reconstrucción a partir de las medidas tomadas fuera posible. Sin embargo, muchas aplicaciones no requieren recuperar la señal. Por lo tanto, una nueva rama del muestreo compresivo pretende realizar problemas de inferencia directamente usando la información codifi cada en las muestras en lugar de recuperar completamente la señal y posteriormente resolver el problema de inferencia sobre la señal reconstruida. En este trabajo se estudia el marco matemático de los problemas de detección, clasifi cación, estimación y fi ltrado. Además, se revisan aplicaciones de inferencia principalmente basadas en el uso de herramientas de aprendizaje automático. Por último, se prueban algoritmos de aprendizaje automático sobre medidas comprimidas.
- Published
- 2021
13. Random projections as regularizers: learning a linear discriminant from fewer observations than dimensions.
- Author
-
Durrant, Robert and Kabán, Ata
- Subjects
MACHINE learning ,FISHER discriminant analysis ,MACHINE theory ,LEARNING classifier systems ,ALGORITHMS - Abstract
We prove theoretical guarantees for an averaging-ensemble of randomly projected Fisher linear discriminant classifiers, focusing on the case when there are fewer training observations than data dimensions. The specific form and simplicity of this ensemble permits a direct and much more detailed analysis than existing generic tools in previous works. In particular, we are able to derive the exact form of the generalization error of our ensemble, conditional on the training set, and based on this we give theoretical guarantees which directly link the performance of the ensemble to that of the corresponding linear discriminant learned in the full data space. To the best of our knowledge these are the first theoretical results to prove such an explicit link for any classifier and classifier ensemble pair. Furthermore we show that the randomly projected ensemble is equivalent to implementing a sophisticated regularization scheme to the linear discriminant learned in the original data space and this prevents overfitting in conditions of small sample size where pseudo-inverse FLD learned in the data space is provably poor. Our ensemble is learned from a set of randomly projected representations of the original high dimensional data and therefore for this approach data can be collected, stored and processed in such a compressed form. We confirm our theoretical findings with experiments, and demonstrate the utility of our approach on several datasets from the bioinformatics domain and one very high dimensional dataset from the drug discovery domain, both settings in which fewer observations than dimensions are the norm. [ABSTRACT FROM AUTHOR]
- Published
- 2015
- Full Text
- View/download PDF
14. Finding needles in compressed haystacks.
- Author
-
Calderbank, Robert and Jafarpour, Sina
- Abstract
In this paper, we investigate the problem of compressed learning, i.e. learning directly in the compressed domain. In particular, we provide tight bounds demonstrating that the linear kernel SVMs classifier in the measurement domain, with high probability, has true accuracy close to the accuracy of the best linear threshold classifier in the data domain. Furthermore, we indicate that for a family of well-known deterministic compressed sensing matrices, compressed learning is provided on the fly. Finally, we support our claims with experimental results in the texture analysis application. [ABSTRACT FROM PUBLISHER]
- Published
- 2012
- Full Text
- View/download PDF
15. Inference on compressive measurements
- Author
-
Jiménez Gómez, Marina, Fernández Berni, Jorge, Rodríguez Sakamoto, Riu, and Universidad de Sevilla. Departamento de Electrónica y Electromagnetismo
- Subjects
Machine Learning ,Detection ,Deep Neural Networks ,Compressed Sensing ,Inference ,Compressed Learning ,Classification ,Filtering ,Estimation - Abstract
Compressed sensing is a new paradigm capable of sampling and compressing signals in one step. Its original purpose was to compress sparse or compressible signals in a way that reconstruction from the measurements taken were possible. However, many applications do not require signal recovery. Therefore, a new branch of compressed sensing aims to directly perform inference using the information encoded in the samples, instead of recovering the complete signal and then solving inference problems over the reconstructed signal. In this thesis, the mathematical framework for detection, classi cation, estimation and ltering on compressed measurements was studied. Moreover, applications of inference mainly based on machine learning as an implementation tool were reviewed. Finally, machine-learning algorithms were tested on compressed measurements. El muestreo compresivo es un nuevo paradigma capaz de muestrear y comprimir señales en un solo paso. Su objetivo original era el de comprimir señales sparse o compresibles de tal modo que su reconstrucción a partir de las medidas tomadas fuera posible. Sin embargo, muchas aplicaciones no requieren recuperar la señal. Por lo tanto, una nueva rama del muestreo compresivo pretende realizar problemas de inferencia directamente usando la información codifi cada en las muestras en lugar de recuperar completamente la señal y posteriormente resolver el problema de inferencia sobre la señal reconstruida. En este trabajo se estudia el marco matemático de los problemas de detección, clasifi cación, estimación y fi ltrado. Además, se revisan aplicaciones de inferencia principalmente basadas en el uso de herramientas de aprendizaje automático. Por último, se prueban algoritmos de aprendizaje automático sobre medidas comprimidas. Universidad de Sevilla. Doble Grado en Física y Matemáticas
- Published
- 2021
16. The Benefits of an Accelerated Learning Format in Teacher Education Programs.
- Author
-
Richards, Jan
- Subjects
ACCELERATED teaching ,EDUCATIONAL acceleration ,ADVANCED placement programs (Education) ,SCHOOL administration ,ABILITY grouping (Education) ,CONCENTRATED study ,EDUCATIONAL programs ,LITERACY programs - Abstract
Because our student population is increasing, 50,000 additional teachers will be needed in the United States within the next 10 years. Overcrowded teacher education programs in traditional universities cannot guarantee the availability of necessary classes, however, and students desiring a teaching credential anticipate an extended time frame for credential completion. There is growing interest in the benefits of accelerated programs to meet this critical need. Researchers have found that outcomes from such compressed courses equal (or surpass) outcomes from traditional course formats. National University employs such an accelerated format and has been highly successful in training future teachers for California classrooms. [ABSTRACT FROM AUTHOR]
- Published
- 2008
17. Enabling on-device classification of ECG with compressed learning for health IoT.
- Author
-
Li, Wenzhuo, Chu, Haoming, Huang, Boming, Huan, Yuxiang, Zheng, Lirong, and Zou, Zhuo
- Subjects
- *
CONVOLUTIONAL neural networks , *COMPUTER performance , *ELECTROCARDIOGRAPHY , *SIGNAL classification , *INTERNET of things , *IMAGE compression - Abstract
In this paper, an on-device classification of electrocardiography (ECG) with Compressed Learning (CL) for health Internet of Things (IoT) is proposed. A CL algorithm combining with one-dimensional (1-D) Convolutional Neural Network (CNN) that directly learns on ECG signals in the compression domain without expanded normalization is proposed. Such an approach bypasses the reconstruction step and minimizes the raw input data dimension that significantly reduces the processing power. An automatic network optimization framework with Automatic Machine Learning (AutoML) tool Neural Network Intelligence (NNI) is suggested to adapt to the network structure search problem introduced by input dimension reduction with Compression Ratios (CRs). That ensures the minimized model size and operation number under a guaranteed classification accuracy. To implement the resized 1-D CL classifier in hardware, which has different kernel sizes, strides, and output channels under different CRs, a flexible architecture is proposed to further lower power consumption. Evaluated on the MIT-BIH database, the specific 1-D CNN network selected under CR = 0. 2 achieves a Macro-F1 of 0.9214 on 5-class ECG signal classification, with a 6.4 × reduction in FLOPs and a 2.6 × decrease in model size with an only 0.028 loss in Macro-F1 compared with the uncompressed situation. Synthesized in UMC 40 nm Low Power process, the hardware architecture with the 1-D CL classifier achieves an energy efficiency of 0.83 μ J/Classification under a 1.1-V power supply at a frequency of 5 MHz. [ABSTRACT FROM AUTHOR]
- Published
- 2021
- Full Text
- View/download PDF
18. Sketching for Large-Scale Learning of Mixture Models
- Author
-
Nicolas Keriven, Anthony Bourrier, Rémi Gribonval, Patrick Pérez, Parcimonie et Nouveaux Algorithmes pour le Signal et la Modélisation Audio (PANAMA), Inria Rennes – Bretagne Atlantique, Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria)-SIGNAUX ET IMAGES NUMÉRIQUES, ROBOTIQUE (IRISA-D5), Institut de Recherche en Informatique et Systèmes Aléatoires (IRISA), CentraleSupélec-Télécom Bretagne-Université de Rennes 1 (UR1), Université de Rennes (UNIV-RENNES)-Université de Rennes (UNIV-RENNES)-Institut National de Recherche en Informatique et en Automatique (Inria)-École normale supérieure - Rennes (ENS Rennes)-Université de Bretagne Sud (UBS)-Centre National de la Recherche Scientifique (CNRS)-Institut National des Sciences Appliquées - Rennes (INSA Rennes), Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-Institut National des Sciences Appliquées (INSA)-CentraleSupélec-Télécom Bretagne-Université de Rennes 1 (UR1), Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-Institut National des Sciences Appliquées (INSA)-Institut de Recherche en Informatique et Systèmes Aléatoires (IRISA), Université de Rennes (UNIV-RENNES)-Université de Rennes (UNIV-RENNES)-École normale supérieure - Rennes (ENS Rennes)-Université de Bretagne Sud (UBS)-Centre National de la Recherche Scientifique (CNRS)-Institut National des Sciences Appliquées - Rennes (INSA Rennes), Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-Institut National des Sciences Appliquées (INSA), Technicolor [Cesson Sévigné], Technicolor, European Project: 277906,EC:FP7:ERC,ERC-2011-StG_20101014,PLEASE(2012), Université de Rennes 1 (UR1), Université de Rennes (UNIV-RENNES), Université de Bretagne Sud (UBS)-Institut National des Sciences Appliquées - Rennes (INSA Rennes), Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-Institut National de Recherche en Informatique et en Automatique (Inria)-École normale supérieure - Rennes (ENS Rennes)-Centre National de la Recherche Scientifique (CNRS)-Université de Rennes 1 (UR1), Université de Rennes (UNIV-RENNES)-CentraleSupélec-IMT Atlantique Bretagne-Pays de la Loire (IMT Atlantique), Institut Mines-Télécom [Paris] (IMT)-Institut Mines-Télécom [Paris] (IMT)-Université de Bretagne Sud (UBS)-Institut National des Sciences Appliquées - Rennes (INSA Rennes), Institut Mines-Télécom [Paris] (IMT)-Institut Mines-Télécom [Paris] (IMT)-Institut de Recherche en Informatique et Systèmes Aléatoires (IRISA), Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-École normale supérieure - Rennes (ENS Rennes)-Centre National de la Recherche Scientifique (CNRS)-Université de Rennes 1 (UR1), Institut Mines-Télécom [Paris] (IMT)-Institut Mines-Télécom [Paris] (IMT), GIPSA - Vision and Brain Signal Processing (GIPSA-VIBS), Département Images et Signal (GIPSA-DIS), Grenoble Images Parole Signal Automatique (GIPSA-lab ), Institut polytechnique de Grenoble - Grenoble Institute of Technology (Grenoble INP )-Institut Polytechnique de Grenoble - Grenoble Institute of Technology-Centre National de la Recherche Scientifique (CNRS)-Université Grenoble Alpes [2016-2019] (UGA [2016-2019])-Institut polytechnique de Grenoble - Grenoble Institute of Technology (Grenoble INP )-Institut Polytechnique de Grenoble - Grenoble Institute of Technology-Centre National de la Recherche Scientifique (CNRS)-Université Grenoble Alpes [2016-2019] (UGA [2016-2019])-Grenoble Images Parole Signal Automatique (GIPSA-lab ), Institut polytechnique de Grenoble - Grenoble Institute of Technology (Grenoble INP )-Institut Polytechnique de Grenoble - Grenoble Institute of Technology-Centre National de la Recherche Scientifique (CNRS)-Université Grenoble Alpes [2016-2019] (UGA [2016-2019])-Institut polytechnique de Grenoble - Grenoble Institute of Technology (Grenoble INP )-Institut Polytechnique de Grenoble - Grenoble Institute of Technology-Centre National de la Recherche Scientifique (CNRS)-Université Grenoble Alpes [2016-2019] (UGA [2016-2019]), Université de Rennes (UR)-Institut National des Sciences Appliquées - Rennes (INSA Rennes), Institut National des Sciences Appliquées (INSA)-Institut National des Sciences Appliquées (INSA)-Université de Bretagne Sud (UBS)-École normale supérieure - Rennes (ENS Rennes)-Institut National de Recherche en Informatique et en Automatique (Inria)-Télécom Bretagne-CentraleSupélec-Centre National de la Recherche Scientifique (CNRS)-Université de Rennes (UR)-Institut National des Sciences Appliquées - Rennes (INSA Rennes), Institut National des Sciences Appliquées (INSA)-Institut National des Sciences Appliquées (INSA)-Université de Bretagne Sud (UBS)-École normale supérieure - Rennes (ENS Rennes)-Institut National de Recherche en Informatique et en Automatique (Inria)-Télécom Bretagne-CentraleSupélec-Centre National de la Recherche Scientifique (CNRS)-Institut de Recherche en Informatique et Systèmes Aléatoires (IRISA), Institut National des Sciences Appliquées (INSA)-Institut National des Sciences Appliquées (INSA)-Université de Bretagne Sud (UBS)-École normale supérieure - Rennes (ENS Rennes)-Télécom Bretagne-CentraleSupélec-Centre National de la Recherche Scientifique (CNRS), Université de Rennes (UR), Institut National des Sciences Appliquées (INSA)-Institut National des Sciences Appliquées (INSA)-Université de Bretagne Sud (UBS)-École normale supérieure - Rennes (ENS Rennes)-Institut National de Recherche en Informatique et en Automatique (Inria)-CentraleSupélec-Centre National de la Recherche Scientifique (CNRS)-IMT Atlantique (IMT Atlantique), Institut Mines-Télécom [Paris] (IMT)-Institut Mines-Télécom [Paris] (IMT)-Université de Rennes (UR)-Institut National des Sciences Appliquées - Rennes (INSA Rennes), Institut National des Sciences Appliquées (INSA)-Institut National des Sciences Appliquées (INSA)-Université de Bretagne Sud (UBS)-École normale supérieure - Rennes (ENS Rennes)-CentraleSupélec-Centre National de la Recherche Scientifique (CNRS)-IMT Atlantique (IMT Atlantique), SIGNAUX ET IMAGES NUMÉRIQUES, ROBOTIQUE (IRISA-D5), Université de Rennes (UNIV-RENNES)-Université de Rennes (UNIV-RENNES)-Institut National des Sciences Appliquées - Rennes (INSA Rennes), Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-Institut National des Sciences Appliquées (INSA)-Université de Bretagne Sud (UBS)-École normale supérieure - Rennes (ENS Rennes)-Institut National de Recherche en Informatique et en Automatique (Inria)-CentraleSupélec-Centre National de la Recherche Scientifique (CNRS)-IMT Atlantique Bretagne-Pays de la Loire (IMT Atlantique), Institut Mines-Télécom [Paris] (IMT)-Institut Mines-Télécom [Paris] (IMT)-Université de Rennes 1 (UR1), Institut Mines-Télécom [Paris] (IMT)-Institut Mines-Télécom [Paris] (IMT)-Inria Rennes – Bretagne Atlantique, Institut National de Recherche en Informatique et en Automatique (Inria), Keriven, Nicolas, and PLEASE: Projections, Learning, and Sparsity for Efficient data-processing - PLEASE - - EC:FP7:ERC2012-01-01 - 2016-12-31 - 277906 - VALID
- Subjects
Statistics and Probability ,FOS: Computer and information sciences ,[MATH.MATH-PR] Mathematics [math]/Probability [math.PR] ,Scale (ratio) ,[INFO.INFO-TS] Computer Science [cs]/Signal and Image Processing ,Iterative method ,Computer science ,compressive sensing ,Context (language use) ,Machine Learning (stat.ML) ,02 engineering and technology ,010501 environmental sciences ,01 natural sciences ,Synthetic data ,Machine Learning (cs.LG) ,010104 statistics & probability ,Dimension (vector space) ,[INFO.INFO-LG]Computer Science [cs]/Machine Learning [cs.LG] ,Compressed Sensing ,[STAT.ML]Statistics [stat]/Machine Learning [stat.ML] ,[INFO.INFO-TS]Computer Science [cs]/Signal and Image Processing ,Statistics - Machine Learning ,0202 electrical engineering, electronic engineering, information engineering ,0101 mathematics ,0105 earth and related environmental sciences ,Numerical Analysis ,compressed learning ,database sketch ,business.industry ,Applied Mathematics ,020206 networking & telecommunications ,Pattern recognition ,gaussian mixture models ,Mixture model ,[STAT.ML] Statistics [stat]/Machine Learning [stat.ML] ,Sketch ,[MATH.MATH-PR]Mathematics [math]/Probability [math.PR] ,Computer Science - Learning ,Compressed sensing ,Kernel method ,Computational Theory and Mathematics ,compressive learning ,Probability distribution ,Artificial intelligence ,Heuristics ,business ,Gaussian mixture ,Algorithm ,Analysis - Abstract
to appear in Information and Inference, a journal of the IMA (available online since December 2017); International audience; Learning parameters from voluminous data can be prohibitive in terms of memory and computational requirements. We propose a " compressive learning " framework where we estimate model parameters from a sketch of the training data. This sketch is a collection of generalized moments of the underlying probability distribution of the data. It can be computed in a single pass on the training set, and is easily computable on streams or distributed datasets. The proposed framework shares similarities with compressive sensing, which aims at drastically reducing the dimension of high-dimensional signals while preserving the ability to reconstruct them. To perform the estimation task, we derive an iterative algorithm analogous to sparse reconstruction algorithms in the context of linear inverse problems. We exemplify our framework with the compressive estimation of a Gaussian Mixture Model (GMM), providing heuristics on the choice of the sketching procedure and theoretical guarantees of reconstruction. We experimentally show on synthetic data that the proposed algorithm yields results comparable to the classical Expectation-Maximization (EM) technique while requiring significantly less memory and fewer computations when the number of database elements is large. We further demonstrate the potential of the approach on real large-scale data (over 10 8 training samples) for the task of model-based speaker verification. Finally, we draw some connections between the proposed framework and approximate Hilbert space embedding of probability distributions using random features. We show that the proposed sketching operator can be seen as an innovative method to design translation-invariant kernels adapted to the analysis of GMMs. We also use this theoretical framework to derive information preservation guarantees, in the spirit of infinite-dimensional compressive sensing.
- Published
- 2016
- Full Text
- View/download PDF
19. Compressive Gaussian Mixture Estimation by Orthogonal Matching Pursuit with Replacement
- Author
-
Keriven, Nicolas, Gribonval, Rémi, Parcimonie et Nouveaux Algorithmes pour le Signal et la Modélisation Audio (PANAMA), SIGNAUX ET IMAGES NUMÉRIQUES, ROBOTIQUE (IRISA-D5), Institut de Recherche en Informatique et Systèmes Aléatoires (IRISA), Université de Rennes 1 (UR1), Université de Rennes (UNIV-RENNES)-Université de Rennes (UNIV-RENNES)-Institut National des Sciences Appliquées - Rennes (INSA Rennes), Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-Institut National des Sciences Appliquées (INSA)-Université de Bretagne Sud (UBS)-École normale supérieure - Rennes (ENS Rennes)-Institut National de Recherche en Informatique et en Automatique (Inria)-Télécom Bretagne-CentraleSupélec-Centre National de la Recherche Scientifique (CNRS)-Université de Rennes 1 (UR1), Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-Institut National des Sciences Appliquées (INSA)-Université de Bretagne Sud (UBS)-École normale supérieure - Rennes (ENS Rennes)-Institut National de Recherche en Informatique et en Automatique (Inria)-Télécom Bretagne-CentraleSupélec-Centre National de la Recherche Scientifique (CNRS)-Institut de Recherche en Informatique et Systèmes Aléatoires (IRISA), Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-Institut National des Sciences Appliquées (INSA)-Université de Bretagne Sud (UBS)-École normale supérieure - Rennes (ENS Rennes)-Institut National de Recherche en Informatique et en Automatique (Inria)-Télécom Bretagne-CentraleSupélec-Centre National de la Recherche Scientifique (CNRS)-Inria Rennes – Bretagne Atlantique, Institut National de Recherche en Informatique et en Automatique (Inria), European Project: 277906,EC:FP7:ERC,ERC-2011-StG_20101014,PLEASE(2012), Inria Rennes – Bretagne Atlantique, Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria)-SIGNAUX ET IMAGES NUMÉRIQUES, ROBOTIQUE (IRISA-D5), Université de Rennes (UR)-Institut National des Sciences Appliquées - Rennes (INSA Rennes), Institut National des Sciences Appliquées (INSA)-Institut National des Sciences Appliquées (INSA)-Université de Bretagne Sud (UBS)-École normale supérieure - Rennes (ENS Rennes)-Institut National de Recherche en Informatique et en Automatique (Inria)-Télécom Bretagne-CentraleSupélec-Centre National de la Recherche Scientifique (CNRS)-Université de Rennes (UR)-Institut National des Sciences Appliquées - Rennes (INSA Rennes), Institut National des Sciences Appliquées (INSA)-Institut National des Sciences Appliquées (INSA)-Université de Bretagne Sud (UBS)-École normale supérieure - Rennes (ENS Rennes)-Institut National de Recherche en Informatique et en Automatique (Inria)-Télécom Bretagne-CentraleSupélec-Centre National de la Recherche Scientifique (CNRS)-Institut de Recherche en Informatique et Systèmes Aléatoires (IRISA), and Institut National des Sciences Appliquées (INSA)-Institut National des Sciences Appliquées (INSA)-Université de Bretagne Sud (UBS)-École normale supérieure - Rennes (ENS Rennes)-Télécom Bretagne-CentraleSupélec-Centre National de la Recherche Scientifique (CNRS)
- Subjects
[MATH.MATH-PR]Mathematics [math]/Probability [math.PR] ,compressed learning ,database sketch ,Compressed Sensing ,[STAT.ML]Statistics [stat]/Machine Learning [stat.ML] ,[INFO.INFO-TS]Computer Science [cs]/Signal and Image Processing ,Gaussian mixture - Abstract
International audience; This work deals with the problem of fitting a Gaussian Mixture Model (GMM) to a large collection of data. Usual approaches such as the classical Expectation Maximization (EM) algorithm are known to perform well but require extensive access to the data. The proposed method compresses the entire database into a single low-dimensional sketch that can be computed in one pass then directly used for GMM estimation. This sketch can be seen as resulting from the application of a linear operator to the underlying probability distribution, thus establishing a connection between our method and generalized compressive sensing. In particular, the new algorithms introduced to estimate GMMs are similar to usual greedy algorithms in compressive sensing.
- Published
- 2015
20. Compressed learning
- Author
-
Zhou, T
- Subjects
Big data ,Compressed learning ,Sparse learning ,Machine learning ,Manifold learning - Abstract
University of Technology, Sydney. Faculty of Engineering and Information Technology. There has been an explosion of data derived from the internet and other digital sources. These data are usually multi-dimensional, massive in volume, frequently incomplete, noisy, and complicated in structure. These "big data" bring new challenges to machine learning (ML), which has historically been designed for small volumes of clearly defined and structured data. In this thesis we propose new methods of "compressed learning", which explore the components and procedures in ML methods that are compressible, in order to improve their robustness, scalability, adaptivity, and performance for big data analysis. We will study novel methodologies that compress different components throughout the learning process, propose more interpretable general compressible structures for big data, and develop effective strategies to leverage these compressible structures to produce highly scalable learning algorithms. We present several new insights into popular learning problems in the context of compressed learning. The theoretical analyses are tested on real data in order to demonstrate the efficacy and efficiency of the methodologies in real-world scenarios. In particular, we propose "manifold elastic net (MEN)" and "double shrinking (DS)" as two fast frameworks extracting low-dimensional sparse features for dimension reduction and manifold learning. These methods compress the features on both their dimension and cardinality, and significantly improve their interpretation and performance in clustering and classification tasks. We study how to derive fewer "anchor points" for representing large datasets in their entirety by proposing "divide-and-conquer anchoring", in which the global solution is rapidly found for near-separable non-negative matrix factorization and completion in a distributed manner. This method represents a compression of the big data itself, rather than features, and the extracted anchors define the structure of the data. Two fast low-rank approximation methods, "bilateral random projections (BRP)" of fast computer closed-form and "greedy bilateral sketch (GreBske)", are proposed based on random projection and greedy augmenting update rules. They can be broadly applied to learning procedures that requires updates of a low-rank matrix variable and result in significant acceleration in performance. We study how to compress noisy data for learning by decomposing it into the sum mixture of low-rank part and sparse part. "GO decomposition (GoDec)" and the "greedy bilateral (GreB)" paradigm are proposed as two efficient approaches to this problem based on randomized and greedy strategies, respectively. Modifications of these two schemes result in novel models and extremely fast algorithms for matrix completion that aim to recover a low-rank matrix from a small number of its entries. In addition, we extend the GoDec problem in order to unmix more than two incoherent structures that are more complicated and expressive than low-rank or sparse matrices. The three proposed variants are not only novel and effective algorithms for motion segmentation in computer vision, multi-label learning, and scoring-function learning in recommendation systems, but also reveal new theoretical insights into these problems. Finally, a compressed learning method termed “compressed labelling (CL) on distilled label sets (DL)" is proposed for solving the three core problems in multi-label learning, namely high-dimensional labels, label correlation modeling, and sample imbalance for each label. By compressing the labels and the number of classifiers in multi-label learning, CL can generate an effective and efficient training algorithm from any single-label classifier.
- Published
- 2013
Catalog
Books, media, physical & digital resources
Discovery Service for Jio Institute Digital Library
For full access to our library's resources, please sign in.