Search

IoT time series data is an important form of big data. How to improve the efficiency of storage system is crucial for IoT time series database to store and manage massive IoT time series data from various IoT devices. Mixing NVM and SSD is an effective method to improve the I/O performance of storage systems. However, there are great differences between HDD and NVM or SSD. As a result, NVM and SSD cannot be directly used in the current time series database effectively. We design an IoT time series database with an embedded engine in storage device drivers for the hybrid solid-state storage system consisting of NVM and SSD. The I/O software stack of storing and managing IoT time series data can be shortened to improve the efficiency. Based upon the intrinsic characteristics of IoT time series data and different features of NVM and SSD, a redundancy elimination and compression fusion strategy, a hierarchical management strategy, and a heterogeneous time series data index are designed to improve the efficiency. Finally, a prototype of embedded IoT time series database named TS-NSM is implemented, and YCSB-TS is used to measure the IOPS. The results show that TS-NSM can improve the write IOPS up to 243.6 times and 174.3 times, respectively, compared with InfluxDB and OpenTSDB, and improve the read IOPS up to 10.1 times and 14.4 times, respectively.

Training of large scale deep neural networks with distributed implementations is an effective way to improve the efficiency. However, high network communication cost for synchronizing gradients and parameters is a major bottleneck in distributed training. In this work, we propose an asynchronous training algorithm based on sampling and mean fusion for distributed recurrent neural network (RNN). In distributed RNN, multiple distributed neuron nodes and an interaction node work together to implement the training. The synchronization overhead is reduced by a unique asynchronous sampling strategy amongst the distributed neuron nodes. Then, in order to make up for the accuracy loss caused by the asynchronous parameter update, a mean fusion algorithm is proposed, where the interaction node averages all local parameters from the distributed neurons. We mathematically prove the convergence of the proposed algorithm. Experimental verification is performed on two language modeling benchmark datasets. The results demonstrate significant speed gains for distributed RNN, while the accuracy loss is less than 1% on average.

As a human-cortex-inspired computing model, hierarchical temporal memory (HTM) has shown great promise in sequence learning and has been applied to various time-series applications. HTM uses the combination of columns and neurons to learn the temporal patterns within the sequence. However, the conventional HTM model compacts the input into two naive column states—active and nonactive, and uses a fixed learning strategy. This simplicity limits the representation capability of HTM and ignores the impacts of active columns on learning the temporal context. To address these issues, we propose a new HTM algorithm based on activation intensity. By introducing the column activation intensity, more useful and fine-grained information from the input is retained for sequence learning. Furthermore, a self-adaptive nonlinear learning strategy is proposed where the synaptic connections are dynamically adjusted according to the activation intensity of columns. Extensive experiments are carried out on two real-world time-series datasets. Compared to the conventional HTM and LSTM model, our method achieved higher accuracy and less time overhead.

Hierarchical Temporal Memory is a new type of artificial neural network model, which imitates the structure and information processing flow of the human brain. Hierarchical Temporal Memory has strong adaptability and fast learning ability and becomes a hot spot in current research. Hierarchical Temporal Memory obtains and saves the temporal characteristics of input sequences by the temporal pool learning algorithm. However, the current algorithm has some problems such as low learning efficiency and poor learning effect when learning time series data. In this paper, a temporal pool learning algorithm based on location awareness is proposed. The cell selection rules based on location awareness and the dendritic updating rules based on adjacent inputs are designed to improve the learning efficiency and effect of the algorithm. Through the algorithm prototype, three different datasets are used to test and analyze the algorithm performance. The experimental results verify that the algorithm can quickly obtain the complete characteristics of the input sequence. No matter whether there are similar segments in the sequence, the proposed algorithm has higher prediction recall and precision than the existing algorithms.

As a new type of artificial neural network model, HTM has become the focus of current research and application. The sparse distributed representation is the basis of the HTM model, but the existing spatial pool learning algorithms have high training time overhead and may cause the spatial pool to become unstable. To overcome these disadvantages, we propose a fast spatial pool learning algorithm of HTM based on minicolumn’s nomination, where the minicolumns are selected according to the load-carrying capacity and the synapses are adjusted using compressed encoding. We have implemented the prototype of the algorithm and carried out experiments on three datasets. It is verified that the training time overhead of the proposed algorithm is almost unaffected by the encoding length, and the spatial pool becomes stable after fewer iterations of training. Moreover, the training of the new input does not affect the already trained results.

Reducing the numerical precision of weights and activations of deep neural networks have proven to be a stunningly efficient way of deploying deep networks on edge devices with limited resources. With the advent of the Transformer model, several quantization techniques have been proposed to reduce the computation and model size. However, these existing quantization techniques use fixed bit-width assignments, which result in a significant degradation in the accuracy of the model. We present in this work an efficient Transformer based on our novel multi-layer quantization technique, which reduces the precision of data based on the characteristics of weights and activations in each layer of the Transformer architecture while at the same time preserving the model’s structure. The WMT2014 DE-EN and WMT2014 FR-EN datasets are used to evaluate. The results show that our efficient Transformer achieves 4x compression with improved accuracy and an overall reduction in the training time overhead. By comparing with existing state-of-the-art techniques, we further proved that with a minimum of 3-bit and a maximum of 8-bit quantization, comparable state-of-the-art BLEU scores can be obtained.

Long short-term memory (LSTM) is an important model for sequential data processing. However, large amounts of matrix computations in LSTM unit seriously aggravate the training when model grows larger and deeper as well as more data become available. In this work, we propose an efficient distributed duration-aware LSTM(D-LSTM) for large scale sequential data analysis. We improve LSTM’s training performance from two aspects. First, the duration of sequence item is explored in order to design a computationally efficient cell, called duration-aware LSTM(D-LSTM) unit. With an additional mask gate, the D-LSTM cell is able to perceive the duration of sequence item and adopt an adaptive memory update accordingly. Secondly, on the basis of D-LSTM unit, a novel distributed training algorithm is proposed, where D-LSTM network is divided logically and multiple distributed neurons are introduced to perform the easier and concurrent linear calculations in parallel. Different from the physical division in model parallelism, the logical split based on hidden neurons can greatly reduce the communication overhead which is a major bottleneck in distributed training. We evaluate the effectiveness of the proposed method on two video datasets. The experimental results shown our distributed D-LSTM greatly reduces the training time and can improve the training efficiency for large scale sequence analysis.

Long short-term memory (LSTM) network is an effective model architecture for deep learning approaches to sequence modeling tasks. However, the current LSTMs can't use the property of sequential data when dealing with the sequence components, which last for a certain period of time. This may make the model unable to benefit from the inherent characteristics of time series and result in poor performance as well as lower efficiency. In this paper, we present a novel adaptive LSTM for durative sequential data which exploits the temporal continuance of the input data in designing a new LSTM unit. By adding a new mask gate and maintaining span, the cell's memory update is not only determined by the input data but also affected by its duration. An adaptive memory update method is proposed according to the change of the sequence input at each time step. This breaks the limitation that the cells calculate the cell state and hidden output for each input always in a unified manner, making the model more suitable for processing the sequences with continuous data. The experimental results on various sequence training tasks show that under the same iteration epochs, the proposed method can achieve higher accuracy, but need relatively less training time compared with the standard LSTM architecture.

The number of hidden neurons in recurrent neural networks (RNNs) greatly affects the model accuracy. When fewer neurons are used, less training time is involved and the accuracy is in a lower degree. Once we increase the number of hidden units, the performance will be encouraged but the training overhead will enhance exponentially. In this work, we propose a novel active propagation training strategy for distributed RNN, which is inspired by the information transmission in biological neuron system. With the aim to improve the training efficiency of RNN, the proposed active propagation avoids the traditional fixed number of neurons in training. Instead, when training the distributed RNN, the autonomous neuron dynamically activates new neurons and propagates the accumulated training information to them. The neurons not only learn from the training data but also from other neurons. Different from a fixed large number of neurons, the valid neurons are increased gradually and the model performance is promoted accordingly. A prototype of active propagation RNN (APRNN) is implemented on Spark and multiple evaluations are carried out on language modeling tasks. The results show that compared with the fixed-neurons training, APRNN can reduce the training time while keeping a comparable accuracy. The improvement is more significant on large scale training corpora.

The Resilient Distributed Dataset (RDD) cache is an important way to improve the efficiency of application on Spark. There are some specific characteristics of iterative application. We design the classified and active caching strategy for iterative application. It can reduce the time delay of RDD creation of data set for iterative application and avoid creating or recomputing it again during the iterative application. At the same time, the RDD of parameter set should be renewed and cached after every iteration to reduce the time overhead of the subsequent iteration. The prototype is implemented based on Spark and one iterative application is used to test and compared with Spark. The results show that the classified and active caching strategy can reduce the 12%-45% time overhead of iterative application.

The Non-Volatile Memory (NVM) device has higher read and write speed compared with disks and SSDs, and it is the hotspot to improve the efficiency of computer system. But it also makes the I/O software stack become the bottleneck of application's efficiency instead of storage devices. We analyse I/O software stack when the application accesses the NVM device and design a direct hybrid NVM file system which can bypass the VFS to reduce the extra time overhead of accessing the NVM device for the application. The application oriented direct file system interface is design to allow the application directly access the NVM device which reducing the response time and improving the I/O performance of applications. On this basis, we implement the prototype of DNVMCFS and use Fio to test its performance comparing with PMFS, NVMCFS and Ext4 on PMBD respectively. The results show that DNVMCFS can improve the I/O performance by 42.3% compared with PMFS, 50% compared with NVMCFS and 53.7% compared with Ext4 on PMBD. Meanwhile, DNVMCFS can also increase the IOPS where Filebench is used to simulate the real application test. The results show that DNMVCFS raises the I/O performance and IOPS by 25%. These results verify that DNVMCFS provides higher I/O performance and better adaptability.

RNNLM (Recurrent Neural Network Language Model) can save the historical information of the training dataset by the last hidden layer and can also as input for training. It has become an interesting topic in the field of Natural Language Processing research. However, the immense training time overhead is a big problem. The large output layer, hidden layer, last hidden layer and the connections among them will generate enormous matrix in training. It is the main facts to influence the efficiency and scalability. At the same time, output layer class and small hidden layer should decrease the accuracy of RNNLM. In general, the lack of parallel for artificial neuron is main reason for these. We change the structure of RNNLM and design the new large-scale RNNLM by the center of distributed artificial neurons in hidden layer to stimulate the parallel characteristic of biological neuron system. Meanwhile, we change training method, and present the coordination strategy for distributed neuron. At last, the prototype of new large-scale RNNLM system is implemented based on Spark. The testing and analysis results show that the training time overhead is far less than the growth rate of the distributed neuron in hidden layer and size of training dataset. These results show our large-scale RNNLM system has efficiency and scalability advantage.

Semantic categorization for the complex videos is an ambiguous task. The semi-supervised learning method based on hypergraph model can achieve multi-semantics labels, but a hypergraph model is sensitive to the radius parameter when it is constructed and the number of vertices belonging to a hyperedge is fixed. In this paper, a semi-supervised incremental learning method based on adaptive probabilistic hypergraph for video semantic detection is presented. In the probabilistic hypergraph modeling, a formula is presented as a measurement to adaptively decide whether a vertex is belonged to a hyperedge or not. The model has high robustness and can overcome the defect of fixed number of vertices belonging to the same hyperedge in the traditional probabilistic hypergraph model. In the semi-supervised incremental learning process, a threshold is defined, which is used to judge whether unlabeled sample can be added into the modeling, in order that the model learning result for unlabeled samples has high certainty. The experimental results show that our method can improve the model generalization ability and utilize the unlabeled samples effectively. In the aspects of recall rate and precision rate for semantic video concept detection from complex videos, our proposed method outperforms the compared methods.

The storage device based on Nonvolatile Memory (NVM devices) has high read/write speed and embedded processor. It is a useful way to improve the efficiency of Key-Value (KV) application. However it still has some limitations such as limited capacity, poorer computing power compared with CPU, and complex I/O system software. Thus it is not an effective way to construct KV storage system with NVM devices directly. We analyze the characteristics of NVM devices and demands of KV application to design the matrix KV storage system based on NVM Devices. The group collaboration management based on Bloomfilter, intragroup optimization based on competition, embedded KV management based on B+-tree, and the new interface of KV storage system are presented. Then, the embedded processor in the NVM device and CPU can be comprehensively utilized to construct a matrix KV pair management system. It can improve the storage and management efficiency of massive KV pairs, and it can also support the efficient execution of KV applications. A prototype is implemented named MKVS (the matrix KV storage system based on NVM devices) to test with YCSB (Yahoo! Cloud System Benchmark) and to compare with the current in-memory KV store. The results show that MKVS can improve the throughput by 5.98 times, and reduce the 99.7% read latency and 77.2% write latency.

The artificial immune algorithm is the hot topic in much research such as the intrusion detection system, the information retrieval system and the data mining system. The negative selection algorithm is the typical artificial immune algorithm. A common representation of binary strings for antibody (detector) and antigen have been associated with inefficiencies when generating detector and inspecting antigen. We use a single integer to represent the detector and provide the basis of improving negative selection algorithm efficiency. In the detector generation algorithm, extracting sub-strings in self that its length is larger than threshold and converting them to single integer in numerical interval, then the rest integers in numerical interval are selected as numerical detectors. It can reduce the time and space overhead of detector generation and provide the facility to analyze the positive and negative errors when antigen inspection. The numerical matching rule is given. The B-tree is used to create index of numerical detector. Extracting sub-strings in antigen that its length is larger than threshold and converting them to some integers. If there is the same value as those integers in the index of numerical detector, then the antigen matches one numerical detector. It can improve the efficiency of antigen inspection. Finally the prototype of the numerical negative selection algorithm and negative selection algorithm are realized to test the overhead of the detector generation and antigen inspection using the live data set. The results show that the numerical negative selection algorithm can reduce the time and space overhead and avoid fluctuation of the overhead.

The K-nearest neighbour classifier is very effective and simple non-parametric technique in pattern classification, however, it only considers the distance closeness, but not the geometricalplacement of the k neighbors. Also, its classification performance is highly influenced by the neighborhood size k and existing outliers. In this paper, we propose a new local mean based k-harmonic nearest centroid neighbor (LMKHNCN) classifier in orderto consider both distance-based proximity, as well as spatial distribution of k neighbors. In our method, firstly the k nearest centroid neighbors in each class are found which are used to find k different local mean vectors, and then employed to compute their harmonic mean distance to the query sample. Lastly, the query sample is assigned to the class with minimum harmonic mean distance. The experimental results based on twenty-six real-world datasets shows that the proposed LMKHNCN classifier achieves lower error rates, particularly in small sample-size situations, and that it is less sensitive to parameter k when compared to therelated four KNN-based classifiers.