Search

Long-term dependence is an essential feature for the predictability of time series. Estimating the parameter that describes long memory is essential to describing the behavior of time series models. However, most long memory estimation methods assume that this parameter has a constant value throughout the time series, and do not consider that the parameter may change over time. In this work, we propose an automated methodology that combines the estimation methodologies of the fractional differentiation parameter (and/or Hurst parameter) with its application to Recurrent Neural Networks (RNNs) in order for said networks to learn and predict long memory dependencies from information obtained in nonlinear time series. The proposal combines three methods that allow for better approximation in the prediction of the values of the parameters for each one of the windows obtained, using Recurrent Neural Networks as an adaptive method to learn and predict the dependencies of long memory in Time Series. For the RNNs, we have evaluated four different architectures: the Simple RNN, LSTM, the BiLSTM, and the GRU. These models are built from blocks with gates controlling the cell state and memory. We have evaluated the proposed approach using both synthetic and real-world data sets. We have simulated ARFIMA models for the synthetic data to generate several time series by varying the fractional differentiation parameter. We have evaluated the proposed approach using synthetic and real datasets using Whittle’s estimates of the Hurst parameter classically obtained in each window. We have simulated ARFIMA models in such a way that the synthetic data generate several time series by varying the fractional differentiation parameter. The real-world IPSA stock option index and Tree Ringtime series datasets were evaluated. All of the results show that the proposed approach can predict the Hurst exponent with good performance by selecting the optimal window size and overlap change.

Air quality forecasting is of great importance in environmental protection, government decision-making, people's daily health, etc. Existing research methods have failed to effectively modeling long-term and complex relationships in time series PM2.5 data and exhibited low precision in long-term prediction. To address this issue, in this paper a new lightweight deep learning model using sparse attention-based Transformer networks (STN) consisting of encoder and decoder layers, in which a multi-head sparse attention mechanism is adopted to reduce the time complexity, is proposed to learn long-term dependencies and complex relationships from time series PM2.5 data for modeling air quality forecasting. Extensive experiments on two real-world datasets in China, i.e., Beijing PM2.5 dataset and Taizhou PM2.5 dataset, show that our proposed method not only has relatively small time complexity, but also outperforms state-of-the-art methods, demonstrating the effectiveness of the proposed STN method on both short-term and long-term air quality prediction tasks. In particular, on singe-step PM2.5 forecasting tasks our proposed method achieves R2 of 0.937 and reduces RMSE to 19.04 µg/m3 and MAE to 11.13 µg/m3 on Beijing PM2.5 dataset. Also, our proposed method obtains R2 of 0.924 and reduces RMSE to 5.79 µg/m3 and MAE to 3.76 µg/m3 on Taizhou PM2.5 dataset. For long-term time step prediction, our proposed method still performs best among all used methods on multi-step PM2.5 forecasting results for the next 6, 12, 24, and 48 h on two real-world datasets. [ABSTRACT FROM AUTHOR]

Long-term dependence is an essential feature for the predictability of time series. Estimating the parameter that describes long memory is essential to describing the behavior of time series models. However, most long memory estimation methods assume that this parameter has a constant value throughout the time series, and do not consider that the parameter may change over time. In this work, we propose an automated methodology that combines the estimation methodologies of the fractional differentiation parameter (and/or Hurst parameter) with its application to Recurrent Neural Networks (RNNs) in order for said networks to learn and predict long memory dependencies from information obtained in nonlinear time series. The proposal combines three methods that allow for better approximation in the prediction of the values of the parameters for each one of the windows obtained, using Recurrent Neural Networks as an adaptive method to learn and predict the dependencies of long memory in Time Series. For the RNNs, we have evaluated four different architectures: the Simple RNN, LSTM, the BiLSTM, and the GRU. These models are built from blocks with gates controlling the cell state and memory. We have evaluated the proposed approach using both synthetic and real-world data sets. We have simulated ARFIMA models for the synthetic data to generate several time series by varying the fractional differentiation parameter. We have evaluated the proposed approach using synthetic and real datasets using Whittle's estimates of the Hurst parameter classically obtained in each window. We have simulated ARFIMA models in such a way that the synthetic data generate several time series by varying the fractional differentiation parameter. The real-world IPSA stock option index and Tree Ringtime series datasets were evaluated. All of the results show that the proposed approach can predict the Hurst exponent with good performance by selecting the optimal window size and overlap change. [ABSTRACT FROM AUTHOR]

Driven by the requirements for a comprehensive understanding of composite dynamic systems in industrial processes, this paper investigates a new soft sensor for quality prediction based on slow and fast time-varying latent variables extraction using layer-wise residuals. First, the slow feature partial least squares were expanded into long-term dependency by introducing explicit expressions of the potential state of the process into the objective function. Then, the multilayer regression model for exploring composite dynamics driven by layer-wise residuals is developed using a serial structure that can extract both slow and fast time-varying latent variables that are completely orthogonal. Finally, the exponential-weighted partial least squares are proposed for extracting fast time-varying dynamic latent variables by learning the exponential decay properties of the time-series data correlation. Case studies on the industrial debutanizer and sulfur recovery unit show that the prediction accuracy of the proposed approach outperforms traditional methods. • The composite dynamics with slow time-varying overall trends accompanied by fast time-varying fluctuations can be described. • The slow time-varying quality-related LVs based on the assumption of long-term dependency are extracted for quality prediction. • The EWMA was integrated into the dynamic inner model to extract fast time-varying dynamic LVs with weak autocorrelation. [ABSTRACT FROM AUTHOR]

Deep summarization models have succeeded in the video summarization field based on the development of gated recursive unit (GRU) and long and short-term memory (LSTM) technology. However, for some long videos, GRU and LSTM cannot effectively capture long-term dependencies. This paper proposes a deep summarization network with auxiliary summarization losses to address this problem. We introduce an unsupervised auxiliary summarization loss module with LSTM and a swish activation function to capture the long-term dependencies for video summarization, which can be easily integrated with various networks. The proposed model is an unsupervised framework for deep reinforcement learning that does not depend on any labels or user interactions. Additionally, we implement a reward function ( R (S) ) that jointly considers the consistency, diversity, and representativeness of generated summaries. Furthermore, the proposed model is lightweight and can be successfully deployed on mobile devices and enhance the experience of mobile users and reduce pressure on server operations. We conducted experiments on two benchmark datasets and the results demonstrate that our proposed unsupervised approach can obtain better summaries than existing video summarization methods. Furthermore, the proposed algorithm can generate higher F scores with a nearly 6.3% increase on the SumMe dataset and a 2.2% increase on the TVSum dataset compared to the DR-DSN model. [ABSTRACT FROM AUTHOR]

Air quality PM2.5 prediction is an effective approach for providing early warning of air pollution. This paper proposes a new deep learning model called temporal difference-based graph transformer networks (TDGTN) to learn long-term temporal dependencies and complex relationships from time series PM2.5 data for air quality PM2.5 prediction. The proposed TDGTN comprises of encoder and decoder layers associated with the developed graph attention mechanism. In particular, considering the similarity of different time moments and the importance of temporal difference between two adjacent moments for air quality PM2.5prediction, we first construct graph-structured data from original time series PM2.5 data at different moments without explicit graph structure. Then we improve the self-attention mechanism with the temporal difference information, and develop a new graph attention mechanism. Finally, the developed graph attention mechanism is embedded into the encoder and decoder layers of the proposed TDGTN to learn long-term temporal dependencies and complex relationships from a graph prospective on air quality PM2.5 prediction tasks. Experiment results on two collected real-world datasets in China, such as Beijing and Taizhou PM2.5 datasets, show that the proposed method outperforms other used methods on both short-term and long-term air quality PM2.5 prediction tasks.

Recent studies have shown that it is necessary to further improve the prediction accuracy of complex and dynamic traffic flow from two angles: temporal dependence and spatial dependence. Although many spatio-temporal networks are extracting features from both temporal and spatial dependencies, there are still two problems that need to be addressed. Firstly, existing methods often focus on modeling local spatial dependence relationship between adjacent nodes, but neglect non-local spatial dependence relationship between distant nodes. Secondly, while recent work has explored various methods such as convolutional and recurrent neural networks for modeling temporal dependence, they may not fully capture the long-term temporal dependence in traffic data. Therefore, addressing the challenges of long-term dependency, explicit spatial dependency, and implicit spatial dependency, this paper proposes a Long-term Explicit–Implicit Spatio-Temporal Network (LEISN) for traffic flow prediction. A Long-term Dependency Module has been designed to store hidden states generated from multiple previous time steps, facilitating the transmission of long-term features. Based on this, two graph convolution-based spatial feature extraction branches are designed to extract explicit spatial features based on the adjacency matrix generated by spatial topology and implicit spatial features based on the implicit adjacency matrix generated by trend similarity, respectively. Then all spatio-temporal features are fused to produce the next state. Additionally, a new encoder framework, LENSI-ED, was proposed based on LEISN. Comparative experiments are conducted on four datasets, and the results showed that our model has advantages over existing methods. The proposed model addresses the issues of local and non-local spatial dependence relationships and long-term temporal dependence in traffic flow forecasting. • A long explicit–implicit spatio-temporal network for traffic flow prediction is proposed. • A new encoder framework named LENSI-ED is proposed based on LEISN. • Comparative experiments are conducted to show the advantages of the neural network. [ABSTRACT FROM AUTHOR]

Most existing unsupervised approaches to detect topic sentiment in social texts consider only the text sequences in corpus and put aside social dynamics, as leads to algorithm’s disability to discover true sentiment of social users. To address the issue, a probabilistic graphical model LDTSM (Long-term Dependence Topic-Sentiment Mixture) is proposed, which introduces dependency distance and uses the dynamics of social media to achieve the perfect combination of inheriting historical topic sentiment and fitting topic sentiment distribution underlying in current social texts. Extensive experiments on real-world SinaWeibo datasets show that LDTSM significantly outperforms JST, TUS-LDA and dNJST in terms of sentiment classification accuracy, with better inference convergence, and topic and sentiment evolution analysis results demonstrate that our approach is promising.

Accurate and real-time perception of the operating status of rolling bearings, which constitute a key component of rotating machinery, is of vital significance. However, most existing solutions not only require substantial expertise to conduct feature engineering, but also seldom consider the temporal correlation of sensor sequences, ultimately leading to complex modeling processes. Therefore, we present a novel model, named Attention-based Equitable Segmentation Gated Recurrent Unit Networks (AESGRU), to improve diagnostic accuracy and model-building efficiency. Specifically, our proposed AESGRU consists of two modules, an equitable segmentation approach and an improved deep model. We first transform the original dataset into time-series segments with temporal correlation, so that the model enables end-to-end learning from the strongly correlated data. Then, we deploy a single-layer bidirectional GRU network, which is enhanced by attention mechanism, to capture the long-term dependency of sensor segments and focus limited attention resources on those informative sampling points. Finally, our experimental results show that the proposed approach outperforms previous approaches in terms of the accuracy.

Recognition of 3D shapes is a fundamental task in computer vision. In recent years, view-based deep learning has emerged as an effective approach for 3D shape recognition. Most existing view-based methods treat the views of an object as an unordered set, which ignores the dynamic relations among the views, e.g. sequential semantic dependencies. In this paper, modeling the views of an object by a sequence, we aim at exploiting the long-term dependencies among different views for shape recognition, which is done by constructing a sequence-aware view aggregation module based on the bi-directional Long Short-Term Memory network. It is shown that our view aggregation module not only captures the bi-directional dependencies in view sequences, but also enjoys the robustness to circular shifts of input sequences. Incorporating the aggregation module into a standard convolutional network architecture, we develop an effective method for 3D shape classification and retrieval. Our method was evaluated on the ModelNet40/10 and ShapeNetCore55 datasets. The results show the encouraging performance gain from exploiting long-term dependencies in view sequences, as well as the superior performance of our method compared to the existing ones.

Exemplar-based video colorization is an essential technique for applications like old movie restoration. Although recent methods perform well in still scenes or scenes with regular movement, they always lack robustness in moving scenes due to their weak ability to model long-term dependency both spatially and temporally, leading to color fading, color discontinuity, or other artifacts. To solve this problem, we propose an exemplar-based video colorization framework with long-term spatiotemporal dependency. To enhance the long-term spatial dependency, a parallelized CNN-Transformer block and a double-head non-local operation are designed. The proposed CNN-Transformer block can better incorporate the long-term spatial dependency with local texture and structural features, and the double-head non-local operation further exploits the performance of the augmented feature. While for the long-term temporal dependency enhancement, we further introduce the novel Linkage subnet. The Linkage subnet propagates motion information across adjacent frame blocks and helps to maintain temporal continuity. Experiments demonstrate that our model outperforms recent state-of-the-art methods both quantitatively and qualitatively. Also, our model can generate more colorful, realistic and stabilized results, especially for scenes where objects change greatly and irregularly. [ABSTRACT FROM AUTHOR]

Long term dependency capture is essentially important for time series prediction and spatial–temporal forecasting. In recent years, many deep learning-based forecasting methods have been proposed, leading to rapid development in this area. We systematically reviewed long-term dependency capture methods, including temporal dependency in sequence (named intra-sequence temporal dependency), temporal dependency out of sequence (named inter-sequence temporal dependency, in this scenario the long-term dependencies are split by many subsequences). Because the batch technique is widely adopted in machine learning and deep learning, the range of temporal capturing ability for many proposed methods is intra-sequence temporal dependency, which limits the capacity of long-term dependency capture. Aiming at the above problems, we designed three type memory mechanisms (i.e., a temporal encoding memory mechanism, a cross-sequence memory mechanism and a query-key based memory mechanism) to solve those long term dependency problems. Moreover, based on the cross-sequence memory mechanism and query-key architecture, an Attention-based Long-Term Dependency Capture model (ALTDC) is proposed for long-term dependency modeling and further solves the temporal dependency coherence problem. ALTDC includes temporal Transformer and spatial Transformer. The temporal Transformer adopts multi-head attention mechanism in temporal dimension and takes into consideration the relative position encoding. The spatial Transformer leverages attention mechanism in spatial dimension, and utilizes learnable position encoding and graph convolution to capture spatial relationships. Experiments demonstrate that the proposed model outperforms the state-of-art baselines on real world time-series datasets and spatial–temporal datasets. [ABSTRACT FROM AUTHOR]