A fully online approach for anomaly detection and change-point detection in streaming data using LSTM

In this thesis, we propose a novel online anomaly detection and change detection algorithm based on contemporary Recurrent Neural Networks such as Long Short-Term Memory (LSTM) for time series anomaly detection in a changing environment posed by either transitory change (point anomaly) or permanent change (change-point) in normal behaviour in streaming data. The proposed online anomaly and change-point detection model is trained incrementally as new data stream becomes available and is capable of adapting to the changes in the data distribution of underlying pattern. LSTM is used to make single or multi-step predictions of the time series which could be from 1 step up to 10 steps ahead, and the prediction errors are used to detect anomalies and change-points as well as update the model. Fundamentally, the proposed algorithm is developing the online model of prediction error such that the large prediction error is employed to indicate the anomalous behaviour in changing environment. Additionally, the prediction errors are used to update the proposed online model in such a way that transitory anomalies do not lead to a radical change in the model. Whereas high computed prediction errors over a period of time due to permanent changes/change-points lead to substantial updates in model. The model automatically and swiftly adapts to the changing statistics and new custom of input data distribution. The proposed online anomaly detection and change-point detection technique is striving for fully on-line performance; therefore, model will not assume any labels in data stream except during evaluation. Furthermore, our novel proposed online anomaly and change detection model is not relying on any user define parameters (such as threshold) but automatically defined and updated by model itself. We validate the efficiency of the proposed novel model-based parametric unsupervised online approach through experiments on publicly accessible and proprietary three real-world and four synthetic benchmark datasets, which are taken and derived from Yahoo Labs Benchmark Dataset (Yahoo Webscope) and Numenta Anomaly Benchmark (NAB) that contains labelled anomalies. We compare the results in term of Area Under Curve (AUC) with state of-the-art algorithm available in literature. We perceive that proposed online anomaly and change-point detection model perform reliably better for multistep predictions as compared to models with single step predictions for all data sets in terms of AUC. We also observe that model trained for predictions of multi-step such as 5 or 10 steps ahead are more competent to detect the temporal changes in the target output distribution and are consequently able to detect changes point to and adapt to the changes much early as compared to model trained to predict only with one-step. Where step is the number of future predictions of the model. Unlike other methods our method has the advantage that it not only detects the anomalies as quickly as possible but did not let the anomalies to make drastic change in distribution. Whereas in case of short-term changes/point anomalies the proposed online model suggests that there is a trade-off between how early the anomaly detection happened and how long do the false positives last is depending on prediction length. It is concluded that our proposed online anomaly and change-point detection model consistently outperform and give the full advantage when it utilizes for multistep time series predictions.