Search

Nowadays, for controlling crime, surveillance cameras are typically installed in all public places to ensure urban safety and security. However, automating Human Activity Recognition (HAR) using computer vision techniques faces several challenges such as lowlighting, complex spatiotemporal features, clutter backgrounds, and inefficient utilization of surveillance system resources. Existing attempts in HAR designed straightforward networks by analyzing either spatial or motion patterns resulting in limited performance while the dual streams methods are entirely based on Convolutional Neural Networks (CNN) that are inadequate to learning the long-range temporal information for HAR. To overcome the above-mentioned challenges, this paper proposes an optimized dual stream framework for HAR which mainly consists of three steps. First, a shots segmentation module is introduced in the proposed framework to efficiently utilize the surveillance system resources by enhancing the lowlight video stream and then it detects salient video frames that consist of human. This module is trained on our own challenging Lowlight Human Surveillance Dataset (LHSD) which consists of both normal and different levels of lowlighting data to recognize humans in complex uncertain environments. Next, to learn HAR from both contextual and motion information, a dual stream approach is used in the feature extraction. In the first stream, it freezes the learned weights of the backbone Vision Transformer (ViT) B-16 model to select the discriminative contextual information. In the second stream, ViT features are then fused with the intermediate encoder layers of FlowNet2 model for optical flow to extract a robust motion feature vector. Finally, a two stream Parallel Bidirectional Long Short-Term Memory (PBiLSTM) is proposed for sequence learning to capture the global semantics of activities, followed by Dual Stream Multi-Head Attention (DSMHA) with a late fusion strategy to optimize the huge features vector for accurate HAR. To assess the strength of the proposed framework, extensive empirical results are conducted on real-world surveillance scenarios and various benchmark HAR datasets that achieve 78.6285%, 96.0151%, and 98.875% accuracies on HMDB51, UCF101, and YouTube Action, respectively. Our results show that the proposed strategy outperforms State-of-the-Art (SOTA) methods. The proposed framework gives superior performance in HAR, providing accurate and reliable recognition of human activities in surveillance systems.

Human Activity Recognition (HAR) plays a crucial role in communication and the Internet of Things (IoT), by enabling vision sensors to understand and respond to human behavior more intelligently and efficiently. Existing deep learning models are complex to deal with the low illumination, diverse viewpoints, and cluttered backgrounds, which require substantial computing resources and are not appropriate for edge devices. Furthermore, without an effective video analysis technique it processes entire frames, resulting inadequate performance. To address these key challenges, a cloud-assisted IoT computing framework is proposed for HAR in uncertain low-lighting environments, which is mainly composed of two tiers: edge and cloud computing. Initially, a lightweight Convolutional Neural Network (CNN) model is developed which is responsible to enhance the low-light frames, followed by the human detection algorithm to process the selective frames, thus enabling efficient resource utilization. Next, these refined frames are then transmitted to the cloud for accurate HAR, where dual stream CNN and transformer fusion network extract both short-and long-range spatiotemporal discriminative features followed by proposed Optimized Parallel Sequential Temporal Network (OPSTN) with squeeze and excitation attention to efficiently learn HAR in complex scenarios. Finally, extensive experiments are conducted over three challenging HAR datasets to deeply examine the proposed framework from various perspectives such as complex activity recognition, lowlighting, etc., where the results are outperformed compared with the state-of-art methods.

Accurate electricity consumption forecasting in residential buildings has a direct impact on energy efficiency and cost management, making it a critical component of sustainable energy practices. Decision tree-based ensemble learning techniques are particularly effective for this task due to their ability to process complex datasets with high accuracy. Furthermore, incorporating explainable artificial intelligence into these predictions provides clarity and interpretability, allowing energy managers and homeowners to make informed decisions that optimize usage and reduce costs. This study comparatively analyzes decision tree-ensemble learning techniques augmented with explainable artificial intelligence for transparency and interpretability in residential building energy consumption forecasting. This approach employs the University Residential Complex and Appliances Energy Prediction datasets, data preprocessing, and decision-tree bagging and boosting methods. The superior model is evaluated using the Shapley additive explanations method within the explainable artificial intelligence framework, explaining the influence of input variables and decision-making processes. The analysis reveals the significant influence of the temperature-humidity index and wind chill temperature on short-term load forecasting, transcending traditional parameters, such as temperature, humidity, and wind speed. The complete study and source code have been made available on our GitHub repository at https://github.com/sodayeong for the purpose of enhancing precision and interpretability in energy system management, thereby promoting transparency and enabling replication.

Renewable energy (RE) offers major environmental and economic benefits compared to nuclear and fuel-based energy; however, the data used for RE include significant randomness, intermittent behaviour, and strong-volatility, hindering their integration into smart grids. Accurate RE prediction is a promising solution to this problem and can provide effective planning and management services. Various predictive models have been developed to improve the prediction performance for better energy management. However, current works focus on improving the prediction accuracy, which is a requirement of power systems, without considering the time complexity of their methodologies. Considering these limitations, we develop a lightweight ESNCNN model for accurate RE prediction, in which an ESN learns the nonlinear mapping relationship and a CNN extracts the spatial information from RE data. The ESN and CNN layers are linearly connected via residual connections to avoid the vanishing gradient problem. Finally, we incorporate fully connected layers to enhance and select the optimal ESNCNN features to predict the future energy production. Our ESNCNN model is evaluated based on RE benchmarks, using various evaluation metrics such as MSE, MBE, MAE and RMSE, and achieves state-of-the-art performance. Further experiments are performed with different machine learning, deep learning, and hybrid models to select the optimum model. To fully assess the generalisation ability of the proposed ESNCNN, additional experiments are performed over electricity consumption datasets, which reveal an extensive decrease in error rates compared to other state-of-the-art approaches. Our model therefore represents a new paradigm for finding an energy equilibrium between grid and energy production resources using a single ESNCNN platform. Our results indicate a substantial reduction in the error rates over the RE dataset (MSE (5.01%), MAE (5.49%), and RMSE (3.76%)) and the electricity consumption dataset (MSE (5.37%), MAE (7.63%), RMSE (0.047%), and MBE (1.2%)).

The prognostics and health management (PHM) plays the main role to handle the risk of failure before its occurrence. Next, it has a broad spectrum of applications including utility networks, energy storage systems (ESS), etc. However, an accurate capacity estimation of batteries in ESS is mandatory for their safe operations and decision making policy. ESS comprises of different storage mechanisms such as batteries, capacitors, etc. Consequently, the measurement of different charging profiles (CPs) has a strong relation to battery capacity. These profiles include temperature (T), voltage (V), and current (I) where the CPs patterns vary as the battery ages with cycles. Consequently, estimating a battery capacity, the conventional methods practice single channel charging profile (SCCP) and hop multiple channel CPs (MCCPs) that cause incorrect battery health estimation. To tackle these issues, this article proposes MCCPs based battery management system (BMS) to estimate batteries health/capacity through the deep learning (DL) concept where the patterns in these CPs are changed as the battery ages with time and cycles. Thus, we deeply investigate both machine learning (ML) and DL based methods to provide a concrete comparative analysis of our method. The adaptive boosting (AB) and support vector regression (SVR) are widely compared with long short-term memory (LSTM), multi-layer perceptron (MLP), bi-directional LSTM (BiLSTM), and convolutional neural network (CNN) to attain the appropriate approach for battery capacity and state of health (SOH) estimation. These approaches have a high learning capability of inter-relation between the battery capacity and variation in CPs patterns. To validate and verify the proposed technique, we use NASA battery dataset and experimentally prove that BiLSTM outperforms all the approaches and obtains the smallest error values for MAE, MSE, RMSE, and MAPE using MCCPs compared to SCCP.

The high-level variation of different energy generation resources makes the reliable power supply significantly challenging to end-users. These variations occur due to the intermittent nature of energy output and time-varying weather conditions. The recent literature focuses on the improvements in power generation and consumption forecasting, which is a demand of the current smart grids’ smooth operations with a balanced amount of energy generation and consumption for the connected customers. Inspired by the applications of load forecasting, therefore, in this work, we develop an efficient and effective hybrid model for power generation and consumption forecasting, thereby contributing to energy harvesting by providing valuable prediction data to the concerned renewable energy analysts. Herein, we integrate a convolutional neural network with an echo state network for robust renewable energy generation and consumption forecasting. The convolutional network is used to extract meaningful patterns from the historical data which is then forwarded to the echo state network for temporal features learning. The output spatiotemporal feature vector is then fed to fully connected layers for final forecasting. The proposed hybrid model is derived after extensive experiments over machine and deep learning models, where the results indicate that the proposed model substantially decreases the forecasting errors using RMSE, MSE, NRMSE, and MAE metrics, when compared to state-of-the-art models and acts as a paradigm towards energy equilibrium between production resources and consumers.

For efficient energy distribution, microgrids (MG) provide significant assistance to main grids and act as a bridge between the power generation and consumption. Renewable energy generation resources, particularly photovoltaics (PVs), are considered as a clean source of energy but are highly complex, volatile, and intermittent in nature making their forecasting challenging. Thus, a reliable, optimized, and a robust forecasting method deployed at MG objectifies these challenges by providing accurate renewable energy production forecasting and establishing a precise power generation and consumption matching at MG. Furthermore, it ensures effective planning, operation, and acquisition from the main grid in the case of superior or inferior amounts of energy, respectively. Therefore, in this work, we develop an end-to-end hybrid network for automatic PV power forecasting, comprising three basic steps. Firstly, data preprocessing is performed to normalize, remove the outliers, and deal with the missing values prominently. Next, the temporal features are extracted using deep sequential modelling schemes, followed by the extraction of spatial features via convolutional neural networks. These features are then fed to fully connected layers for optimal PV power forecasting. In the third step, the proposed model is evaluated on publicly available PV power generation datasets, where its performance reveals lower error rates when compared to state-of-the-art methods.

Over the decades, a rapid upsurge in electricity demand has been observed due to overpopulation and technological growth. The optimum production of energy is mandatory to preserve it and improve the energy infrastructure using the power load forecasting (PLF) method. However, the complex energy systems’ transition towards more robust and intelligent system will ensure its momentous role in the industrial and economical world. The extraction of deep knowledge from complex energy data patterns requires an efficient and computationally intelligent deep learning-based method to examine the future electricity demand. Stand by this, we propose an intelligent deep learning-based PLF method where at first the data collected from the house through meters are fed into the pre-assessment step. Next, the sequence of refined data is passed into a modified convolutional long short-term memory (ConvLSTM) network that captures the spatiotemporal correlations from the sequence and generates the feature maps. The generated feature map is forward propagated into a deep gated recurrent unit (GRU) network for learning, which provides the final PLF. We experimentally proved that the proposed method revealed promising results using mean square error (MSE) and root mean square error (RMSE) and outperformed state of the art using the competitive power load dataset.(Github Code). (Github code: https://github.com/FathUMinUllah3797/ConvLSTM-Deep_GRU).

The extensive video surveillance networks gather an enormous amount of data exponentially on a daily basis and its management is a challenging task, requiring efficient and effective techniques for searching, indexing, and retrieval. The employed mainstream techniques are focusing on general category videos, where the important events in surveillance require fine-grained events retrieval. In this paper, we introduce an event-oriented feature selection mechanism by utilizing the intermediate convolutional layer of a pre-trained 3D-CNN model, that is selected after deep investigation of its weights and response to a particular event. The extracted exclusive features represent an event semantically and effectively eliminate those neurons which do not respond to an event. Furthermore, the event-oriented convolutional features are of very high-dimensions, requiring additional storage, and take more time in feature comparison for retrieval. Therefore, we generate compact binary codes from these features using principle component analysis (PCA) algorithm. This makes our system more efficient to retrieve videos from large scale database. We evaluated our approach on the challenging events of UCF101 and HMDB51 datasets for original features and generated compact codes to achieve reduced execution time and better precision and recall scores.

Electric energy forecasting domain attracts researchers due to its key role in saving energy resources, where mainstream existing models are based on Gradient Boosting Regression (GBR), Artificial Neural Networks (ANNs), Extreme Learning Machine (ELM) and Support Vector Machine (SVM). These models encounter high-level of non-linearity between input data and output predictions and limited adoptability in real-world scenarios. Meanwhile, energy forecasting domain demands more robustness, higher prediction accuracy and generalization ability for real-world implementation. In this paper, we achieve the mentioned tasks by developing a hybrid sequential learning-based energy forecasting model that employs Convolution Neural Network (CNN) and Gated Recurrent Units (GRU) into a unified framework for accurate energy consumption prediction. The proposed framework has two major phases: (1) data refinement and (2) training, where the data refinement phase applies preprocessing strategies over raw data. In the training phase, CNN features are extracted from input dataset and fed in to GRU, that is selected as optimal and observed to have enhanced sequence learning abilities after extensive experiments. The proposed model is an effective alternative to the previous hybrid models in terms of computational complexity as well prediction accuracy, due to the representative features' extraction potentials of CNNs and effectual gated structure of multi-layered GRU. The experimental evaluation over existing energy forecasting datasets reveal the better performance of our method in terms of preciseness and efficiency. The proposed method achieved the smallest error rate on Appliances Energy Prediction (AEP) and Individual Household Electric Power Consumption (IHEPC) datasets, when compared to other baseline models.

Excessive Power Consumption (PC) and demand for power is increasing on a daily basis, due to advancements in technology, the rise in electricity-dependent machinery, and the growth of the human population. It has become necessary to predict PC in order to improve power management and co-operation between the energy used in a building and the power grid. State-of-the-art Energy Consumption Prediction (ECP) methods are limited in terms of predicting the energy effectively, due to various challenges such as weather conditions and the dynamic behaviour of occupants. Thus, to overcome the drawbacks of these methods, we present an intelligent hybrid technique that combines a Convolutional Neural Network (CNN) with a Multi-layer Bi-directional Long-short Term Memory (M-BDLSTM) method using three steps. When applied to short-term power ECP, this approach helps to provide efficient power management i.e. it can assist the supplier to produce the optimum amount of power. The first step in our proposed method integrates the pre-processing and data organisation mechanisms to refine the data and remove abnormalities. The second step employs a deep learning network, where the sequence of refined data is fed into the CNN via the M-BDLSTM network to learn the sequence pattern effectively. The third step generates the ECP/PC by comparing actual and predicted data series and evaluates the prediction using error metrics. The proposed method achieves better prediction results than existing techniques, thus demonstrating its effectiveness. Furthermore, it achieved the smallest value of the Mean Square Error (MSE) and Root Mean Square Error (RMSE) for individual household dataset using 10-fold Cross Validation (CV) and a hold-out (CV) method.

Recommender systems are intelligent data mining applications that deal with the issue of information overload significantly. The available literature discusses several methodologies to generate recommendations and proposes different techniques in accordance with users' needs. The majority of the work in the recommender system domain focuses on increasing the recommendation accuracy by employing several proposed approaches where the main motive remains to maximize the accuracy of recommendations while ignoring other design objectives, such as a user's an item's context. The biggest challenge for a recommender system is to produce meaningful recommendations by using contextual user-item rating information. A context is a vast term that may consider various aspects; for example, a user's social circle, time, mood, location, weather, company, day type, an item's genre, location, and language. Typically, the rating behavior of users varies under different contexts. From this line of research, we have proposed a new algorithm, namely Kernel Context Recommender System, which is a flexible, fast, and accurate kernel mapping framework that recognizes the importance of context and incorporates the contextual information using kernel trick while making predictions. We have benchmarked our proposed algorithm with pre- and post-filtering approaches as they have been the favorite approaches in the literature to solve the context-aware recommendation problem. Our experiments reveal that considering the contextual information can increase the performance of a system and provide better, relevant, and meaningful results on various evaluation metrics.

Recent advances in the film industry have given rise to exponential growth in movie/drama production and adaptation of the Big Data concept. Automatic identification and classification of movie characters have received tremendous attention from researchers due to its applications in video semantic analysis, video summarization, and personalized video retrieval for which several methods have been recently presented. However, these methods cannot detect main characters properly due to their variation in pose and style in different scenes of a movie. To address this problem we present DeepStar, a novel framework for starring character identification based on deep high-level robust features. The proposed framework is threefold: the extraction of shots with clear faces from the input video; face clustering using discriminative deep features; and the occurrence matrix generation, helping in the selection of starring characters. The promising results obtained using representative Hollywood movies demonstrate the effectiveness of our method in detecting starring characters over the state-of-the-art methods.

Abstract The huge amount of video data available these days requires effective management techniques for storage, indexing, and retrieval. Video summarization, a method to manage video data, provides concise versions of the videos for efficient browsing and retrieval. Key frame extraction is a form of video summarization which selects only the most salient frames from a given video. Since the automatic semantic understanding of the video contents is not possible so far, most of the existing works employ low level index features for extracting key frames. However, the usage of low level features results in loss of semantic details, thus leading to a semantic gap. In this context, the saliency-based user attention modeling technique can be used to bridge this semantic gap. In this paper, a key frame extraction scheme based on a visual attention mechanism is proposed. The proposed scheme builds static visual attention method based on multi-scale contrast instead of usual color contrast. The dynamic visual attention model is developed based on novel relative motion intensity and relative motion orientation. An efficient fusion scheme for combining three visual attention values is then proposed. A flexible technique is then used for key frame extraction. The experimental results demonstrate that the proposed mechanism provides excellent results as compared to the some of the other prominent techniques in the literature.

The recent advances in embedded processing have enabled the vision based systems to detect fire during surveillance using convolutional neural networks (CNNs). However, such methods generally need more computational time and memory, restricting its implementation in surveillance networks. In this research paper, we propose a cost-effective fire detection CNN architecture for surveillance videos. The model is inspired from GoogleNet architecture, considering its reasonable computational complexity and suitability for the intended problem compared to other computationally expensive networks such as AlexNet. To balance the efficiency and accuracy, the model is fine-tuned considering the nature of the target problem and fire data. Experimental results on benchmark fire datasets reveal the effectiveness of the proposed framework and validate its suitability for fire detection in CCTV surveillance systems compared to state-of-the-art methods.

Recurrent neural network (RNN) and long short-term memory (LSTM) have achieved great success in processing sequential multimedia data and yielded the state-of-the-art results in speech recognition, digital signal processing, video processing, and text data analysis. In this paper, we propose a novel action recognition method by processing the video data using convolutional neural network (CNN) and deep bidirectional LSTM (DB-LSTM) network. First, deep features are extracted from every sixth frame of the videos, which helps reduce the redundancy and complexity. Next, the sequential information among frame features is learnt using DB-LSTM network, where multiple layers are stacked together in both forward pass and backward pass of DB-LSTM to increase its depth. The proposed method is capable of learning long term sequences and can process lengthy videos by analyzing features for a certain time interval. Experimental results show significant improvements in action recognition using the proposed method on three benchmark data sets including UCF-101, YouTube 11 Actions, and HMDB51 compared with the state-of-the-art action recognition methods.

Renewable energy (RE) power plants are deployed globally because the renewable energy sources (RESs) are sustainable, clean, and environmentally friendly. However, the demand for power increases on a daily basis due to population growth, technology, marketing, and the number of installed industries. This challenge has raised a critical issue of how to intelligently match the power generation with the consumption for efficient energy management. To handle this issue, we propose a novel architecture called ‘AB-Net’: a one-step forecast of RE generation for short-term horizons by incorporating an autoencoder (AE) with bidirectional long short-term memory (BiLSTM). Firstly, the data acquisition step is applied, where the data are acquired from various RESs such as wind and solar. The second step performs deep preprocessing of the acquired data via several de-noising and cleansing filters to clean the data and normalize them prior to actual processing. Thirdly, an AE is employed to extract the discriminative features from the cleaned data sequence through its encoder part. BiLSTM is used to learn these features to provide a final forecast of power generation. The proposed AB-Net was evaluated using two publicly available benchmark datasets where the proposed method obtains state-of-the-art results in terms of the error metrics.

Nowadays, for efficient energy management, local demand-supply matching in power grid is emerging research domain. However, energy demand is increasing day by day in many countries due to rapid growth of the population and most of their work being reliant on electronic devices. This problem has highlighted the significance of effectively matching power demand with supply for optimal energy management. To resolve this issue, we present an intelligent deep learning framework that integrates Atrous Convolutional Layers (ACL) with Residual Gated Recurrent Units (RGRU) to establish balance between the demand and supply. Moreover, it accurately predicts short-term energy and delivers a systematic method of communication between consumers and energy distributors as well. To cope with the varying nature of electricity data, first data acquisition step is performed where data are collected from various sources such as smart meters and solar plants. In the second step a pre-processing method is applied on raw data to normalize and clean the data. Next, the refined data are passed to ACL for spatial feature extraction. Finally, a sequential learning model RGRU is used that learns from complicated patterns for the final output. The proposed model obtains the smallest values of Mean Square Error (MSE) including 0.1753, 0.0001, 0.0177 over IHEPC, KCB, and Solar datasets, respectively, which manifests better performance as compared to existing approaches.

Movie data has a prominent role in the exponential growth of multimedia data over the Internet, and its analysis has become a hot topic with computer vision. The initial step towards movie analysis is scene segmentation. In this article, we investigated this problem through a novel intelligent Convolutional Neural Network (CNN) based three folded framework. The first fold segments the input movie into shots, the second fold detects objects in the segmented shots and the third fold performs object-based shots matching for detecting scene boundaries. Texture and shape features are fused for shots segmentation, and each shot is represented by a set of detected objects acquired from a light-weight CNN model. Finally, we apply set theory with the sliding window–based approach to integrate the same shots to decide scene boundaries. The experimental evaluation indicates that our proposed approach outran the existing movie scene segmentation approaches.

Video anomaly recognition in smart cities is an important computer vision task that plays a vital role in smart surveillance and public safety but is challenging due to its diverse, complex, and infrequent occurrence in real-time surveillance environments. Various deep learning models use significant amounts of training data without generalization abilities and with huge time complexity. To overcome these problems, in the current work, we present an efficient light-weight convolutional neural network (CNN)-based anomaly recognition framework that is functional in a surveillance environment with reduced time complexity. We extract spatial CNN features from a series of video frames and feed them to the proposed residual attention-based long short-term memory (LSTM) network, which can precisely recognize anomalous activity in surveillance videos. The representative CNN features with the residual blocks concept in LSTM for sequence learning prove to be effective for anomaly detection and recognition, validating our model’s effective usage in smart cities video surveillance. Extensive experiments on the real-world benchmark UCF-Crime dataset validate the effectiveness of the proposed model within complex surveillance environments and demonstrate that our proposed model outperforms state-of-the-art models with a 1.77%, 0.76%, and 8.62% increase in accuracy on the UCF-Crime, UMN and Avenue datasets, respectively.

The use of electrical energy is directly proportional to the increase in global population, both concerning growing industrialization and rising residential demand. The need to achieve a balance between electrical energy production and consumption inspires researchers to develop forecasting models for optimal and economical energy use. Mostly, the residential and industrial sectors use metering sensors that only measure the consumed energy but are unable to manage electricity. In this paper, we present a comparative analysis of a variety of deep features with several sequential learning models to select the optimized hybrid architecture for energy consumption prediction. The best results are achieved using convolutional long short-term memory (ConvLSTM) integrated with bidirectional long short-term memory (BiLSTM). The ConvLSTM initially extracts features from the input data to produce encoded sequences that are decoded by BiLSTM and then proceeds with a final dense layer for energy consumption prediction. The overall framework consists of preprocessing raw data, extracting features, training the sequential model, and then evaluating it. The proposed energy consumption prediction model outperforms existing models over publicly available datasets, including Household and Korean commercial building datasets.

In this article, we present an in-depth comparative analysis of the conventional and sequential learning algorithms for electricity load forecasting and optimally select the most appropriate algorithm for energy consumption prediction (ECP). ECP reduces the misusage and wastage of energy using mathematical modeling and supervised learning algorithms. However, the existing ECP research lacks comparative analysis of various algorithms to reach the optimal model with real-world implementation potentials and convincingly reduced error rates. Furthermore, these methods are less friendly towards the energy management chain between the smart grids and residential buildings, with limited contributions in saving energy resources and maintaining an appropriate equilibrium between energy producers and consumers. Considering these limitations, we dive deep into load forecasting methods, analyze their performance, and finally, present a novel three-tier framework for ECP. The first tier applies data preprocessing for its refinement and organization, prior to the actual training, facilitating its effective output generation. The second tier is the learning process, employing ensemble learning algorithms (ELAs) and sequential learning techniques to train over energy consumption data. In the third tier, we obtain the final ECP model and evaluate our method; we visualize the data for energy data analysts. We experimentally prove that deep sequential learning models are dominant over mathematical modeling techniques and its several invariants by utilizing available residential electricity consumption data to reach an optimal proposed model with smallest mean square error (MSE) of value 0.1661 and root mean square error (RMSE) of value 0.4075 against the recent rivals.

Smart cities are a future reality for municipalities around the world. Healthcare services play a vital role in the transformation of traditional cities into smart cities. In this paper, we present a ubiquitous and quality computer-aided blood analysis service for the detection and counting of white blood cells (WBCs) in blood samples. WBCs also called leukocytes or leucocytes are the cells of the immune system that are involved in protecting the body against both infectious disease and foreign invaders. Analysis of leukocytes provides valuable information to medical specialists, helping them in diagnosing different important hematic diseases, such as AIDS and blood cancer (Leukaemia). However, this task is prone to errors and can be time-consuming. A mobile-cloud-assisted detection and classification of leukocytes from blood smear images can enhance accuracy and speed up the detection of WBCs. In this paper, we propose a smartphone-based cloud-assisted resource aware framework for localization of WBCs within microscopic blood smear images using a trained multi-class ensemble classification mechanism in the cloud. In the proposed framework, nucleus is first segmented, followed by extraction of texture, statistical, and wavelet features. Finally, the detected WBCs are categorized into five classes: basophil, eosinophil, neutrophil, lymphocyte, and monocyte. Experimental results on numerous benchmark databases validate the effectiveness and efficiency of the proposed system in comparison to the other state-of-the-art schemes.

Smart grid technology based on renewable energy and energy storage systems are attracting considerable attention towards energy crises. Accurate and reliable model for electricity prediction is considered a key factor for a suitable energy management policy. Currently, electricity consumption is rapidly increasing due to the rise in human population and technology development. Therefore, in this study, we established a two-step methodology for residential building load prediction, which comprises two stages: in the first stage, the raw data of electricity consumption are refined for effective training; and the second step includes a hybrid model with the integration of convolutional neural network (CNN) and multilayer bidirectional gated recurrent unit (MB-GRU). The CNN layers are incorporated into the model as a feature extractor, while MB-GRU learns the sequences between electricity consumption data. The proposed model is evaluated using the root mean square error (RMSE), mean square error (MSE), and mean absolute error (MAE) metrics. Finally, our model is assessed over benchmark datasets that exhibited an extensive drop in the error rate in comparison to other techniques. The results indicated that the proposed model reduced errors over the individual household electricity consumption prediction (IHEPC) dataset (i.e., RMSE (5%), MSE (4%), and MAE (4%)), and for the appliances load prediction (AEP) dataset (i.e., RMSE (2%), and MAE (1%)).

In the current technological era, energy-efficient buildings have a significant research body due to increasing concerns about energy consumption and its environmental impact. Designing an appropriate energy-efficient building depends on its layout, such as relative compactness, overall area, height, orientation, and distribution of the glazing area. These factors directly influence the cooling load (CL) and heating load (HL) of residential buildings. An accurate prediction of these load facilitates a better management of energy consumption and enhances the living standards of inhabitants. Most of the traditional machine learning (ML)-based approaches are designed for single-output (SO) prediction, which is a tedious task due to separate training processes for each output with low performance. In addition, these approaches have a high level of nonlinearity between input and output, which need more enhancement in terms of robustness, predictability, and generalization. To tackle these issues, we propose a novel framework based on gated recurrent unit (GRU) that reliably predicts the CL and HL concurrently. To the best of our knowledge, we are the first to propose a multi-output (MO) sequential learning model followed by utility preprocessing under the umbrella of a unified framework. A comprehensive set of ablation studies on ML and deep learning (DL) techniques is done over an energy efficiency dataset, where the proposed model reveals an incredible performance as compared to other existing models.

Electric energy consumption forecasting is an interesting, challenging, and important issue in energy management and equipment efficiency improvement. Existing approaches are predictive models that have the ability to predict for a specific profile, i.e., a time series of a whole building or an individual household in a smart building. In practice, there are many profiles in each smart building, which leads to time-consuming and expensive system resources. Therefore, this study develops a robust framework for the Multiple Electric Energy Consumption forecasting (MEC) of a smart building using Transfer Learning and Long Short-Term Memory (TLL), the so-called MEC-TLL framework. In this framework, we first employ a k-means clustering algorithm to cluster the daily load demand of many profiles in the training set. In this phase, we also perform Silhouette analysis to specify the optimal number of clusters for the experimental datasets. Next, this study develops the MEC training algorithm, which utilizes a cluster-based strategy for transfer learning the Long Short-Term Memory models to reduce the computational time. Finally, extensive experiments are conducted to compare the computational time and different performance metrics for multiple electric energy consumption forecasting on two smart buildings in South Korea. The experimental results indicate that our proposed approach is capable of economical overheads while achieving superior performances. Therefore, the proposed approach can be applied effectively for intelligent energy management in smart buildings.

Personalized movie summarization is demand of the current era due to an exponential growth in movies production. The employed methods for movies summarization fail to satisfy the user’s requirements due to the subjective nature of movies data. Therefore, in this paper, we present a user-preference based movie summarization scheme. First, we segmented movie into shots using a novel entropy-based shots segmentation mechanism. Next, temporal saliency of shots is computed, resulting in highly salient shots in which character faces are detected. The resultant shots are then forward propagated to our trained deep CNN model for facial expression recognition (FER) to analyze the emotional state of the characters. The final summary is generated based on user-preferred emotional moments from the seven emotions, i.e., afraid, angry, disgust, happy, neutral, sad, and surprise. The subjective evaluation over five Hollywood movies proves the effectiveness of our proposed scheme in terms of user satisfaction. Furthermore, the objective evaluation verifies the superiority of the proposed scheme over state-of-the-art movie summarization methods.

The diagnosis of bankruptcy companies becomes extremely important for business owners, banks, governments, securities investors, and economic stakeholders to optimize the profitability as well as to minimize risks of investments. Many studies have been developed for bankruptcy prediction utilizing different machine learning approaches on various datasets around the world. Due to the class imbalance problem occurring in the bankruptcy datasets, several special techniques would be used to improve the prediction performance. Oversampling technique and cost-sensitive learning framework are two common methods for dealing with class imbalance problem. Using oversampling techniques and cost-sensitive learning framework independently also improves predictability. However, for datasets with very small balancing ratios, combining two above techniques will produce the better results. Therefore, this study develops a hybrid approach using oversampling technique and cost-sensitive learning, namely, HAOC for bankruptcy prediction on the Korean Bankruptcy dataset. The first module of HAOC is oversampling module with an optimal balancing ratio found in the first experiment that will give the best overall performance for the validation set. Then, the second module uses the cost-sensitive learning model, namely, CBoost algorithm to bankruptcy prediction. The experimental results show that HAOC will give the best performance value for bankruptcy prediction compared with the existing approaches.

For efficient and effective energy management, accurate energy consumption forecasting is required in energy management systems (EMSs). Recently, several artificial intelligence-based techniques have been proposed for accurate electric load forecasting; moreover, perfect energy consumption data are critical for the prediction. However, owing to diverse reasons, such as device malfunctions and signal transmission errors, missing data are frequently observed in the actual data. Previously, many imputation methods have been proposed to compensate for missing values; however, these methods have achieved limited success in imputing electric energy consumption data because the period of data missing is long and the dependency on historical data is high. In this study, we propose a novel missing-value imputation scheme for electricity consumption data. The proposed scheme uses a bagging ensemble of multilayer perceptrons (MLPs), called softmax ensemble network, wherein the ensemble weight of each MLP is determined by a softmax function. This ensemble network learns electric energy consumption data with explanatory variables and imputes missing values in this data. To evaluate the performance of our scheme, we performed diverse experiments on real electric energy consumption data and confirmed that the proposed scheme can deliver superior performance compared to other imputation methods.

Due to industrialization and the rising demand for energy, global energy consumption has been rapidly increasing. Recent studies show that the biggest portion of energy is consumed in residential buildings, i.e., in European Union countries up to 40% of the total energy is consumed by households. Most residential buildings and industrial zones are equipped with smart sensors such as metering electric sensors, that are inadequately utilized for better energy management. In this paper, we develop a hybrid convolutional neural network (CNN) with an long short-term memory autoencoder (LSTM-AE) model for future energy prediction in residential and commercial buildings. The central focus of this research work is to utilize the smart meters’ data for energy forecasting in order to enable appropriate energy management in buildings. We performed extensive research using several deep learning-based forecasting models and proposed an optimal hybrid CNN with the LSTM-AE model. To the best of our knowledge, we are the first to incorporate the aforementioned models under the umbrella of a unified framework with some utility preprocessing. Initially, the CNN model extracts features from the input data, which are then fed to the LSTM-encoder to generate encoded sequences. The encoded sequences are decoded by another following LSTM-decoder to advance it to the final dense layer for energy prediction. The experimental results using different evaluation metrics show that the proposed hybrid model works well. Also, it records the smallest value for mean square error (MSE), mean absolute error (MAE), root mean square error (RMSE) and mean absolute percentage error (MAPE) when compared to other state-of-the-art forecasting methods over the UCI residential building dataset. Furthermore, we conducted experiments on Korean commercial building data and the results indicate that our proposed hybrid model is a worthy contribution to energy forecasting.

The exponential growth in population and their overall reliance on the usage of electrical and electronic devices have increased the demand for energy production. It needs precise energy management systems that can forecast the usage of the consumers for future policymaking. Embedded smart sensors attached to electricity meters and home appliances enable power suppliers to effectively analyze the energy usage to generate and distribute electricity into residential areas based on their level of energy consumption. Therefore, this paper proposes a clustering-based analysis of energy consumption to categorize the consumers’ electricity usage into different levels. First, a deep autoencoder that transfers the low-dimensional energy consumption data to high-level representations was trained. Second, the high-level representations were fed into an adaptive self-organizing map (SOM) clustering algorithm. Afterward, the levels of electricity energy consumption were established by conducting the statistical analysis on the obtained clustered data. Finally, the results were visualized in graphs and calendar views, and the predicted levels of energy consumption were plotted over the city map, providing a compact overview to the providers for energy utilization analysis.

Smart grid systems, which have gained much attention due to its ability to reduce operation and management costs of power systems, consist of diverse components including energy storage, renewable energy, and combined cooling, heating and power (CCHP) systems. The CCHP has been investigated to reduce energy costs by using the thermal energy generated during the power generation process. For efficient utilization of CCHP and numerous power generation systems, accurate short-term load forecasting (STLF) is necessary. So far, even though many single algorithm-based STLF models have been proposed, they showed limited success in terms of applicability and coverage. This problem can be alleviated by combining such single algorithm-based models in ways that take advantage of their strengths. In this paper, we propose a novel two-stage STLF scheme; extreme gradient boosting and random forest models are executed in the first stage, and deep neural networks are executed in the second stage to combine them. To show the effectiveness of our proposed scheme, we compare our model with other popular single algorithm-based forecasting models and then show how much electric charges can be saved by operating CCHP based on the schedules made by the economic analysis on the predicted electric loads.

Wireless capsule endoscopy (WCE) has great advantages over traditional endoscopy because it is portable and easy to use, especially in remote monitoring health-services. However, during the WCE process, the large amount of captured video data demands a significant deal of computation to analyze and retrieve informative video frames. In order to facilitate efficient WCE data collection and browsing task, we present a resource- and bandwidth-aware WCE video summarization framework that extracts the representative keyframes of the WCE video contents by removing redundant and non-informative frames. For redundancy elimination, we use Jeffrey-divergence between color histograms and inter-frame Boolean series-based correlation of color channels. To remove non-informative frames, multi-fractal texture features are extracted to assist the classification using an ensemble-based classifier. Owing to the limited WCE resources, it is impossible for the WCE system to perform computationally intensive video summarization tasks. To resolve computational challenges, mobile-cloud architecture is incorporated, which provides resizable computing capacities by adaptively offloading video summarization tasks between the client and the cloud server. The qualitative and quantitative results are encouraging and show that the proposed framework saves information transmission cost and bandwidth, as well as the valuable time of data analysts in browsing remote sensing data.

Visual sensor networks (VSNs) usually generate a low-resolution (LR) frame-sequence due to energy and processing constraints. These LR-frames are not very appropriate for use in certain surveillance applications. It is very important to enhance the resolution of the captured LR-frames using resolution enhancement schemes. In this paper, an effective framework for a super-resolution (SR) scheme is proposed that enhances the resolution of LR key-frames extracted from frame-sequences captured by visual-sensors. In a VSN, a visual processing hub (VPH) collects a huge amount of visual data from camera sensors. In the proposed framework, at the VPH, key-frames are extracted using our recent key-frame extraction technique and are streamed to the base station (BS) after compression. A novel effective SR scheme is applied at BS to produce a high-resolution (HR) output from the received key-frames. The proposed SR scheme uses optimized orthogonal matching pursuit (OOMP) for sparse-representation recovery in SR. OOMP does better in terms of detecting true sparsity than orthogonal matching pursuit (OMP). This property of the OOMP helps produce a HR image which is closer to the original image. The K-SVD dictionary learning procedure is incorporated for dictionary learning. Batch-OMP improves the dictionary learning process by removing the limitation in handling a large set of observed signals. Experimental results validate the effectiveness of the proposed scheme and show its superiority over other state-of-the-art schemes.

Movies have become one of the major sources of entertainment in the current era, which are based on diverse ideas. Action movies have received the most attention in last few years, which contain violent scenes, because it is one of the undesirable features for some individuals that is used to create charm and fantasy. However, these violent scenes have had a negative impact on kids, and they are not comfortable even for mature age people. The best way to stop under aged people from watching violent scenes in movies is to eliminate these scenes. In this paper, we proposed a violence detection scheme for movies that is comprised of three steps. First, the entire movie is segmented into shots, and then a representative frame from each shot is selected based on the level of saliency. Next, these selected frames are passed from a light-weight deep learning model, which is fine-tuned using a transfer learning approach to classify violence and non-violence shots in a movie. Finally, all the non-violence scenes are merged in a sequence to generate a violence-free movie that can be watched by children and as well violence paranoid people. The proposed model is evaluated on three violence benchmark datasets, and it is experimentally proved that the proposed scheme provides a fast and accurate detection of violent scenes in movies compared to the state-of-the-art methods.

The electric energy consumption prediction (EECP) is an essential and complex task in intelligent power management system. EECP plays a significant role in drawing up a national energy development policy. Therefore, this study proposes an Electric Energy Consumption Prediction model utilizing the combination of Convolutional Neural Network (CNN) and Bi-directional Long Short-Term Memory (Bi-LSTM) that is named EECP-CBL model to predict electric energy consumption. In this framework, two CNNs in the first module extract the important information from several variables in the individual household electric power consumption (IHEPC) dataset. Then, Bi-LSTM module with two Bi-LSTM layers uses the above information as well as the trends of time series in two directions including the forward and backward states to make predictions. The obtained values in the Bi-LSTM module will be passed to the last module that consists of two fully connected layers for finally predicting the electric energy consumption in the future. The experiments were conducted to compare the prediction performances of the proposed model and the state-of-the-art models for the IHEPC dataset with several variants. The experimental results indicate that EECP-CBL framework outperforms the state-of-the-art approaches in terms of several performance metrics for electric energy consumption prediction on several variations of IHEPC dataset in real-time, short-term, medium-term and long-term timespans.

The worldwide utilization of surveillance cameras in smart cities has enabled researchers to analyze a gigantic volume of data to ensure automatic monitoring. An enhanced security system in smart cities, schools, hospitals, and other surveillance domains is mandatory for the detection of violent or abnormal activities to avoid any casualties which could cause social, economic, and ecological damages. Automatic detection of violence for quick actions is very significant and can efficiently assist the concerned departments. In this paper, we propose a triple-staged end-to-end deep learning violence detection framework. First, persons are detected in the surveillance video stream using a light-weight convolutional neural network (CNN) model to reduce and overcome the voluminous processing of useless frames. Second, a sequence of 16 frames with detected persons is passed to 3D CNN, where the spatiotemporal features of these sequences are extracted and fed to the Softmax classifier. Furthermore, we optimized the 3D CNN model using an open visual inference and neural networks optimization toolkit developed by Intel, which converts the trained model into intermediate representation and adjusts it for optimal execution at the end platform for the final prediction of violent activity. After detection of a violent activity, an alert is transmitted to the nearest police station or security department to take prompt preventive actions. We found that our proposed method outperforms the existing state-of-the-art methods for different benchmark datasets.

In recent years, image databases are growing at exponential rates, making their management, indexing, and retrieval, very challenging. Typical image retrieval systems rely on sample images as queries. However, in the absence of sample query images, hand-drawn sketches are also used. The recent adoption of touch screen input devices makes it very convenient to quickly draw shaded sketches of objects to be used for querying image databases. This paper presents a mechanism to provide access to visual information based on users' hand-drawn partially colored sketches using touch screen devices. A key challenge for sketch-based image retrieval systems is to cope with the inherent ambiguity in sketches due to the lack of colors, textures, shading, and drawing imperfections. To cope with these issues, we propose to fine-tune a deep convolutional neural network (CNN) using augmented dataset to extract features from partially colored hand-drawn sketches for query specification in a sketch-based image retrieval framework. The large augmented dataset contains natural images, edge maps, hand-drawn sketches, de-colorized, and de-texturized images which allow CNN to effectively model visual contents presented to it in a variety of forms. The deep features extracted from CNN allow retrieval of images using both sketches and full color images as queries. We also evaluated the role of partial coloring or shading in sketches to improve the retrieval performance. The proposed method is tested on two large datasets for sketch recognition and sketch-based image retrieval and achieved better classification and retrieval performance than many existing methods.

Medical image collections contain a wealth of information which can assist radiologists and medical experts in diagnosis and disease detection for making well-informed decisions. However, this objective can only be realized if efficient access is provided to semantically relevant cases from the ever-growing medical image repositories. In this paper, we present an efficient method for representing medical images by incorporating visual saliency and deep features obtained from a fine-tuned convolutional neural network (CNN) pre-trained on natural images. Saliency detector is employed to automatically identify regions of interest like tumors, fractures, and calcified spots in images prior to feature extraction. Neuronal activation features termed as neural codes from different CNN layers are comprehensively studied to identify most appropriate features for representing radiographs. This study revealed that neural codes from the last fully connected layer of the fine-tuned CNN are found to be the most suitable for representing medical images. The neural codes extracted from the entire image and salient part of the image are fused to obtain the saliency-injected neural codes (SiNC) descriptor which is used for indexing and retrieval. Finally, locality sensitive hashing techniques are applied on the SiNC descriptor to acquire short binary codes for allowing efficient retrieval in large scale image collections. Comprehensive experimental evaluations on the radiology images dataset reveal that the proposed framework achieves high retrieval accuracy and efficiency for scalable image retrieval applications and compares favorably with existing approaches.

Recently, a standard dataset namely SCADI (Self-Care Activities Dataset) based on the International Classification of Functioning, Disability, and Health for Children and Youth framework for self-care problems identification of children with physical and motor disabilities was introduced. This is a very interesting, important and challenging topic due to its usefulness in medical diagnosis. This study proposes a robust framework using a sampling technique and extreme gradient boosting (FSX) to improve the prediction performance for the SCADI dataset. The proposed framework first converts the original dataset to a new dataset with a smaller number of dimensions. Then, our proposed framework balances the new dataset in the previous step using oversampling techniques with different ratios. Next, extreme gradient boosting was used to diagnose the problems. The experiments in terms of prediction performance and feature importance were conducted to show the effectiveness of FSX as well as to analyse the results. The experimental results show that FSX that uses the Synthetic Minority Over-sampling Technique (SMOTE) for the oversampling module outperforms the ANN (Artificial Neural Network) -based approach, Support vector machine (SVM) and Random Forest for the SCADI dataset. The overall accuracy of the proposed framework reaches 85.4%, a pretty high performance, which can be used for self-care problem classification in medical diagnosis.

Bankruptcy prediction has been a popular and challenging research topic in both computer science and economics due to its importance to financial institutions, fund managers, lenders, governments, as well as economic stakeholders in recent years. In a bankruptcy dataset, the problem of class imbalance, in which the number of bankruptcy companies is smaller than the number of normal companies, leads to a standard classification algorithm that does not work well. Therefore, this study proposes a cluster-based boosting algorithm as well as a robust framework using the CBoost algorithm and Instance Hardness Threshold (RFCI) for effective bankruptcy prediction of a financial dataset. This framework first resamples the imbalance dataset by the undersampling method using Instance Hardness Threshold (IHT), which is used to remove the noise instances having large IHT value in the majority class. Then, this study proposes a Cluster-based Boosting algorithm, namely CBoost, for dealing with the class imbalance. In this algorithm, the majority class will be clustered into a number of clusters. The distance from each sample to its closest centroid will be used to initialize its weight. This algorithm will perform several iterations for finding weak classifiers and combining them to create a strong classifier. The resample set resulting from the previous module, will be used to train CBoost, which will be used to predict bankruptcy for the validation set. The proposed framework is verified by the Korean bankruptcy dataset (KBD), which has a very small balancing ratio in both the training and the testing phases. The experimental results of this research show that the proposed framework achieves 86.8% in AUC (area under the ROC curve) and outperforms several methods for dealing with the imbalanced data problem for bankruptcy prediction such as GMBoost algorithm, the oversampling-based method using SMOTEENN, and the clustering-based undersampling method for bankruptcy prediction in the experimental dataset.

In medical field visualization of the organs is very imperative for accurate diagnosis and treatment of any disease. Brain tumor diagnosis and surgery also required impressive 3D visualization of the brain to the radiologist. Detection and 3D reconstruction of brain tumors from MRI is a computationally time consuming and error-prone task. Proposed system detects and presents a 3D visualization model of the brain and tumor inside which greatly helps the radiologist to effectively diagnose and analyze the brain tumor. We proposed a multi-phase segmentation and visualization technique which overcomes the many problems of 3D volume segmentation methods like lake of fine details. In this system segmentation is done in three different phases which reduces the error chances. The system finds contours for skull, brain and tumor. These contours are stacked over and two novel methods are used to find the 3D visualization models. The results of these techniques, particularly of interpolation based, are impressive. Proposed system is tested against publically available data set [41] and MRI datasets available from MRI & CT center Rawalpindi, Pakistan [42].