Search

Analyzing surveillance videos is mandatory for the public and industrial security. Overwhelming growth in computer vision fields has been made to automate the surveillance system in terms of human activity recognition such as behavior analysis, Violence Detection (VD), etc. However, it is challenging to detect and analyze the violent scenes intelligently to fulfill the notion of Industrial Internet of Things (IIoT)-based surveillance buoyed by constrained resources to reduce computational power. To tackle this challenge, an Artificial Intelligence enabled IIoT-based framework with VD-Network (VD-Net) is proposed. First, the input video frames are passed to light-weight convolutional neural network model for important information collection including humans or suspicious objects such as Knives/Guns. Upon suspicious object detection, an alert is generated as an earlier VD in IIoT network while the information is shared with concern departments. Only the frames with objects are forwarded to cloud for detail investigation where features are extracted using Convolutional Long Short-term Memory (ConvLSTM). The latter from ConvLSTM is propagated to Gated Recurrent Unit for final VD. The conducted experiments and ablation study on the existing surveillance and non-surveillance datasets empirically validate the effectiveness of the proposed VD-Net by improving 3.9% increase in the accuracy compared to state-of-the-art (SOTA) VD methods.

Due to recent advances in the film industry, the production of movies has grown exponentially, which has led to challenges in what is referred to as discoverability: given the overwhelming number of choices, choosing which film to watch has become a tedious task for audiences. Movie summarization (MS) could help, as it presents the central theme of the movie in a compact format and makes browsing more efficient for the audience. In this paper, we present an automatic MS framework coined as ‘QuickLook’, which identifies the leading characters and fuses multiple cues extracted from a movie. Firstly, the movie data is preprocessed for its division into scenes, followed by shot segmentation. Secondly, the leading characters in each segmented scene are determined. Next, four visual cues that capture the film's scenic beauty, memorability, informativeness and emotional resonance are extracted from shots containing the leading characters. These extracted features are then intelligently fused based on the assignment of different weights; shots with a fusion score above a certain threshold are selected for the final summary. The proposed MS framework is assessed by comparison with official trailers from ten Hollywood movies, providing a novel baseline for future fair comparison in the MS literature. The proposed framework is shown to outperform other state-of-the-art MS methods in terms of enjoyability and informativeness.

In current technological era, surveillance systems generate an enormous volume of video data on a daily basis, making its analysis a difficult task for computer vision experts. Manually searching for unusual events in these massive video streams is a challenging task, since they occur inconsistently and with low probability in real-world surveillance. In contrast, deep learning-based anomaly detection reduces human labour and its decision making ability is comparatively reliable, thereby ensuring public safety. In this paper, we present an efficient deep features-based intelligent anomaly detection framework that can operate in surveillance networks with reduced time complexity. In the proposed framework, we first extract spatiotemporal features from a series of frames by passing each one to a pre-trained Convolutional Neural Network (CNN) model. The features extracted from the sequence of frames are valuable in capturing anomalous events. We then pass the extracted deep features to multi-layer Bi-directional Long Short-term Memory (BD-LSTM) model, which can accurately classify ongoing anomalous/normal events in complex surveillance scenes of smart cities. We performed extensive experiments on various anomaly detection benchmark datasets to validate the functionality of the proposed framework within complex surveillance scenarios. We reported a 3.41% and 8.09% increase in accuracy on UCF-Crime and UCFCrime2Local datasets compared to state-of-the-art methods.

Similar to a fingerprint search system, face recognition technology can assist law enforcement agencies in identifying suspects or finding missing persons. Face recognition technology lets the police detect a suspect’s face and compare it with image databases of known criminals and provides investigators with a match list of the most similar faces. Face recognition is a highly efficient and accurate tool in investigation processes. However, in some sensitive scenarios covert methods are required for the detection of suspects or missing persons without risking the lives of police. With the availability of the nano devices such as Raspberry Pi, law enforcement agencies such as police can be equipped with a concealed and secure face recognition system. In this paper, a Raspberry Pi and cloud assisted face recognition framework is proposed. A small-sized portable wireless camera is mounted on a police officer’s uniform to capture a video stream, which is passed to Raspberry Pi for face detection and recognition. The proposed method uses Bag of Words for extraction of oriented FAST and rotated BRIEF points from the detected face, followed by support vector machine for identification of suspects. Raspberry Pi has limited resources such as storage space, memory, and processing power, and therefore the proposed classifier is stored and trained on the cloud. The proposed method is implemented on Raspberry Pi 3 model B in Python 2.7 and is tested on various standard datasets. Experimental results validate the efficiency of the proposed method in accurate detection of faces compared to state-of-the-art face detection and recognition methods, and verify its effectiveness for enhancing law-enforcement services in smart cities.

The widespread usage of surveillance cameras in smart cities has resulted in a gigantic volume of video data whose indexing, retrieval and management is a challenging issue. Video summarization tends to detect important visual data from the surveillance stream and can help in efficient indexing and retrieval of required data from huge surveillance datasets. In this research article, we propose an efficient convolutional neural network based summarization method for surveillance videos of resource-constrained devices. Shot segmentation is considered as a backbone of video summarization methods and it affects the overall quality of the generated summary. Thus, we propose an effective shot segmentation method using deep features. Furthermore, our framework maintains the interestingness of the generated summary using image memorability and entropy. Within each shot, the frame with highest memorability and entropy score is considered as a keyframe. The proposed method is evaluated on two benchmark video datasets and the results are encouraging compared to state-of-the-art video summarization methods.

Mining uncertain frequent patterns (UFPs) from uncertain databases was recently introduced, and there are various approaches to solve this problem in the last decade. However, systems are often faced with the problem of too many UFPs being discovered by the traditional approaches to this issue, and thus will spend a lot of time and resources to rank and find the most promising patterns. Therefore, this paper introduces a task named mining top-k UFPs from uncertain databases. We then propose an efficient method named TUFP (mining Top-k UFPs) to carry this out. Effective threshold raising strategies are introduced to help the proposed algorithm reduce the number of generated candidates to enhance the performance in terms of the runtime as well as memory usage. Finally, several experiments on the number of generated candidates, mining time, memory usage and scalability of TUFP and two state-of-the-art approaches (CUFP-mine and LUNA) were conducted. The performance studies show that TUFP is efficient in terms of mining time, memory usage and scalability for mining top-k UFPs.

Electric energy forecasting domain attracts researchers due to its key role in saving energy resources, where mainstream existing models are based on Gradient Boosting Regression (GBR), Artificial Neural Networks (ANNs), Extreme Learning Machine (ELM) and Support Vector Machine (SVM). These models encounter high-level of non-linearity between input data and output predictions and limited adoptability in real-world scenarios. Meanwhile, energy forecasting domain demands more robustness, higher prediction accuracy and generalization ability for real-world implementation. In this paper, we achieve the mentioned tasks by developing a hybrid sequential learning-based energy forecasting model that employs Convolution Neural Network (CNN) and Gated Recurrent Units (GRU) into a unified framework for accurate energy consumption prediction. The proposed framework has two major phases: (1) data refinement and (2) training, where the data refinement phase applies preprocessing strategies over raw data. In the training phase, CNN features are extracted from input dataset and fed in to GRU, that is selected as optimal and observed to have enhanced sequence learning abilities after extensive experiments. The proposed model is an effective alternative to the previous hybrid models in terms of computational complexity as well prediction accuracy, due to the representative features' extraction potentials of CNNs and effectual gated structure of multi-layered GRU. The experimental evaluation over existing energy forecasting datasets reveal the better performance of our method in terms of preciseness and efficiency. The proposed method achieved the smallest error rate on Appliances Energy Prediction (AEP) and Individual Household Electric Power Consumption (IHEPC) datasets, when compared to other baseline models.

Excessive Power Consumption (PC) and demand for power is increasing on a daily basis, due to advancements in technology, the rise in electricity-dependent machinery, and the growth of the human population. It has become necessary to predict PC in order to improve power management and co-operation between the energy used in a building and the power grid. State-of-the-art Energy Consumption Prediction (ECP) methods are limited in terms of predicting the energy effectively, due to various challenges such as weather conditions and the dynamic behaviour of occupants. Thus, to overcome the drawbacks of these methods, we present an intelligent hybrid technique that combines a Convolutional Neural Network (CNN) with a Multi-layer Bi-directional Long-short Term Memory (M-BDLSTM) method using three steps. When applied to short-term power ECP, this approach helps to provide efficient power management i.e. it can assist the supplier to produce the optimum amount of power. The first step in our proposed method integrates the pre-processing and data organisation mechanisms to refine the data and remove abnormalities. The second step employs a deep learning network, where the sequence of refined data is fed into the CNN via the M-BDLSTM network to learn the sequence pattern effectively. The third step generates the ECP/PC by comparing actual and predicted data series and evaluates the prediction using error metrics. The proposed method achieves better prediction results than existing techniques, thus demonstrating its effectiveness. Furthermore, it achieved the smallest value of the Mean Square Error (MSE) and Root Mean Square Error (RMSE) for individual household dataset using 10-fold Cross Validation (CV) and a hold-out (CV) method.

Over the last two decades, the diagnosis of bankruptcy firms has become extremely important to business owners, banks, governments, securities investors, and economic stakeholders, in order to hedge against financial damage. This topic has attracted the attention of many computer scientists, as well as the financial sector. In computer science, a predictive model will be developed to analyse a firm's financial statements to predict its fate in the future based on the existing bankruptcy dataset. Many machine learning models have been developed to predict bankruptcy using specific datasets. This manuscript proposes a fast and accurate approach that utilises GPU-based extreme gradient boosting machine using squared logistics loss (SqLL) denoted by gXGBS for bankruptcy forecasting on both scenarios, including an imbalanced dataset (Korean bankruptcy dataset [KRBDS]) and two balanced datasets (USA bankruptcy dataset [USABDS] and Japanese bankruptcy dataset [JPNBDS]). To improve the performance of this prediction task, the study uses a custom loss function, namely SqLL, for an extreme gradient boosting machine to propose XGBS algorithm. Then, this study employs a GPU-based implementation of a decision tree construction algorithm (gDTC) to accelerate the processing time of XGBS. This is called a gXGBS algorithm. In addition, we also utilise a histogram-based tree construction algorithm using multi-GPU (gHTC), an approximation algorithm , for a gXGBS algorithm that will propose the third algorithm, namely gXGBS_hist, for the large datasets. The comprehensive experiments were conducted on three experimental datasets to evaluate the proposed approaches. The experimental results indicate that our proposed approaches outperform the state-of-the-art machine learning approaches for bankruptcy forecasting in terms of geometric mean (G-mean), Area Under the Receiver Operating Characteristic Curve (AUC) and processing time for all experimental datasets.

Video anomaly recognition in smart cities is an important computer vision task that plays a vital role in smart surveillance and public safety but is challenging due to its diverse, complex, and infrequent occurrence in real-time surveillance environments. Various deep learning models use significant amounts of training data without generalization abilities and with huge time complexity. To overcome these problems, in the current work, we present an efficient light-weight convolutional neural network (CNN)-based anomaly recognition framework that is functional in a surveillance environment with reduced time complexity. We extract spatial CNN features from a series of video frames and feed them to the proposed residual attention-based long short-term memory (LSTM) network, which can precisely recognize anomalous activity in surveillance videos. The representative CNN features with the residual blocks concept in LSTM for sequence learning prove to be effective for anomaly detection and recognition, validating our model’s effective usage in smart cities video surveillance. Extensive experiments on the real-world benchmark UCF-Crime dataset validate the effectiveness of the proposed model within complex surveillance environments and demonstrate that our proposed model outperforms state-of-the-art models with a 1.77%, 0.76%, and 8.62% increase in accuracy on the UCF-Crime, UMN and Avenue datasets, respectively.

The use of electrical energy is directly proportional to the increase in global population, both concerning growing industrialization and rising residential demand. The need to achieve a balance between electrical energy production and consumption inspires researchers to develop forecasting models for optimal and economical energy use. Mostly, the residential and industrial sectors use metering sensors that only measure the consumed energy but are unable to manage electricity. In this paper, we present a comparative analysis of a variety of deep features with several sequential learning models to select the optimized hybrid architecture for energy consumption prediction. The best results are achieved using convolutional long short-term memory (ConvLSTM) integrated with bidirectional long short-term memory (BiLSTM). The ConvLSTM initially extracts features from the input data to produce encoded sequences that are decoded by BiLSTM and then proceeds with a final dense layer for energy consumption prediction. The overall framework consists of preprocessing raw data, extracting features, training the sequential model, and then evaluating it. The proposed energy consumption prediction model outperforms existing models over publicly available datasets, including Household and Korean commercial building datasets.

Person Re-identification (P-ReID) task searches for true matches of a given query from a large repository of non-overlapping camera’s images/videos. In smart cities surveillance, P-ReID is challenging due to variation in human’s appearance, illumination affects, and difference in viewpoints. The mainstream approaches achieve P-ReID by implementing supervised learning strategies, requiring exhaustive manual annotation, which is probably erroneous due to human involvement. In addition, the employed methods use high-dimensional feature maps to identify a person, which is not a realistic approach in terms of storage resources and computational complexity. To tackle these issues, we incorporate learned features and deep autoencoder under the umbrella of a unified framework for P-ReID. First, we apply a unique image patching strategy by dividing the input image into two parts (upper and lower) and acquire learned features from fully connected layer of a pretrained Convolutional Neural Network (CNN) model for both patches. To achieve efficient and high performance, the proposed framework utilizes a self-tuned autoencoder to acquire low-dimensional representative features. The obtained features are matched with the patterns of database via cosine similarity measurement to re-identify a person’s appearance. The proposed framework provides a trade-off between time complexity and accuracy, where a lightweight model can be incorporated with reduced number of autoencoder layers to obtain fast and comparatively flexible results. The major novelty of the proposed framework includes implementation of a hybrid network mechanism for P-ReID, which shows convincing real-time results and best fits for smart cities surveillance. The proposed framework is tested over several P-ReID datasets to prove its influence over the existing works with reduced computational complexity.

[EN] There has been an exponential growth in the amount of visual data on a daily basis acquired from single or multi-view surveillance camera networks. This massive amount of data requires efficient mechanisms such as video summarization to ensure that only significant data are reported and the redundancy is reduced. Multi-view video summarization (MVS) is a less redundant and more concise way of providing information from the video content of all the cameras in the form of either keyframes or video segments. This paper presents an overview of the existing strategies proposed for MVS, including their advantages and drawbacks. Our survey covers the genericsteps in MVS, such as the pre-processing of video data, feature extraction, and post-processing followed by summary generation. We also describe the datasets that are available for the evaluation of MVS. Finally, we examine the major current issues related to MVS and put forward the recommendations for future research(1). (C) 2020 Elsevier Ltd. All rights reserved., This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2019R1A2B5B01070067)

Tuberculosis disease is one of the most leading cause of fatality worldwide. however, it can be reduced if diagnosed and treated on time. Normally the method name Ziehl-Neelsen is used to diagnose Tuberculosis and a human specialist analyzes it using an optical microscope to find tuberculosis bacilli. Since this process is time-consuming, an automatic bacilli recognition system allows the diagnosis process faster. In this work, an automatic tuberculosis bacilli segmentation system is developed. Initially, the input image is preprocessed by applying adaptive mean filter (AMD) to remove impulse noise and power law transformation to enhance the image then transform the color space from RGB to HSV. The HSV color space is more suitable for image processing because each element is isolated in it. Next, we employed the multi-level thresholding algorithm to correctly segment each bacillus in the input sample and improved 2.13% accuracy when compared to state-of-the-art techniques.

One-shot image recognition has been explored for many applications in computer vision community. However, its applications in video analytics is not deeply investigated yet. For instance, surveillance anomaly recognition is an open challenging problem and one of its hurdles is the lack of accurate temporally annotated data. This paper addresses the lack of data issue using one-shot learning strategy and proposes an anomaly recognition framework which exploits a 3D CNN siamese network that yields the similarity between two anomaly sequences. This paper also investigates the existing 3D CNNs for this task and then proposes a lightweight 3D CNN model that efficiently handles one-shot anomaly recognition. Once our network is trained, then we can use the powerful discriminative 3D CNN features to predict anomalies not only for the new data but also for entirely new classes. The proposed model is trained using temporally annotated test set of UCF Crime dataset. Finally, the trained model is used to recognize the anomalies and produce temporal automatic labels for the video level weakly annotated training set of the dataset.

Recent advances in the film industry have given rise to exponential growth in movie/drama production and adaptation of the Big Data concept. Automatic identification and classification of movie characters have received tremendous attention from researchers due to its applications in video semantic analysis, video summarization, and personalized video retrieval for which several methods have been recently presented. However, these methods cannot detect main characters properly due to their variation in pose and style in different scenes of a movie. To address this problem we present DeepStar, a novel framework for starring character identification based on deep high-level robust features. The proposed framework is threefold: the extraction of shots with clear faces from the input video; face clustering using discriminative deep features; and the occurrence matrix generation, helping in the selection of starring characters. The promising results obtained using representative Hollywood movies demonstrate the effectiveness of our method in detecting starring characters over the state-of-the-art methods.

A variety of sensors have been introduced for different applications such as setting off a fire alarm, detecting vehicle obstacles, visualizing the interior of the human body for diagnosis, monitoring animals and ships, and surveillance. Early fire detection in the context of disaster management systems during surveillance of public areas, forests, and nuclear power plants can result in the prevention of ecological, economic, and social damage. Using smart surveillance systems, various abnormal events such as road accidents, fires and medical emergencies can be detected at early stages, and the appropriate authority can be autonomously informed. The chapter presents an overview of image representation techniques for fire detection based on traditional hand-crafted features and deep learning approaches. It introduces the main contributions to fire detection literature for both indoor and outdoor surveillance. The chapter proposes a cost-effective fire detection convolutional neural networks architecture for surveillance videos.

The exponential growth in population and their overall reliance on the usage of electrical and electronic devices have increased the demand for energy production. It needs precise energy management systems that can forecast the usage of the consumers for future policymaking. Embedded smart sensors attached to electricity meters and home appliances enable power suppliers to effectively analyze the energy usage to generate and distribute electricity into residential areas based on their level of energy consumption. Therefore, this paper proposes a clustering-based analysis of energy consumption to categorize the consumers&rsquo, electricity usage into different levels. First, a deep autoencoder that transfers the low-dimensional energy consumption data to high-level representations was trained. Second, the high-level representations were fed into an adaptive self-organizing map (SOM) clustering algorithm. Afterward, the levels of electricity energy consumption were established by conducting the statistical analysis on the obtained clustered data. Finally, the results were visualized in graphs and calendar views, and the predicted levels of energy consumption were plotted over the city map, providing a compact overview to the providers for energy utilization analysis.

Discovering Erasable Patterns (EPs) consists of identifying product parts that will produce a small profit loss if their production is stopped. It is a data mining problem that has attracted the attention of numerous researchers in recent years due to the possibility of using EPs to reduce profit loss of manufacturers. Though, many algorithms have been designed to mine EPs, an important limitation of state-of-the-art EP mining algorithms is that they are batch algorithms, that is, they are designed to be applied on static databases. But in real-life applications, databases are dynamic, as they are constantly updated by adding or removing products and parts. To be informed about EPs in real-time, traditional EP mining algorithms must be applied over and over again on a database. This is inefficient as those algorithms are always applied from scratch without taking advantage of results generated by previous executions. Considering this important drawback of previous work for handling real-life dynamic data, this paper proposes an efficient algorithm named MSPPC for mining EPs in data streams. It relies on a novel tree structure named SPPC (Streaming Pre-Post Code) tree, which extends the WPPC tree structure for maintaining a compact tree representation of EPs in a data stream. Experimental results show that the designed MSPPC algorithm outperforms the state-of-the-art batch MERIT and dMERIT algorithms when they are run in batch mode using a sliding-window. Besides, the proposed algorithm is also faster than the state-of-the-art algorithms for mining EPs, namely MERIT, dMERIT + , MEI and EIFDD.

The file attached to this record is the author's final peer reviewed version. The Publisher's final version can be found by following the DOI link Association rule mining (ARM) aims to find out association rules that satisfy predefined minimum support and confidence from a given database. However, in many cases ARM generates extremely large number of association rules, which are impossible for end users to comprehend or validate, thereby limiting the usefulness of data mining results. In this paper, we propose a new mining algorithm based on Animal Migration Optimization (AMO), called ARM-AMO, to reduce the number of association rules. It is based on the idea that rules which are not of high support and unnecessary are deleted from the data. Firstly, Apriori algorithm is applied to generate frequent itemsets and association rules. Then, AMO is used to reduce the number of association rules with a new fitness function that incorporates frequent rules. It is observed from the experiments that, in comparison with the other relevant techniques, ARM-AMO greatly reduces the computational time for frequent item set generation, memory for association rule generation, and the number of rules generated.

Malware detection is an important factor in the security of the smart devices. However, currently utilized signature-based methods cannot provide accurate detection of zero-day attacks and polymorphic viruses. In this context, an efficient hybrid framework is presented for detection of malware in Android Apps. The proposed framework considers both signature and heuristic-based analysis for Android Apps. We have reverse engineered the Android Apps to extract manifest files, and binaries, and employed state-of-the-art machine learning algorithms to efficiently detect malwares. For this purpose, a rigorous set of experiments are performed using various classifiers such as SVM, Decision Tree, W-J48 and KNN. It has been observed that SVM in case of binaries and KNN in case of manifest.xml files are the most suitable options in robustly detecting the malware in Android devices. The proposed framework is tested on benchmark datasets and results show improved accuracy in malware detection.

The quality of health services provided by medical centers varies widely, and there is often a large gap between the optimal standard of services when judged based on the locality of patients (rural or urban environments). This quality gap can have serious health consequences and major implications for patient’s timely and correct treatment. These deficiencies can manifest, for example, as a lack of quality services, misdiagnosis, medication errors, and unavailability of trained professionals. In medical imaging, MRI analysis assists radiologists and surgeons in developing patient treatment plans. Accurate segmentation of anomalous tissues and its correct 3D visualization plays an important role inappropriate treatment. In this context, we aim to develop an intelligent computer-aided diagnostic system focusing on human brain MRI analysis. We present brain tumor detection, segmentation, and its 3D visualization system, providing quality clinical services, regardless of geographical location, and level of expertise of medical specialists. In this research, brain magnetic resonance (MR) images are segmented using a semi-automatic and adaptive threshold selection method. After segmentation, the tumor is classified into malignant and benign based on a bag of words (BoW) driven robust support vector machine (SVM) classification model. The BoW feature extraction method is further amplified via speeded up robust features (SURF) incorporating its procedure of interest point selection. Finally, 3D visualization of the brain and tumor is achieved using volume marching cube algorithm which is used for rendering medical data. The effectiveness of the proposed system is verified over a dataset collected from 30 patients and achieved 99% accuracy. A subjective comparative analysis is also carried out between the proposed method and two state-of-the-art tools ITK-SNAP and 3D-Doctor. Experimental results indicate that the proposed system performed better than existing systems and assists radiologist determining the size, shape, and location of the tumor in the human brain.

Large scale visual surveillance generates huge volumes of data at a rapid pace, giving rise to massive image repositories. Efficient and reliable access to relevant data in these ever growing databases is a highly challenging task due to the complex nature of surveillance objects. Furthermore, inter-class visual similarity between vehicles requires extraction of fine-grained and highly discriminative features. In recent years, features from deep convolutional neural networks (CNN) have exhibited state-of-the-art performance in image retrieval. However, these features have been used without regard to their sensitivity to objects of a particular class. In this paper, we propose an object-oriented feature selection mechanism for deep convolutional features from a pre-trained CNN. Convolutional feature maps from a deep layer are selected based on the analysis of their responses to surveillance objects. The selected features serve to represent semantic features of surveillance objects and their parts with minimal influence of the background, effectively eliminating the need for background removal procedure prior to features extraction. Layer-wise mean activations from the selected features maps form the discriminative descriptor for each object. These object-oriented convolutional features (OOCF) are then projected onto low-dimensional hamming space using locality sensitive hashing approaches. The resulting compact binary hash codes allow efficient retrieval within large scale datasets. Results on five challenging datasets reveal that OOCF achieves better precision and recall than the full feature set for objects with varying backgrounds. Proposed to represent vehicle images with appropriate convolutional features.Our method reduces number of feature maps without performance degradation.Selected features yield better retrieval performance than the full feature set.

Mining erasable patterns (EPs) is one of the emerging tasks in data mining which helps factory managers to establish plans for the development of new systems of products. However, systems usually face the problem of many EPs. Therefore, the problem of mining top-rank-k EPs, and an algorithm for mining these using the PID_List structure named VM, were proposed in 2013. In this paper, we propose two efficient methods, named the TEP (Top-rank-k Erasable Pattern mining) and TEPUS (Top-rank-k Erasable Pattern mining Using the Subsume concept) algorithms, for mining top-rank-k EPs. The TEP algorithm uses the dPidset structure to reduce the memory usage and a dynamic threshold pruning strategy to accelerate the mining process. The TEPUS algorithm is the extension of the TEP algorithm using the subsume concept and the index strategy to further speed up the mining time and reduce the memory usage. Finally, we conduct an experiment to compare the mining time, memory usage and scalability of TEP, TEPUS and two state-of-the-art algorithms (VM and dVM) for mining top-rank-k EPs. Our performance studies show that TEPUS outperforms TEP, VM and dVM.

The recent advances in embedded processing have enabled the vision based systems to detect fire during surveillance using convolutional neural networks (CNNs). However, such methods generally need more computational time and memory, restricting its implementation in surveillance networks. In this research paper, we propose a cost-effective fire detection CNN architecture for surveillance videos. The model is inspired from GoogleNet architecture, considering its reasonable computational complexity and suitability for the intended problem compared to other computationally expensive networks such as AlexNet. To balance the efficiency and accuracy, the model is fine-tuned considering the nature of the target problem and fire data. Experimental results on benchmark fire datasets reveal the effectiveness of the proposed framework and validate its suitability for fire detection in CCTV surveillance systems compared to state-of-the-art methods.

Renewable energy (RE) power plants are deployed globally because the renewable energy sources (RESs) are sustainable, clean, and environmentally friendly. However, the demand for power increases on a daily basis due to population growth, technology, marketing, and the number of installed industries. This challenge has raised a critical issue of how to intelligently match the power generation with the consumption for efficient energy management. To handle this issue, we propose a novel architecture called ‘AB-Net’: a one-step forecast of RE generation for short-term horizons by incorporating an autoencoder (AE) with bidirectional long short-term memory (BiLSTM). Firstly, the data acquisition step is applied, where the data are acquired from various RESs such as wind and solar. The second step performs deep preprocessing of the acquired data via several de-noising and cleansing filters to clean the data and normalize them prior to actual processing. Thirdly, an AE is employed to extract the discriminative features from the cleaned data sequence through its encoder part. BiLSTM is used to learn these features to provide a final forecast of power generation. The proposed AB-Net was evaluated using two publicly available benchmark datasets where the proposed method obtains state-of-the-art results in terms of the error metrics.

Emotion recognition from speech signals is an interesting research with several applications like smart healthcare, autonomous voice response systems, assessing situational seriousness by caller affective state analysis in emergency centers, and other smart affective services. In this paper, we present a study of speech emotion recognition based on the features extracted from spectrograms using a deep convolutional neural network (CNN) with rectangular kernels. Typically, CNNs have square shaped kernels and pooling operators at various layers, which are suited for 2D image data. However, in case of spectrograms, the information is encoded in a slightly different manner. Time is represented along the x-axis and y-axis shows frequency of the speech signal, whereas, the amplitude is indicated by the intensity value in the spectrogram at a particular position. To analyze speech through spectrograms, we propose rectangular kernels of varying shapes and sizes, along with max pooling in rectangular neighborhoods, to extract discriminative features. The proposed scheme effectively learns discriminative features from speech spectrograms and performs better than many state-of-the-art techniques when evaluated its performance on Emo-DB and Korean speech dataset.

Display Omitted Presents an effective statistical method to determine optimal subset of convolution kernels.Spatial maximal activator pooling strategy to aggregate feature maps into a single feature map.Discriminative and low-dimensional representation allow efficient and accurate retrieval.Modularity of proposed framework allows its convenient enhancement via complicated pooling. Modern surveillance networks are large collections of computational sensor nodes, where each node can be programmed to capture, prioritize, segment salient objects, and transmit them to central repositories for indexing. Visual data from such networks grow exponentially and present many challenges concerning their transmission, storage, and retrieval. Searching for particular surveillance objects is a common but challenging task. In this paper, we present an efficient features extraction framework which utilizes an optimal subset of kernels from the first layer of a convolutional neural network pre-trained on ImageNet dataset for object-based surveillance image search. The input image is convolved with the set of kernels to generate feature maps, which are aggregated into a single feature map using a novel spatial maximal activator pooling approach. A low-dimensional feature vector is computed to represent surveillance objects. The proposed system provides improvements in both performance and efficiency over other similar approaches for surveillance datasets.

Erasable itemset (EI) mining, a branch of pattern mining, helps managers to establish new plans for the development of new products. Although the problem of mining EIs was first proposed in 2009, many efficient algorithms for mining these have since been developed. However, these algorithms usually require a lot of time and memory usage. In reality, users only need a small number of EIs which satisfy a particular condition. Having this observation in mind, in this study we develop an efficient algorithm for mining EIs with subset and superset itemset constraints (C0 ⊆ X ⊆ C1). Firstly, based on the MEI (Mining Erasable Itemsets) algorithm, we present the MEIC (Mining Erasable Itemsets with subset and superset itemset Constraints) algorithm in which each EI is checked with regard to the constraints before being added to the results. Next, two propositions supporting quick pruning of nodes that do not satisfy the constraints are established. Based on these, we propose an efficient algorithm for mining EIs with subset and superset itemset constraints (called pMEIC – p: pruning). The experimental results show that pMEIC outperforms MEIC in terms of mining time and memory usage.

Social networks are major sources of image sharing and secret messaging among the people. To date, such networks are not strictly bounded by copyright laws due to which image sharing, secret messaging, and its authentication is vulnerable to many risks. In addition to this, maintaining the confidentiality, integrity, and authenticity of secret messages is an open challenge of today's communication systems. Steganography is one of the solutions to tackle these problems. This paper proposes a secure crystographic framework for authenticity of visual contents using image steganography, utilizing color model transformation, three-level encryption algorithm (TLEA), and Morton scanning (MS)-directed least significant bit (LSB) substitution. The method uses I-plane of the input image in HSI for secret data embedding using MS-directed LSB substitution method. Furthermore, the secret data is encrypted using TLEA prior to embedding, adding an additional level of security for secure authentication. The qualitative and quantitative results verify the better performance of the proposed scheme and provide one of the best mechanisms for authenticity of visual contents in social networks.

Refereed/Peer-reviewed The massive amount of video data produced by surveillance networks in industries instigate various challenges in exploring these videos for many applications, such as video summarization (VS), analysis, indexing, and retrieval. The task of multiview video summarization (MVS) is very challenging due to the gigantic size of data, redundancy, overlapping in views, light variations, and interview correlations. To address these challenges, various low-level features and clustering-based soft computing techniques are proposed that cannot fully exploit MVS. In this article, we achieve MVS by integrating deep neural network based soft computing techniques in a two-tier framework. The first online tier performs target-appearance-based shots segmentation and stores them in a lookup table that is transmitted to cloud for further processing. The second tier extracts deep features from each frame of a sequence in the lookup table and pass them to deep bidirectional long short-term memory (DB-LSTM) to acquire probabilities of informativeness and generates a summary. Experimental evaluation on benchmark dataset and industrial surveillance data from YouTube confirms the better performance of our system compared to the state-of-the-art MVS methods. usc

Movies have become one of the major sources of entertainment in the current era, which are based on diverse ideas. Action movies have received the most attention in last few years, which contain violent scenes, because it is one of the undesirable features for some individuals that is used to create charm and fantasy. However, these violent scenes have had a negative impact on kids, and they are not comfortable even for mature age people. The best way to stop under aged people from watching violent scenes in movies is to eliminate these scenes. In this paper, we proposed a violence detection scheme for movies that is comprised of three steps. First, the entire movie is segmented into shots, and then a representative frame from each shot is selected based on the level of saliency. Next, these selected frames are passed from a light-weight deep learning model, which is fine-tuned using a transfer learning approach to classify violence and non-violence shots in a movie. Finally, all the non-violence scenes are merged in a sequence to generate a violence-free movie that can be watched by children and as well violence paranoid people. The proposed model is evaluated on three violence benchmark datasets, and it is experimentally proved that the proposed scheme provides a fast and accurate detection of violent scenes in movies compared to the state-of-the-art methods.

The worldwide utilization of surveillance cameras in smart cities has enabled researchers to analyze a gigantic volume of data to ensure automatic monitoring. An enhanced security system in smart cities, schools, hospitals, and other surveillance domains is mandatory for the detection of violent or abnormal activities to avoid any casualties which could cause social, economic, and ecological damages. Automatic detection of violence for quick actions is very significant and can efficiently assist the concerned departments. In this paper, we propose a triple-staged end-to-end deep learning violence detection framework. First, persons are detected in the surveillance video stream using a light-weight convolutional neural network (CNN) model to reduce and overcome the voluminous processing of useless frames. Second, a sequence of 16 frames with detected persons is passed to 3D CNN, where the spatiotemporal features of these sequences are extracted and fed to the Softmax classifier. Furthermore, we optimized the 3D CNN model using an open visual inference and neural networks optimization toolkit developed by Intel, which converts the trained model into intermediate representation and adjusts it for optimal execution at the end platform for the final prediction of violent activity. After detection of a violent activity, an alert is transmitted to the nearest police station or security department to take prompt preventive actions. We found that our proposed method outperforms the existing state-of-the-art methods for different benchmark datasets.

Customer retention is invariably the top priority of all consumer businesses, and certainly it is one of the most critical challenges as well. Identifying and gaining insights into the most probable cause of churn can save from five to ten times in terms of cost for the company compared with finding new customers. Therefore, this study introduces a full-fledged geodemographic segmentation model, assessing it, testing it, and deriving insights from it. A bank dataset consisting 11,000 instances, which consists of 10,000 instances for training and 10,000 instances for testing, with 14 attributes, has been used, and the likelihood of a person staying with the bank or leaving the bank is computed with the help of logistic regression. Base on the proposed model, insights are drawn and recommendations are provided. Stepwise logistic regression methods, namely, backward elimination method, forward selection method, and bidirectional model are constructed and contrasted to choose the best among them. Future forecasting of the models has been done by using cumulative accuracy profile (CAP) curve analysis.

Smart cities are a future reality for municipalities around the world. Healthcare services play a vital role in the transformation of traditional cities into smart cities. In this paper, we present a ubiquitous and quality computer-aided blood analysis service for the detection and counting of white blood cells (WBCs) in blood samples. WBCs also called leukocytes or leucocytes are the cells of the immune system that are involved in protecting the body against both infectious disease and foreign invaders. Analysis of leukocytes provides valuable information to medical specialists, helping them in diagnosing different important hematic diseases, such as AIDS and blood cancer (Leukaemia). However, this task is prone to errors and can be time-consuming. A mobile-cloud-assisted detection and classification of leukocytes from blood smear images can enhance accuracy and speed up the detection of WBCs. In this paper, we propose a smartphone-based cloud-assisted resource aware framework for localization of WBCs within microscopic blood smear images using a trained multi-class ensemble classification mechanism in the cloud. In the proposed framework, nucleus is first segmented, followed by extraction of texture, statistical, and wavelet features. Finally, the detected WBCs are categorized into five classes: basophil, eosinophil, neutrophil, lymphocyte, and monocyte. Experimental results on numerous benchmark databases validate the effectiveness and efficiency of the proposed system in comparison to the other state-of-the-art schemes.

In the millions of emergency reporting calls made each year, about a quarter are non-emergencies. To avoid responding to such situations, forensic examination of the reported situation in the presence of speech as evidence has become an indispensable requirement for emergency response centers. Caller profile information like gender, age, emotional state, transcript, and contextual sounds determined from emergency calls, may be highly beneficial for their sophisticated forensic analysis. However, callers reporting emergency situations often express emotional stress which cause variations in speech production. Furthermore, low voice quality, and background noise make it very difficult to efficiently recognize caller attributes in such unconstrained environments. To overcome limitations of traditional classification systems in such situations, a hybrid two-stage classification scheme is proposed in this paper. Our framework consist of an ensemble of support vector machines (e-SVM) and deep neural networks (DNN) in a cascade. The first stage e-SVM consists of two models discriminatively trained on normal and stressful speech from emergency calls. Deep neural network forming the second stage of classification pipeline, is utilized only in case of ambiguous prediction results from the first stage. The adaptive nature of this two stage classification scheme helps achieve efficiency and high performance. Experiments conducted with a large dataset affirm the suitability of proposed architecture for efficient real-time speaker attribute recognition. The framework is evaluated for gender recognition from emergency calls in the presence of emotions and background noise. The framework yields significant performance improvements in comparison with other similar state-of-the-art gender recognition approaches.

In this paper, the problem of outsourcing the selective encryption of a medical image to cloud by resource-constrained devices such as smart phone is addressed, without revealing the cover image to cloud using steganography. In the proposed framework, the region of interest of the medical image is first detected using a visual saliency model. The detected important data is then embedded in a host image, producing a stego image which is outsourced to cloud for encryption. The cloud which has powerful resources, encrypts the image and sent back the encrypted marked image to the client. The client can then extract the selectively encrypted region of interest and can combine it with the region of non-interest to form a selectively encrypted image, which can be sent to medical specialists and healthcare centers. Experimental results and analysis validate the effectiveness of the proposed framework in terms of security, image quality, and computational complexity and verify its applicability in remote patient monitoring centers.

The availability and affordability of handheld smart devices have made life easier by enabling us to do work on the go. Their widespread use brings with it concerns relating to data security and privacy. The rising demand to secure private and highly confidential data found on smart devices has motivated researchers to devise means for ensuring privacy and security at all times. This kind of continuous user authentication scheme would add an additional layer of much needed security to smart devices. In this context, touch screen interactions have recently been studied as an effective modality to perform active user authentication on mobile devices. In this paper, a visual analysis based active authentication framework has been presented. Considering the touch screen as a canvas, interaction trace maps are constructed as a result of user interactions within various applications. The user touch gestures are captured and represented as drawing strokes on the canvas. The behavioral and physiological characteristics of users are modeled as signatures by combining texture and shape features from the interaction trace maps. A two-step mechanism with support vector machines exploit this signature to perform active user authentication. Experiments conducted with various datasets show that the proposed framework compares favorably with other state-of-the-art methods.

For efficient and effective energy management, accurate energy consumption forecasting is required in energy management systems (EMSs). Recently, several artificial intelligence-based techniques have been proposed for accurate electric load forecasting, moreover, perfect energy consumption data are critical for the prediction. However, owing to diverse reasons, such as device malfunctions and signal transmission errors, missing data are frequently observed in the actual data. Previously, many imputation methods have been proposed to compensate for missing values, however, these methods have achieved limited success in imputing electric energy consumption data because the period of data missing is long and the dependency on historical data is high. In this study, we propose a novel missing-value imputation scheme for electricity consumption data. The proposed scheme uses a bagging ensemble of multilayer perceptrons (MLPs), called softmax ensemble network, wherein the ensemble weight of each MLP is determined by a softmax function. This ensemble network learns electric energy consumption data with explanatory variables and imputes missing values in this data. To evaluate the performance of our scheme, we performed diverse experiments on real electric energy consumption data and confirmed that the proposed scheme can deliver superior performance compared to other imputation methods.

Due to industrialization and the rising demand for energy, global energy consumption has been rapidly increasing. Recent studies show that the biggest portion of energy is consumed in residential buildings, i.e., in European Union countries up to 40% of the total energy is consumed by households. Most residential buildings and industrial zones are equipped with smart sensors such as metering electric sensors, that are inadequately utilized for better energy management. In this paper, we develop a hybrid convolutional neural network (CNN) with an long short-term memory autoencoder (LSTM-AE) model for future energy prediction in residential and commercial buildings. The central focus of this research work is to utilize the smart meters&rsquo, data for energy forecasting in order to enable appropriate energy management in buildings. We performed extensive research using several deep learning-based forecasting models and proposed an optimal hybrid CNN with the LSTM-AE model. To the best of our knowledge, we are the first to incorporate the aforementioned models under the umbrella of a unified framework with some utility preprocessing. Initially, the CNN model extracts features from the input data, which are then fed to the LSTM-encoder to generate encoded sequences. The encoded sequences are decoded by another following LSTM-decoder to advance it to the final dense layer for energy prediction. The experimental results using different evaluation metrics show that the proposed hybrid model works well. Also, it records the smallest value for mean square error (MSE), mean absolute error (MAE), root mean square error (RMSE) and mean absolute percentage error (MAPE) when compared to other state-of-the-art forecasting methods over the UCI residential building dataset. Furthermore, we conducted experiments on Korean commercial building data and the results indicate that our proposed hybrid model is a worthy contribution to energy forecasting.

Bankruptcy prediction has been a popular and challenging research topic in both computer science and economics due to its importance to financial institutions, fund managers, lenders, governments, as well as economic stakeholders in recent years. In a bankruptcy dataset, the problem of class imbalance, in which the number of bankruptcy companies is smaller than the number of normal companies, leads to a standard classification algorithm that does not work well. Therefore, this study proposes a cluster-based boosting algorithm as well as a robust framework using the CBoost algorithm and Instance Hardness Threshold (RFCI) for effective bankruptcy prediction of a financial dataset. This framework first resamples the imbalance dataset by the undersampling method using Instance Hardness Threshold (IHT), which is used to remove the noise instances having large IHT value in the majority class. Then, this study proposes a Cluster-based Boosting algorithm, namely CBoost, for dealing with the class imbalance. In this algorithm, the majority class will be clustered into a number of clusters. The distance from each sample to its closest centroid will be used to initialize its weight. This algorithm will perform several iterations for finding weak classifiers and combining them to create a strong classifier. The resample set resulting from the previous module, will be used to train CBoost, which will be used to predict bankruptcy for the validation set. The proposed framework is verified by the Korean bankruptcy dataset (KBD), which has a very small balancing ratio in both the training and the testing phases. The experimental results of this research show that the proposed framework achieves 86.8% in AUC (area under the ROC curve) and outperforms several methods for dealing with the imbalanced data problem for bankruptcy prediction such as GMBoost algorithm, the oversampling-based method using SMOTEENN, and the clustering-based undersampling method for bankruptcy prediction in the experimental dataset.

Recent advances in multimedia technology have led to tremendous increases in the available volume of video data, thereby creating a major requirement for efficient systems to manage such huge data volumes. Video summarization is one of the key techniques for accessing and managing large video libraries. Video summarization can be used to extract the affective contents of a video sequence to generate a concise representation of its content. Human attention models are an efficient means of affective content extraction. Existing visual attention driven summarization frameworks have high computational cost and memory requirements, as well as a lack of efficiency in accurately perceiving human attention. To cope with these issues, we propose a divide-and-conquer based framework for an efficient summarization of big video data. We divide the original video data into shots, where an attention model is computed from each shot in parallel. Viewer's attention is based on multiple sensory perceptions, i.e., aural and visual, as well as the viewer's neuronal signals. The aural attention model is based on the Teager energy, instant amplitude, and instant frequency, whereas the visual attention model employs multi-scale contrast and motion intensity. Moreover, the neuronal attention is computed using the beta-band frequencies of neuronal signals. Next, an aggregated attention curve is generated using an intra- and inter-modality fusion mechanism. Finally, the affective content in each video shot is extracted. The fusion of multimedia and neuronal signals provides a bridge that links the digital representation of multimedia with the viewer's perceptions. Our experimental results indicate that the proposed shot-detection based divide-and-conquer strategy mitigates the time and computational complexity. Moreover, the proposed attention model provides an accurate reflection of the user preferences and facilitates the extraction of highly affective and personalized summaries.

The exponential growth in the volume of digital image databases is making it increasingly difficult to retrieve relevant information from them. Efficient retrieval systems require distinctive features extracted from visually rich contents, represented semantically in a human perception-oriented manner. This paper presents an efficient framework to model image contents as an undirected attributed relational graph, exploiting color, texture, layout, and saliency information. The proposed method encodes salient features into this rich representative model without requiring any segmentation or clustering procedures, reducing the computational complexity. In addition, an efficient graph-matching procedure implemented on specialized hardware makes it more suitable for real-time retrieval applications. The proposed framework has been tested on three publicly available datasets, and the results prove its superiority in terms of both effectiveness and efficiency in comparison with other state-of-the-art schemes.

In wireless visual sensor networks (WVSNs), streaming all imaging data is impractical due to resource constraints. Moreover, the sheer volume of surveillance videos inhibits the ability of analysts to extract actionable intelligence. In this work, an energy-efficient image prioritization framework is presented to cope with the fragility of traditional WVSNs. The proposed framework selects semantically relevant information before it is transmitted to a sink node. This is based on salient motion detection, which works on the principle of human cognitive processes. Each camera node estimates the background by a bootstrapping procedure, thus increasing the efficiency of salient motion detection. Based on the salient motion, each sensor node is classified as being high or low priority. This classification is dynamic, such that camera nodes toggle between high-priority and low-priority status depending on the coverage of the region of interest. High-priority camera nodes are allowed to access reliable radio channels to ensure the timely and reliable transmission of data. We compare the performance of this framework with other state-of-the-art methods for both single and multi-camera monitoring. The results demonstrate the usefulness of the proposed method in terms of salient event coverage and reduced computational and transmission costs, as well as in helping analysts find semantically relevant visual information.

This paper describes current trends in the prediction of crop pests using machine learning technology. With the advent of data mining, the field of agriculture is also focused on it. Currently, various studies, domestic and overseas, are under progress using machine learning technology, and cases of its utilization are increasing. This paper classifies and introduces SVM (Support Vector Machine), Multiple Linear Regression, Neural Network, and Bayesian Network based techniques, and describes some cases of their utilization.

The growing number of videos on the internet requires effective management strategies. Video summarization is a method for managing video data which provides succinct versions of the videos for efficient browsing and retrieval. The inter-frame disparity based key frame extraction is a popular scheme for summarizing videos. However, the performance of such schemes is limited by the fact that most of these techniques select the first frame in a shot as key frame which may not be representative of the shot. In this paper, we propose a saliency inspired inter-frame difference based summarization scheme which selects the most important frame of the shot based on color contrast saliency. The preliminary experimental results prove the efficacy of the proposed method.

Mixed reality (MR) systems provide an immersive virtual environment where the gap between real and virtual world is minimal. Current MR applications use fixed display to create this environment. However, the environment is never fixed in real world and it is difficult to achieve this non-static environment using traditional MR frameworks. Therefore, in this paper, an efficient multi-display MR system is proposed. The proposed system uses moveable displays to create dynamically changing immersive virtual environment. An agent-mediator based framework is used for communication in which a mediator is connected to several agents. Mediator administers the position of display and synchronizes the view across all agents. Agents gather interaction data from user which is passed to mediator. Moreover, distributed graphic contents rendering technique is used to efficiently render complex graphical contents. In this technique, mediator divides the contents and transmits these to each agent. This division of contents optimises the rendering process on agents, thus increasing the efficiency of the MR system.

In this paper, the problem of secure transmission of sensitive contents over the public network Internet is addressed by proposing a novel data hiding method in encrypted images with dual-level security. The secret information is divided into three blocks using a specific pattern, followed by an encryption mechanism based on the three-level encryption algorithm (TLEA). The input image is scrambled using a secret key, and the encrypted sub-message blocks are then embedded in the scrambled image by cyclic18 least significant bit (LSB) substitution method, utilizing LSBs and intermediate LSB planes. Furthermore, the cover image and its planes are rotated at different angles using a secret key prior to embedding, deceiving the attacker during data extraction. The usage of message blocks division, TLEA, image scrambling, and the cyclic18 LSB method results in an advanced security system, maintaining the visual transparency of resultant images and increasing the security of embedded data. In addition, employing various secret keys for image scrambling, data encryption, and data hiding using the cyclic18 LSB method makes the data recovery comparatively more challenging for attackers. Experimental results not only validate the effectiveness of the proposed framework in terms of visual quality and security compared to other state-of-the-art methods, but also suggest its feasibility for secure transmission of diagnostically important keyframes to healthcare centers and gastroenterologists during wireless capsule endoscopy.

Speech signals carry valuable information about the speaker including age, gender, and emotional state. Gender information can act as a vital preprocessing ingredient for enhancing speech analysis applications like adaptive human-machine interfaces, multi-modal security applications, and sophisticated intent and context analysis based forensic systems. In uncontrolled environments like telephone speech applications, the gender recognition system should be adaptive, accurate, and robust to noisy environments. This paper presents a reasoning method governed by Dempster-Shafer theory of evidence for automatic gender recognition from telephone speech. The proposed method uses mel-frequency cepstral coefficients with a support vector machine to generate the initial prediction results for individual speech segments. The reasoning scheme collects and validates results from support vector machine and treats convincing predictions as valid evidence. It is argued that the consideration of valid evidence in the reasoning process improves recognition performance by avoiding unconvincing classification results. Experiments conducted on large speech datasets reveal the superiority of the proposed gender recognition scheme for speech analysis applications.

In developing new personal electronic products, the development time has shortened to a few months due to high competition in the market. Now it has become very hard to meet the time to market by evaluating products by making physical prototypes. To overcome this problem, we propose an immersive modeling system (IMMS) that enables us to interact with a digital product model in the augmented virtual environment using a multi-modal interface. The IMMS allows the user to evaluate the product model realistically using visual, auditory, and tactile/force senses. The architecture and main modules of the system are described in detail. The integration problems encountered during the development of the test bed are discussed. Application examples to personal electronic products are also included.