Search

The increasing development of cryptocurrencies has brought cryptojacking as a new security threat in which attackers steal computing resources for cryptomining. The digitization of the supply chain is a potential major target for cryptojacking due to the large number of different infrastructures involved. These different infrastructures provide information sources that can be useful to detect cryptojacking, but with a wide variety of data formats and encodings. This paper describes the semantic data aggregator (SDA), a normalization and aggregation system based on data modelling and low-latency processing of data streams that facilitates the integration of heterogeneous information sources. As a use case, the paper describes a cryptomining detection system (CDS) based on network traffic flows processed by a machine learning engine. The results show how the SDA is leveraged in this use case to obtain aggregated information that improves the performance of the CDS. [ABSTRACT FROM AUTHOR]

Botnets pose a significant challenge to network security but are difficult to detect because of their dynamic and evolving nature, which limits the effectiveness of conventional supervised neural network detection methods. To address this problem, the present study proposes a novel neural network-based self-training framework for botnet detection, in which pseudo-labels are generated from unlabeled data by a trained classifier, which is iteratively refined over time using a combined dataset containing both training and pseudo-labeled data. Although not all of the generated pseudo-labels are applicable to every botnet, the self-training framework can label unseen botnets with behaviors similar to those of known botnets with high confidence. Several strategies are proposed for enhancing the robustness of the classification performance by minimizing the number of incorrect pseudo-labels, mitigating the effects of erroneous pseudo-labels on the overall performance of the network, and optimizing the proportion of unlabeled data for labeling. Experiments conducted on both synthetic datasets confirm the superiority of the proposed method over the base model, particularly when the training data constitutes only a small portion of the total amount dataset. Subsequent experiments also demonstrate the efficacy of the framework in successfully detecting unseen botnet variants and its commendable performance in real-world campus network traffic.

The paper is focuses on the methods that have come into practice, key functions and software instruments for collecting, analysis and visualization of network traffic statistics. The source of information is NetFlow telemetry data collected from network equipment. In addition to being used by network engineers and technicians, including for the purposes of network monitoring, incident handling, identification of network congestion and the main bandwidth utilizers with details on autonomous systems or IP addresses of sources and recipients, protocols, services and applications, NetFlow data is of interest in the context of monitoring and analysis of network interaction between users, service providers and consumers. The paper provides a detailed description of the developed and implemented on the basis of the new generation National Research Computer Network of Russia of the up-to-date and high-performance software solution for working with network telemetry data; specific examples of the capabilities are given in order to advanced analytics and descriptive data visualization in real time, taking into account the special needs of industry telecommunications networks in the field of research and education. [ABSTRACT FROM AUTHOR]

Network traffic classification is of fundamental importance to a wide range of network activities, such as security monitoring, accounting, quality of service and forecasting for long-term provisioning purposes. This task has been increasingly implemented using machine learning methods due to the inability of conventional approaches to accommodate the increasing use of encryption. However, the application of machine learning methods to network traffic classification based on sampled NetFlow data is poorly developed despite the fact that NetFlow is a widely extended monitoring solution routinely employed by network operators. This study addresses this issue by proposing a network traffic classification module using NetFlow data in conjunction with a deep neural network. The performance of the proposed classification module is demonstrated by its application to two real-world datasets, and an average classification accuracy of 95% is obtained for |$\sim $| 1.4 million test cases. Moreover, the performance of the proposed classifier is demonstrated to be superior to three other state-of-the-art classifiers. Accordingly, the proposed module represents a promising alternative for network traffic classification. [ABSTRACT FROM AUTHOR]

Abstract With the massive resources and strategies accessible to attackers, countering Denial of Service (DoS) attacks is getting increasingly difficult. One of these techniques is application-layer DoS. Due to these challenges, network security has become increasingly more challenging to ensure. Hypertext Transfer Protocol (HTTP), Domain Name Service (DNS), Simple Mail Transfer Protocol (SMTP), and other application protocols have had increased attacks over the past several years. It is common for application-layer attacks to concentrate on these protocols because attackers can exploit some weaknesses. Flood and “low and slow” attacks are examples of application-layer attacks. They target weaknesses in HTTP, the most extensively used application-layer protocol on the Internet. Our experiment proposes a generalized detection approach to identify features for application-layer DoS attacks that is not specific to a single slow DoS attack. We combine four application-layer DoS attack datasets: Slow Read, HTTP POST, Slowloris, and Apache Range Header. We perform a feature-scaling technique that applies a normalization filter to the combined dataset. We perform a feature extraction technique, Principal Component Analysis (PCA), on the combined dataset to reduce dimensionality. We examine ways to enhance machine learning techniques for detecting slow application-layer DoS attacks that employ these methodologies. The machine learners effectively identify multiple slow DoS attacks, according to our findings. The experiment shows that classifiers are good predictors when combined with our selected Netflow characteristics and feature selection techniques.

Intrusion detection systems (IDS) play a vital role in protecting networks from malicious attacks. Modern IDS use machine-learning or deep-learning models to deal with the diversity of attacks that malicious users may employ. However, effective machine-learning methods incur a considerable cost in both the pretraining stage and the online detection process itself. Accordingly, this study proposes a quantitative logarithmic transformation-based intrusion detection system (QLT-IDS) that uses a straightforward statistical approach to analyze network behavior. Compared with machine-learning or deep-learning-based IDS methods, the proposed system requires neither a time-consuming and expensive data collection and training process, nor a GPU-included device to achieve a real-time detection performance. Furthermore, the system can deal not only with North-South attacks, but also East-West attacks, which pose a significant risk in real-world operations. The effectiveness of the proposed system is evaluated for both real-world campus network traffic and simulated traffic. The results confirm that QLT-IDS is able to detect a wide range of malicious attacks with a high precision, even under high down-sampling rate of the NetFlow records.

Traffic analysis is a common question for most of the production systems in various segments of computer networks. Attacks, configuration mistakes, and other factors can cause network increased accessibility and as a result danger for data privacy. Analyzing network flow and their single packets can be helpful for anomalies detection. Well-known network equipment has predeveloped network flow monitoring software. “NetFlow” data collector software “Nfsen” is an open-source way to collect information from agents. Also “Nfsen” is designed for data sorting and dataset for instruction detection system preparation. Prepared data can be split into fragments for artificial intelligent learning and testing. As AI unit can be used multilayer perceptron developed in a python programming language. This paper focused on real-world traffic dataset collection and multilayer perceptron deployment for TCP flood traffic detection. Article in English. Perteklinių TCP sesijų sudarymo atakų analizavimas naudojant „NetFlow“ Santrauka Srauto analizė – vienas pagrindinių įrankių anomalijoms kompiuteriniame tinkle aptikti. Atakos, konfigūracijos klaidos gali padėti lengviau pasiekti kompiuterinį tinklą ir galiausiai padidinti duomenų saugumo pavojų. Duomenų perdavimo tinklo srauto ir pavienių paketų analizė gali būti naudojama anomalijoms aptikti. Daugelis įrangos gamintojų įdiegia į savo įrangą srauto stebėjimo įrankius. „NetFlow“ protokolu perduodamu srautų duomenų kolektorius „Nfsen“ yra atvirojo kodo programinė įranga, padedanti surinkti informaciją iš agentų. Taip pat „Nfsen“ yra suprojektuota duomenų rinkinio įsibrovimo aptikimo sistemoms paruošti. Paruoštas duomenų rinkinys gali būti padalytas siekiant apmokyti ir testuoti dirbtinio intelekto modelį. Intelektinės sistemos srautui klasifikuoti gali būti naudojamas daugiasluoksnis perceptronas. Šiame darbe siekiama išanalizuoti, kaip interneto tiekėjo tinkle aptikti TCP perteklinį srautą ir jį klasifikuoti. Reikšminiai žodžiai: „NetFlow“, tcpdump, TCP, paketas, ugniasienė, srautas, GRE, ataka.

The uses of machine learning (ML) technologies in the detection of network attacks have been proven to be effective when designed and evaluated using data samples originating from the same organisational network. However, it has been very challenging to design an ML-based detection system using heterogeneous network data samples originating from different sources and organisations. This is mainly due to privacy concerns and the lack of a universal format of datasets. In this paper, we propose a collaborative cyber threat intelligence sharing scheme to allow multiple organisations to join forces in the design, training, and evaluation of a robust ML-based network intrusion detection system. The threat intelligence sharing scheme utilises two critical aspects for its application; the availability of network data traffic in a common format to allow for the extraction of meaningful patterns across data sources and the adoption of a federated learning mechanism to avoid the necessity of sharing sensitive users’ information between organisations. As a result, each organisation benefits from the intelligence of other organisations while maintaining the privacy of its data internally. In this paper, the framework has been designed and evaluated using two key datasets in a NetFlow format known as NF-UNSW-NB15-v2 and NF-BoT-IoT-v2. In addition, two other common scenarios are considered in the evaluation process; a centralised training method where local data samples are directly shared with other organisations and a localised training method where no threat intelligence is shared. The results demonstrate the efficiency and effectiveness of the proposed framework by designing a universal ML model effectively classifying various benign and intrusive traffic types originating from multiple organisations without the need for inter-organisational data exchange. [ABSTRACT FROM AUTHOR]

With the massive resources and strategies accessible to attackers, countering Denial of Service (DoS) attacks is getting increasingly difficult. One of these techniques is application-layer DoS. Due to these challenges, network security has become increasingly more challenging to ensure. Hypertext Transfer Protocol (HTTP), Domain Name Service (DNS), Simple Mail Transfer Protocol (SMTP), and other application protocols have had increased attacks over the past several years. It is common for application-layer attacks to concentrate on these protocols because attackers can exploit some weaknesses. Flood and "low and slow" attacks are examples of application-layer attacks. They target weaknesses in HTTP, the most extensively used application-layer protocol on the Internet. Our experiment proposes a generalized detection approach to identify features for application-layer DoS attacks that is not specific to a single slow DoS attack. We combine four application-layer DoS attack datasets: Slow Read, HTTP POST, Slowloris, and Apache Range Header. We perform a feature-scaling technique that applies a normalization filter to the combined dataset. We perform a feature extraction technique, Principal Component Analysis (PCA), on the combined dataset to reduce dimensionality. We examine ways to enhance machine learning techniques for detecting slow application-layer DoS attacks that employ these methodologies. The machine learners effectively identify multiple slow DoS attacks, according to our findings. The experiment shows that classifiers are good predictors when combined with our selected Netflow characteristics and feature selection techniques. [ABSTRACT FROM AUTHOR]

Traffic analysis is a common question for most of the production systems in various segments of computer networks. Attacks, configuration mistakes, and other factors can cause network increased accessibility and as a result danger for data privacy. Analyzing network flow and their single packets can be helpful for anomalies detection. Well-known network equipment has predeveloped network flow monitoring software. "NetFlow" data collector software "Nfsen" is an open-source way to collect information from agents. Also "Nfsen" is designed for data sorting and dataset for instruction detection system preparation. Prepared data can be split into fragments for artificial intelligent learning and testing. As AI unit can be used multilayer perceptron developed in a python programming language. This paper focused on real-world traffic dataset collection and multilayer perceptron deployment for TCP flood traffic detection. [ABSTRACT FROM AUTHOR]

This paper presents an online anomaly detection system capable of handling operational network traffic of large networks (such as an ISP). We also aim for an effective and practical diagnosis of anomalies diagnosis to produce actionable intelligence that enables automated response. To achieve these objectives, we use the following approaches. (1) We model the status of the network by a stream of tensors where each tensor cell contains a time series. (2) We detect anomalous tensors at discrete time steps using an unsupervised tensor representation learning model. (3) We produce actionable intelligence by diagnosing anomaly detection results and identifying the abnormal time series that are the most likely causes of each anomaly in the tensor. (4) We further analyze the traffic corresponding to each anomalous time series by an innovative method that extracts and isolates the attack traffic. (5) We provide solutions for streaming data anomaly detection challenges such as large volume, high velocity, seasonality, and concept drift. We apply our approach to the complete test set of UGR data to show its practicality and effectiveness. Not only can we detect and isolate most of the labelled attack traffic, but we also identify many organic attack activities in the UGR data. Our results on the complete UGR dataset show high detection and isolation rates for the labelled attacks in the dataset. We also report on additional organic attacks we detected that were originally labelled as background in the dataset. Our analysis shows that the isolated background traffic represents interesting and potentially malicious behavior and can provide invaluable insight for cyber-threat researchers.

Abstract Illicit cryptocurrency mining has become one of the prevalent methods for monetization of computer security incidents. In this attack, victims’ computing resources are abused to mine cryptocurrency for the benefit of attackers. The most popular illicitly mined digital coin is Monero as it provides strong anonymity and is efficiently mined on CPUs.Illicit mining crucially relies on communication between compromised systems and remote mining pools using the de facto standard protocol Stratum. While prior research primarily focused on endpoint-based detection of in-browser mining, in this paper, we address network-based detection of cryptomining malware in general. We propose XMR-Ray, a machine learning detector using novel features based on reconstructing the Stratum protocol from raw NetFlow records. Our detector is trained offline using only mining traffic and does not require privacy-sensitive normal network traffic, which facilitates its adoption and integration.In our experiments, XMR-Ray attained 98.94% detection rate at 0.05% false alarm rate, outperforming the closest competitor. Our evaluation furthermore demonstrates that it reliably detects previously unseen mining pools, is robust against common obfuscation techniques such as encryption and proxies, and is applicable to mining in the browser or by compiled binaries. Finally, by deploying our detector in a large university network, we show its effectiveness in protecting real-world systems.

With the advent and realization of Software Defined Network (SDN) architecture, many organizations are now shifting towards this paradigm. SDN brings more control, higher scalability, and serene elasticity. The SDN spontaneously changes the network configuration according to the dynamic network requirements inside the constrained environments. Therefore, a monitoring system that can monitor the physical and virtual entities is needed to operate this type of network technology with high efficiency and proficiency. In this manuscript, we propose a real-time monitoring system for data collection and visualization that includes the Prometheus, node exporter, and Grafana. A node exporter is configured on the physical devices to collect the physical and virtual entities resources utilization logs. A real-time Prometheus database is configured to collect and store the data from all the exporters. Furthermore, the Grafana is affixed with Prometheus to visualize the current network status and device provisioning. A monitoring system is deployed on the physical infrastructure of the KOREN topology. Data collected by the monitoring system is further pre-processed and restructured into a dataset. A monitoring system is further enhanced by including machine learning techniques applied on the formatted datasets to identify the elephant flows. Additionally, a Random Forest is trained on our generated labeled datasets, and the classification models' performance are verified using accuracy metrics. [ABSTRACT FROM AUTHOR]

Web applications have become central in the digital landscape, providing users instant access to information and allowing businesses to expand their reach. Injection attacks, such as SQL injection (SQLi), are prominent attacks on web applications, given that most web applications integrate a database system. While there have been solutions proposed in the literature for SQLi attack detection using learning-based frameworks, the problem is often formulated as a binary, single-attack vector problem without considering the prioritization and prevention component of the attack. In this work, we propose a holistic solution, SQLR34P3R, that formulates the SQLi attack as a multi-class, multi-attack vector, prioritization, and prevention problem. For attack detection and classification, we gathered 457,233 samples of benign and malicious network traffic, as well as 70,023 samples that had SQLi and benign payloads. After evaluating several machine-learning-based algorithms, the hybrid CNN-LSTM models achieve an average F1-Score of 97% in web and network traffic filtering. Furthermore, by using CVEs of SQLi vulnerabilities, SQLR34P3R incorporates a novel risk analysis approach which reduces additional effort while maintaining reasonable coverage to assist businesses in allocating resources effectively by focusing on patching vulnerabilities with high exploitability. We also present an in-the-wild evaluation of the proposed solution by integrating SQLR34P3R into the pipeline of known vulnerable web applications such as Damn Vulnerable Web Application (DVWA) and Vulnado and via network traffic captured using Wireshark from SQLi DNS exfiltration conducted with SQLMap for real-time detection. Finally, we provide a comparative analysis with state-of-the-art SQLi attack detection and risk ratings solutions. [ABSTRACT FROM AUTHOR]

Network Intrusion Detection Systems (NIDSs) are important tools for the protection of computer networks against increasingly frequent and sophisticated cyber attacks. Recently, a lot of research effort has been dedicated to the development of Machine Learning (ML) based NIDSs. As in any ML-based application, the availability of high-quality datasets is critical for the training and evaluation of ML-based NIDS. One of the key problems with the currently available NIDS datasets is the lack of a standard feature set. The use of a unique and proprietary set of features for each of the publicly available datasets makes it virtually impossible to compare the performance of ML-based traffic classifiers on different datasets, and hence to evaluate the ability of these systems to generalise across different network scenarios. To address that limitation, this paper proposes and evaluates standard NIDS feature sets based on the NetFlow network meta-data collection protocol and system. We evaluate and compare two NetFlow-based feature set variants, a version with 12 features, and another one with 43 features. For our evaluation, we converted four widely used NIDS datasets (UNSW-NB15, BoT-IoT, ToN-IoT, CSE-CIC-IDS2018) into new variants with our proposed NetFlow based feature sets. Based on an Extra Tree classifier, we compared the classification performance of the NetFlow-based feature sets with the proprietary feature sets provided with the original datasets. While the smaller feature set cannot match the classification performance of the proprietary feature sets, the larger set with 43 NetFlow features, surprisingly achieves a consistently higher classification performance compared to the original feature set, which was tailored to each of the considered NIDS datasets. The proposed NetFlow-based NIDS feature set, together with four benchmark datasets, made available to the research community, allow a fair comparison of ML-based network traffic classifiers across different NIDS datasets. We believe that having a standard feature set is critical for allowing a more rigorous and thorough evaluation of ML-based NIDSs and that it can help bridge the gap between academic research and the practical deployment of such systems. [ABSTRACT FROM AUTHOR]

Illicit cryptocurrency mining has become one of the prevalent methods for monetization of computer security incidents. In this attack, victims' computing resources are abused to mine cryptocurrency for the benefit of attackers. The most popular illicitly mined digital coin is Monero as it provides strong anonymity and is efficiently mined on CPUs.Illicit mining crucially relies on communication between compromised systems and remote mining pools using the de facto standard protocol Stratum. While prior research primarily focused on endpoint-based detection of in-browser mining, in this paper, we address network-based detection of cryptomining malware in general. We propose XMR-Ray, a machine learning detector using novel features based on reconstructing the Stratum protocol from raw NetFlow records. Our detector is trained offline using only mining traffic and does not require privacy-sensitive normal network traffic, which facilitates its adoption and integration.In our experiments, XMR-Ray attained 98.94% detection rate at 0.05% false alarm rate, outperforming the closest competitor. Our evaluation furthermore demonstrates that it reliably detects previously unseen mining pools, is robust against common obfuscation techniques such as encryption and proxies, and is applicable to mining in the browser or by compiled binaries. Finally, by deploying our detector in a large university network, we show its effectiveness in protecting real-world systems. [ABSTRACT FROM AUTHOR]

Anomaly detection is the first step with a challenging task of securing a communication network, as the anomalies may indicate suspicious behaviors, attacks, network malfunctions, or failures. In this paper, we address the problem of not only detecting different anomalies, such as volume based (e.g., DDoS or Flash crowd) and spatial based (e.g., network scan), that arise simultaneously in the wild but also of attributing the anomalous point to a single-anomaly event causing it. Besides, we also tackle the problem of low-detection accuracy caused by the phenomenon of traffic drift. To this end, a novel adaptive profile-based anomaly detection scheme is proposed. More specifically, a more comprehensive metrics set is defined from the dimensions of temporal, spatial, category, and intensity to compose IP traffic behavior characteristic spectrum for fine-grained traffic characterization. Then, the digital signature matrix obtained by using the ant colony optimization (ACO) algorithm is applied to construct the baseline profile of the normal traffic behavior. Anomalous points are identified and analyzed by using confidence bands and a generic clustering technique, respectively. Finally, a lightweight updating strategy is applied to reduce the number of false positives. Real-world data of China Education Research Network backbone and synthetic data are collected to verify our proposal. The experimental results demonstrate that our approach provides a fine-grained behavior description ability and has significantly increased the detection accuracy compared with other similar alternatives.

The inherent features of software-defined networking (SDN) architecture revolutionize traditional network infrastructure and provide the opportunity for integrated and centralized network monitoring. One of the shortcomings of SDNs is related to its high vulnerability to distributed denial of service attacks and other similar ones. In this paper, a novel multi-stage modular approach is proposed for detecting and mitigating security anomalies in SDN environment (SADM-SDNC). The proposed approach uses NetFlow protocol for gathering information and generating dataset and information gain ratio in order to select the effective features. Also, the C-support vector classification algorithm with radial basis function kernel, and features of Floodlight controller for developing a structure with desirable performance were used in the proposed scheme. The experimental results demonstrate that the proposed approach performs better than other methods in terms of enhancing accuracy and detection rate, and reducing classification error and false alarm rate, which were measured as 99.67%, 99.26%, 0.33%, and 0.08% respectively. Finally, thanks to utilizing REST API and Static Entry Pusher technologies in the Floodlight controller, it makes it possible to disconnect any communications with the attacking factors and remove destructive users. [ABSTRACT FROM AUTHOR]

An analysis of a host behavior is an essential key for modern network management and security. A robust behavior profile enables the network managers to detect anomalies with high accuracy, predict the host behavior, or group host to clusters for better management. Hence, host profiling methods attract the interest of many researchers, and novel methods for host profiling are being introduced. However, these methods are frequently developed on preprocessed and small datasets. Therefore, they do not reflect the real-world artifacts of the host profiling, such as missing observations, temporal patterns, or variability in the profile characteristics in time. To provide the needed insight into the artifacts of host profiling in real-world settings, we present a study of the host behavior in a network conducted on a one-year-long real-world network dataset. In the study, we inspect the availability of the data for host profiling, identify the temporal patterns in host behavior, introduce a method for stable labeling of the hosts, and assess the variability of the host characteristics in the course of the year using the coefficient of variance. Moreover, we make the one-year dataset containing nine characteristics used for host behavior analysis available for public use and further research, including selected use cases representing host profiling caveats. We also share the record of analysis presented in the paper. [ABSTRACT FROM AUTHOR]

The implementation of a set of measures aimed to regular monitoring and analyzing the activity of users of the National Research Computer Network of Russia in the inter-network interaction, evaluation of the level of its involvement in joint research projects, the intensity of using the technological infrastructure of Russian and the world's national research and education networks are discussed in the paper. Information on the developed methods, hardware and software solutions based on NetFlow data and provided collecting, analysis and visualization of network traffic is given; some results of work in identifying and visualizing of data exchange and the main directions of network connectivity are presented and discussed. [ABSTRACT FROM AUTHOR]

We present a processing pipeline for flow-based traffic classification using a machine learning component leveraging Deep Neural Networks (DNNs). The system is trained to predict likely characteristics of real-world traffic flows from a campus network ahead of time, e.g., a flow’s throughput or duration. Training and evaluation of DNN models are continuously performed on a flow data stream collected from a university data center. Instead of the common binary classification into “mice” and “elephant” (throughput) or “short-term” and “long-term” (duration) flows, predicted flow characteristics are quantized into three classes. Various communication contexts (subset of network traffic, e.g., only TCP) and flow feature groups (subset of flow features, e.g., only a flow’s 5-tuple), which are supported through an enrichment strategy, are considered and investigated. An in-depth description of the data acquisition process, including preprocessing steps and anonymization used to protect sensitive information, is given. Additionally, we employ an accelerated variant of t-distributed Stochastic Neighbor Embedding (t-SNE) to visualize network traffic data. This enables the understanding of traffic characteristics and relations between communication flows at a glance. Furthermore, possible use-cases and a high-level architecture for flow-based routing scenarios utilizing the developed pipeline are proposed. [ABSTRACT FROM AUTHOR]

Summary: Nowadays, software‐defined networking (SDN) is regarded as the best solution for the centralized handling and monitoring of large networks. However, it should be noted that SDN architecture suffers from the same security issues, which are the case with common networks. As a case in point, one of the shortcomings of SDNs is related to its high vulnerability to distributed denial of service (DDoS) attacks and other similar ones. Indeed, anomaly detection systems have been considered to deal with these attacks. The challenges are related to designing these systems including gathering data, extracting effective features, and selecting the best model for anomaly detection. In this paper, a novel combined approach is proposed; this method uses NetFlow protocol for gathering information and generating dataset, information gain ratio (IGR), in order to select the effective and relevant features and ensemble learning scheme (Stacking) for developing a structure with desirable performance and efficiency for detecting anomaly in SDN environment. The results obtained from the experiments revealed that the proposed method performs better than other methods in terms of enhancing accuracy (AC) and detection rate (DR) and reducing classification error (CE) and false alarm rate (FAR). The AC, DR, CE, and FAR of the proposed model were measured as 99.92%, 99.83%, 0.08%, and 0.03%, respectively. Furthermore, the proposed method prevents the occurrence of excessive overload on the controller and OpenFlow. [ABSTRACT FROM AUTHOR]

The number of security breaches in the cyberspace is on the rise. This threat is met with intensive work in the intrusion detection research community. To keep the defensive mechanisms up to date and relevant, realistic network traffic datasets are needed. The use of flow-based data for machine-learning-based network intrusion detection is a promising direction for intrusion detection systems. However, many contemporary benchmark datasets do not contain features that are usable in the wild. The main contribution of this work is to cover the research gap related to identifying and investigating valuable features in the NetFlow schema that allow for effective, machine-learning-based network intrusion detection in the real world. To achieve this goal, several feature selection techniques have been applied on five flow-based network intrusion detection datasets, establishing an informative flow-based feature set. The authors’ experience with the deployment of this kind of system shows that to close the research-to-market gap, and to perform actual real-world application of machine-learning-based intrusion detection, a set of labeled data from the end-user has to be collected. This research aims at establishing the appropriate, minimal amount of data that is sufficient to effectively train machine learning algorithms in intrusion detection. The results show that a set of 10 features and a small amount of data is enough for the final model to perform very well.

This research attempts to introduce the production methodology of an anomaly detection dataset using ten desirable requirements. Subsequently, the article presents the produced dataset named UGRansome, created with up-to-date and modern network traffic (netflow), which represents cyclostationary patterns of normal and abnormal classes of threatening behaviours. It was discovered that the timestamp of various network attacks is inferior to one minute and this feature pattern was used to record the time taken by the threat to infiltrate a network node. The main asset of the proposed dataset is its implication in the detection of zero-day attacks and anomalies that have not been explored before and cannot be recognised by known threats signatures. For instance, the UDP Scan attack has been found to utilise the lowest netflow in the corpus, while the Razy utilises the highest one. In turn, the EDA2 and Globe malware are the most abnormal zero-day threats in the proposed dataset. These feature patterns are included in the corpus, but derived from two well-known datasets, namely, UGR’16 and ransomware that include real-life instances. The former incorporates cyclostationary patterns while the latter includes ransomware features. The UGRansome dataset was tested with cross-validation and compared to the KDD99 and NSL-KDD datasets to assess the performance of Ensemble Learning algorithms. False alarms have been minimized with a null empirical error during the experiment, which demonstrates that implementing the Random Forest algorithm applied to UGRansome can facilitate accurate results to enhance zero-day threats detection. Additionally, most zero-day threats such as Razy, Globe, EDA2, and TowerWeb are recognised as advanced persistent threats that are cyclostationary in nature and it is predicted that they will be using spamming and phishing for intrusion. Lastly, achieving the UGRansome balance was found to be NP-Hard due to real life-threatening classes that do not have a uniform distribution in terms of several instances.

This paper presents a method of identifying and classifying network anomalies using an artificial neural network for analyzing data gathered via Netflow protocol. Potential anomalies and their properties are described. We propose using a multilayer perceptron, trained with the backpropagation algorithm. We experiment both with datasets acquired from a real ISP monitoring system and with datasets modified to simulate the presence of anomalies; some Netflow records are modified to contain known patterns of several network attacks. We evaluate the viability of the approach by practical experimentation with various anomalies and iteration sizes.