Search

Traditional tools like NetFlow face great challenges as both the speed and the complexity of the network traffic increase. To keep the pace up, we propose HashFlow for more efficient and accurate collection of flow records. HashFlow keeps large flows in its main flow table and uses an ancillary table to summarize the other flows when the main table is full. With our flow collision resolution and flow record promotion schemes, a flow in the ancillary table is promoted back to the main flow table with a guaranteed probability when it becomes large enough. These operations can be performed highly efficiently, so HashFlow can keep up with ultra-high traffic speed. We implement HashFlow in a Tofino switch, and using traces from different operational networks, we compare its performance against some state-of-the-art flow measurement algorithms. Our experiments show that, for various types of traffic analysis applications, HashFlow consistently demonstrates clearly better performance than its competitors. For example, the performance of HashFlow in flow size estimation, flow size distribution estimation and heavy hitter detection is up to 21, 60 and 35 percent better than those of the best competitors respectively, and these merits of HashFlow come with almost no degradation of throughput.

[EN] SQL injections rank in the OWASP Top 3. The literature shows that analyzing network datagrams allows for detecting or preventing such attacks. Unfortunately, such detection usually implies studying all packets flowing in a computer network. Therefore, routers in charge of routing significant traffic loads usually cannot apply the solutions proposed in the literature. This work demonstrates that detecting SQL injection attacks on flow data from lightweight protocols is possible. For this purpose, we gathered two datasets collecting flow data from several SQL injection attacks on the most popular database engines. After evaluating several machine learning-based algorithms, we get a detection rate of over 97% with a false alarm rate of less than 0.07% with a Logistic Regression-based model. SI Instituto Nacional de Ciberseguridad de España (INCIBE) Universidad de León

Autonomous or self-driving networks are expected to provide a solution to the myriad of extremely demanding new applications in the Future Internet. The key to handle complexity is to perform tasks like network optimization and failure recovery with minimal human supervision. For this purpose, the community relies on the development of new Machine Learning (ML) models and techniques. However, ML can only be as good as the data it is fitted with. Datasets provided to the community as benchmarks for research purposes, which have a relevant impact in research findings and directions, are often assumed to be of good quality by default. In this paper, we show that relatively minor modifications on the same benchmark dataset (UGR’16, a flow-based real-traffic dataset for anomaly detection) cause significantly more impact on model performance than the specific ML technique considered. To understand this finding, we contribute a methodology to investigate the root causes for those differences, and to assess the quality of the data labelling. Our findings illustrate the need to devote more attention into (automatic) data quality assessment and optimization techniques in the context of autonomous networks., This work was supported by the Agencia Estatal de Investigaci´on in Spain, grant No PID2020-113462RB-I00, and the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 893146.

Illicit cryptocurrency mining has become one of the prevalent methods for monetization of computer security incidents. In this attack, victims’ computing resources are abused to mine cryptocurrency for the benefit of attackers. The most popular illicitly mined digital coin is Monero as it provides strong anonymity and is efficiently mined on CPUs.Illicit mining crucially relies on communication between compromised systems and remote mining pools using the de facto standard protocol Stratum. While prior research primarily focused on endpoint-based detection of in-browser mining, in this paper, we address network-based detection of cryptomining malware in general. We propose XMR-Ray, a machine learning detector using novel features based on reconstructing the Stratum protocol from raw NetFlow records. Our detector is trained offline using only mining traffic and does not require privacy-sensitive normal network traffic, which facilitates its adoption and integration.In our experiments, XMR-Ray attained 98.94% detection rate at 0.05% false alarm rate, outperforming the closest competitor. Our evaluation furthermore demonstrates that it reliably detects previously unseen mining pools, is robust against common obfuscation techniques such as encryption and proxies, and is applicable to mining in the browser or by compiled binaries. Finally, by deploying our detector in a large university network, we show its effectiveness in protecting real-world systems.

Network traces are considered a primary source of information to researchers, who use them to investigate research problems such as identifying user behavior, analyzing network hierarchy, maintaining network security, classifying packet flows, and much more. However, most organizations are reluctant to share their data with a third party or the public due to privacy concerns. Therefore, data anonymization prior to sharing becomes a convenient solution to both organizations and researchers. Although several anonymization algorithms are available, few of them allow sufficient privacy (organization need), acceptable data utility (researcher need), and efficient data analysis at the same time. This article introduces a condensation-based differential privacy anonymization approach that achieves an improved tradeoff between privacy and utility compared to existing techniques and produces anonymized network trace data that can be shared publicly without lowering its utility value. Our solution also does not incur extra computation overhead for the data analyzer. A prototype system has been implemented, and experiments have shown that the proposed approach preserves privacy and allows data analysis without revealing the original data even when injection attacks are launched against it. When anonymized datasets are given as input to graph-based intrusion detection techniques, they yield almost identical intrusion detection rates as the original datasets with only a negligible impact.

NetFlow is a network protocol system that is used to represent an overall summary of computer network conversations. A NetFlow record can convert previously captured packet captures or obtain NetFlow session data in real time. This research examines the use of machine-learning techniques to identify anomalies in NetFlow records and classify malware behavior for further investigation. The intent is to identify low-cost solutions leveraging open-source software capable of deployment on computer hardware of currently in-use data networks. This work seeks to determine whether expert selection of features can improve machine-learning detection algorithm performance and evaluate the trade-offs associated with eliminating redundant or excessive numbers of features. We identify the Random Forest algorithm as the strongest single algorithm across three of four metrics, with our chosen NetFlow features cutting the testing and training times in half while incurring minor reductions in two metrics. The experiment demonstrates that the chosen NetFlow features are sufficiently discriminative to detect attacks with a success rate higher than 94%. NCWDG Lieutenant, United States Navy Approved for public release. Distribution is unlimited.

DNS hijacking represents a security threat to users because it enables bypassing existing DNS security measures. Several malware families exploit this by changing the client DNS configuration to point to a malicious DNS resolver. Following the assumption that users will never actively choose to use a resolver that is not well-known, our paper introduces the idea of detecting client-based DNS hijacking by classifying public resolvers based on whether they are well-known or not. Furthermore, we propose to use NetFlow-based features to classify a resolver as well-known or malicious. By characterizing and manually labelling the 405 resolvers seen in four weeks of NetFlow data from a national ISP, we show that classification of both well-known and malicious servers can be made with an AUROC of 0.85.

Legacy industrial control systems (ICSs) are not designed to be exposed to the Internet and linking them to corporate networks has introduced a large number of cyber security vulnerabilities. Due to the distributed nature of ICS devices, a detection-in-depth strategy is required to simultaneously monitor the behaviour of multiple sources of ICS data. While a detection-in-depth method leads to detecting attacks, like flooding attacks in earlier phases before the attacker can reach the end target, most research papers have focused on anomaly detection based on a single source of ICS data. Here, we present a detection-in-depth method for an ICS network. The new method is called automated flooding attack detection (AFAD) which consists of three stages: data acquisition, pre-processing, and a flooding anomaly detector. Data acquisition includes data collection from different sources like programmable logic controller (PLC) logs and network traffic. We then generate NetFlow data to provide light-weight anomaly detection in ICS networks. NetFlow-based analysis has been used as a scalable method for anomaly detection in high-speed networks. It only analyses packet headers, and it is an efficient method for detecting flooding attacks like denial of service attacks, and its performance is not affected by encrypted data. However, it has not been sufficiently studied in industrial control systems. Besides NetFlow data, ICS device logs are a rich source of information that can be used to detect abnormal behaviour. Both NetFlow traffic and log data are processed in our pre-processing stage. The third stage of AFAD is anomaly detection which consists of two parallel machine learning analysis methods, which respectively analyse the behaviours of device logs and NetFlow records. Due to the lack of enough labelled training datasets in most environments, an unsupervised predictor and an unsupervised clustering method are respectively used in the anomaly detection stage. We validated our approach using traffic captured in a factory automation dataset, Modbus dataset, and SWAT dataset. These datasets contain physical and network level normal and abnormal data. The performance of AFAD was compared with single-layer anomaly detection and with other studies. Results showed the high precision of AFAD in detecting flooding attacks.

In recent years, several research works have proposed the analysis of network flow information using machine learning in order to detect threats or anomalous activities. In this sense, NetFlow-based systems stand out as one of the main sources of network flow information. In these systems, NetFlow collectors provide the flow monitoring information to be analyzed, but the particular information structure and format provided by different collector implementations is a recurring problem. In this paper, a new YANG data model is proposed as a standard model to use NetFlow-based monitoring data. In order to validate the proposal, a NetFlow collector incorporating the proposed NetFlow YANG model has been developed, to be integrated in a network scenario in which network flows are analyzed to detect malicious cryptomining activity. This collector extends an existing one, and provides design patterns to incorporate other existing collectors into this common data model. Our results show how, by using the YANG modeling language, network flow information can be handled and aggregated in a formal and unified way that provides flexibility and facilitates data analysis applied to threat detection.

Cloud computing has become the most needed technology for the IT industry. Impersonation attacks are among the most dangerous threats that Clouds face. In this paper, we present an approach to detect masquerade attacks in Clouds. The efficient detection of these attacks should correlate user behaviors in distinct environments and also should apply to several deployment models. We present and evaluate three approaches to detect Impersonation and masquerade attacks. The first approach analyzes sequences of correlated system calls from the VMs operating systems, while the second analyzes the NetFlow data from the network environment. The third approach integrates these two approaches by using a neural network that will produce better detections than any of the first two approaches. To simplify the testing and the evaluation of the three methods, the Cloud Intrusion Detection Dataset (CIDD) is used as a source for cloud audits data. The evaluation has considered alternative deployment models through our two intrusion detection frameworks, CIDS and CIDS-VIRT. The paper also shows that the proposed detection approaches are more accurate and outperform the SWAD-MMD, a recent masquerade detection framework that works in the cloud computing systems. Furthermore, the paper details our experimental results and evaluates the computational performance and the detection accuracy of these approaches.

Anomaly detection is an important method for monitoring network traffic where is important to successfully distinguish normal traffic from abnormal traffic. For this purpose, one could use the existing classification algorithms as a part of the machine learning (ML) process. In this paper, some of the classification algorithms (Stochastic Gradient Descent (SGD), Support Vector Machines (SVM), K-Nearest Neighbor (K-NN), Gaussian Naive Bayes (GNB), Decision Tree (DT), Random Forest (RF), AdaBoost (AB)) were tested on the public UNSW-NB15 dataset. Different encoding methods and ratios of training and test data resulted in the optimal parameters classifiers. Due to the imbalanced distribution of normal and abnormal network traffic data, both standard performance scores and additional classification performance scores (F2-score, Area Under ROC Curve (AUC)) were used, that better describe the obtained results. The RF Classifier with F2-score = 97.68% and AUC score = 98.47% obtained the best results using a representative subset within the original dataset due to the shorter duration of the computations. Features in the referential dataset were reduced by 82% and selected following the structure of the NetFlow data stream. Concerning similar studies, this paper compares several algorithms for anomaly detection and selects the best one for NetFlow data streams. The F2 and AUC metric is applied, which achieves very high accuracy compared to classic metrics that do not show realistic accuracy in imbalanced datasets. Less time was spent using Label enoding (LE) with the same accuracy compared to One-hot (OH) encoding used in similar research. The novelty introduced by this paper is in the optimization of the ML process and influence of the ratio of data for learning and testing, different encoding methods of categorical features, and feature reduction on the Netflow data streams

Network log data is significant for network administrators, since it contains information on every event that occurs in a network, including system errors, alerts, and packets sending statuses. Effectively analyzing large volumes of diverse log data brings opportunities to identify issues before they become problems and to prevent future cyberattacks; however, processing of the diverse NetFlow data poses challenges such as volume, velocity, and veracity of log data. In this study, by means of Elasticsearch, Logstash, and Kibana, i.e., the ELK Stack, we construct an analysis and management system for network log data, which provides functions to filter, analyze, and display network log data for further applications and creates data visualization on a Web browser. In addition, an advanced cyberattack detection model is facilitated using deep neural network (DNN), recurrent neural networks (RNN), and long short-term memory (LSTM) approaches. By knowing cyberattack behaviors and cross-validating with the log analysis system, one can learn from this model the characteristics of a variety of cyberattacks. Finally, we also implement Grafana to perform metrics monitoring.

An analysis of a host behavior is an essential key for modern network management and security. A robust behavior profile enables the network managers to detect anomalies with high accuracy, predict the host behavior, or group host to clusters for better management. Hence, host profiling methods attract the interest of many researchers, and novel methods for host profiling are being introduced. However, these methods are frequently developed on preprocessed and small datasets. Therefore, they do not reflect the real-world artifacts of the host profiling, such as missing observations, temporal patterns, or variability in the profile characteristics in time. To provide the needed insight into the artifacts of host profiling in real-world settings, we present a study of the host behavior in a network conducted on a one-year-long real-world network dataset. In the study, we inspect the availability of the data for host profiling, identify the temporal patterns in host behavior, introduce a method for stable labeling of the hosts, and assess the variability of the host characteristics in the course of the year using the coefficient of variance. Moreover, we make the one-year dataset containing nine characteristics used for host behavior analysis available for public use and further research, including selected use cases representing host profiling caveats. We also share the record of analysis presented in the paper.

The continuing development of 5G technology increases the number of devices connected to the internet, this provides an increasing potential for cybercriminals to orchestrate detrimental Distributed Denial of Service (DDoS) attacks. The research community continues to develop new techniques to respond to the growing demand for DDoS mitigation. The internet service provider (ISP) provides internet access for users, so the attack traffic arrives at this location before reaching the victim. Deploying the mitigation system within the ISP domain offers an efficient solution. Therefore, we propose a dynamic network traffic managing (DNTM) system, which encompasses an Attack Detector, an IP Prioritiser, a Traffic Manager, and a Netflow Classifier, for the ISP. The IP prioritiser categorises IP addresses into normal and suspicious classes. The Traffic Manager makes use of the existing ISP mechanisms including ingress & egress filtering, rate limiting, blackholing and normal routing to take different mitigation actions. The Netflow Classifier is a hybrid ensemble model that utilises both unsupervised and supervised learning techniques. The classifier employs two self-organising maps (SOMs) to label data to train a supervised ensemble unit, which includes Random Forests, Decision Trees, and Gradient Boosted Trees (SRDG), to get the final classification. The Netflow Classifier achieved over 96% average on recall, precision and F1 score on UDP flood, ICMP flood and TCP flood attack data sets.

Part 2: Connection Security; International audience; This paper presents a novel approach to detect SSH brute-force (BF) attacks in high-speed networks. Contrary to host-based approaches, we focus on network traffic analysis to identify attackers. Recent papers describe how to detect BF attacks using pure NetFlow data. However, our evaluation shows significant false-positive (FP) results of the current solution. To overcome the issue of high FP rate, we propose a machine learning (ML) approach to detection using specially extended IP Flows. The contributions of this paper are a new dataset from real environment, experimentally selected ML method, which performs with high accuracy and low FP rate, and an architecture of the detection system. The dataset for training was created using extensive evaluation of captured real traffic, manually prepared legitimate SSH traffic with characteristics similar to BF attacks, and, finally, using a packet trace with SSH logs from real production servers.

Botnet is one of the most dangerous cyber-security issues. The botnet infects unprotected machines and keeps track of the communication with the command and control server to send and receive malicious commands. The attacker uses botnet to initiate dangerous attacks such as DDoS, fishing, data stealing, and spamming. The size of the botnet is usually very large, and millions of infected hosts may belong to it. In this paper, we addressed the problem of botnet detection based on network’s flows records and activities in the host. Thus, we propose a general technique capable of detecting new botnets in early phase. Our technique is implemented in both sides: host side and network side. The botnet communication traffic we are interested in includes HTTP, P2P, IRC, and DNS using IP fluxing. HANABot algorithm is proposed to preprocess and extract features to distinguish the botnet behavior from the legitimate behavior. We evaluate our solution using a collection of real datasets (malicious and legitimate). Our experiment shows a high level of accuracy and a low false positive rate. Furthermore, a comparison between some existing approaches was given, focusing on specific features and performance. The proposed technique outperforms some of the presented approaches in terms of accurately detecting botnet flow records within Netflow traces.

Figuring the network’s hidden abnormal behavior can reduce network vulnerability. This paper presents a detailed architecture in which the collected log data of the network can be processed and analyzed. We process and integrate on-campus network information from every router and store the integrated NetFlow log data. Ceph is used as an open-source distributed storage platform that offers high efficiency, high reliability, scalability, and preliminary preprocessing of raw data with Python, removing redundant areas and unification. In the subanalysis, we discover the anomaly event and absolute flow by three times of standard deviation rule. Keras has been used to classify in-time data collected via a cyber-attack and to construct an automatic identifier template through the Recurring Neural Network (RNN) test. The identification accuracy of the optimization model is around 98% in attack detection. Finally, in the MySQL server, the results of the real-time evaluation can be obtained, and the results of the assessment can be displayed via ECharts.

Netflow log files commonly contain massive transfer records in tiny time interval, making analytical works complex and burdensome. By combining human cognition abilities with computerized techniques, visual analytics systems have become efficient tools for showing network states and locating abnormal behaviors. However, traditional visual analytics systems tend to be designed for solving certain problems and unable to synthesize various types of data sources. Despite recent advances in network security visualization, academia still starves for a proper solution to visualize IPv4 address behavior modes and IPv4 connection patterns within limited drawing space. Thus, we propose a visual analytics system called `Owleyes' which reprocesses Netflow log data with simple statistical operations in basic dimensions and fulfills the aforementioned requirements with proper novel graphs such as `sunburst-hive-plot graph' (SHG) and link-wheel graph (LW). The SHG provides a stable and comparable means of visualizing connection patterns efficiently in a limited drawing space. The LW represents the hourly connection counts of main ports in a specific IPv4 connection during one day. With the use case dealing with the ChinaVis 2016 Challenge I data, the efficiency and practicability of Owleyes are demonstrated.

DNS resolvers perform the essential role of translating domain names into IP addresses. The default DNS resolver offered by an Internet Service Provider (ISP) can be undesirable for a number of reasons such as censorship, lack of malware filtering options and low service quality. In this paper, we propose a novel method for estimating the amount of DNS traffic directed at non-ISP resolvers by using DNS and NetFlow data from an ISP. This method is extended to also estimate the amount of DNS traffic towards resolvers that offer malware filtering or parental control functionality. Finally, we propose a novel method for estimating the amount of DNS traffic at non-ISP resolvers that would have been censored by ISP resolvers. The results of applying these methods on an ISP dataset shows to which extent 3rd party resolvers are chosen by users for either malware filtering or censorship circumvention purposes. DNS resolvers perform the essential role of translating domain names into IP addresses. The default DNS resolver offered by an Internet Service Provider (ISP) can be undesirable for a number of reasons such as censorship, lack of malware filtering options and low service quality. In this paper, we propose a novel method for estimating the amount of DNS traffic directed at non-ISP resolvers by using DNS and NetFlow data from an ISP. This method is extended to also estimate the amount of DNS traffic towards resolvers that offer malware filtering or parental control functionality. Finally, we propose a novel method for estimating the amount of DNS traffic at non-ISP resolvers that would have been censored by ISP resolvers. The results of applying these methods on an ISP dataset shows to which extent 3rd party resolvers are chosen by users for either malware filtering or censorship circumvention purposes.

To prevent user exposure to a wide range of cyber security threats, organizations and companies often resort to deploying blacklists in DNS resolvers or DNS firewalls. The impact of such a deployment is often measured by comparing the coverage of individual blacklists, by counting the number of blocked DNS requests, or by counting the number of flows redirected to a benign web page that contains a warning to the user. This paper suggests an alternative to this by using NetFlow data to measure the effect of a DNS-based blacklist deployment. Our findings suggest that only 38-40% of blacklisted flows are web traffic. Furthermore, the paper analyzes the flows blacklisted by IP address, and it is shown that the majority of these are potentially benign, such as flows towards a web server hosting both benign and malicious sites. Finally, the flows blacklisted by domain name are categorized as either spam or malware, and it is shown that less than 6% are considered malicious. To prevent user exposure to a wide range of cyber security threats, organizations and companies often resort to deploying blacklists in DNS resolvers or DNS firewalls. The impact of such a deployment is often measured by comparing the coverage of individual blacklists, by counting the number of blocked DNS requests, or by counting the number of flows redirected to a benign web page that contains a warning to the user. This paper suggests an alternative to this by using NetFlow data to measure the effect of a DNS-based blacklist deployment. Our findings suggest that only 38-40% of blacklisted flows are web traffic. Furthermore, the paper analyzes the flows blacklisted by IP address, and it is shown that the majority of these are potentially benign, such as flows towards a web server hosting both benign and malicious sites. Finally, the flows blacklisted by domain name are categorized as either spam or malware, and it is shown that less than 6% are considered malicious.

The number of security breaches in the cyberspace is on the rise. This threat is met with intensive work in the intrusion detection research community. To keep the defensive mechanisms up to date and relevant, realistic network traffic datasets are needed. The use of flow-based data for machine-learning-based network intrusion detection is a promising direction for intrusion detection systems. However, many contemporary benchmark datasets do not contain features that are usable in the wild. The main contribution of this work is to cover the research gap related to identifying and investigating valuable features in the NetFlow schema that allow for effective, machine-learning-based network intrusion detection in the real world. To achieve this goal, several feature selection techniques have been applied on five flow-based network intrusion detection datasets, establishing an informative flow-based feature set. The authors’ experience with the deployment of this kind of system shows that to close the research-to-market gap, and to perform actual real-world application of machine-learning-based intrusion detection, a set of labeled data from the end-user has to be collected. This research aims at establishing the appropriate, minimal amount of data that is sufficient to effectively train machine learning algorithms in intrusion detection. The results show that a set of 10 features and a small amount of data is enough for the final model to perform very well.

The article considers network traffic as an aggregation of information flows from typical information objects, which can be apartment buildings, student campuses, office centers, sports and entertainment facilities of general use, etc. The study showed that the amount of information transferred by the TCP protocol significantly exceeds (by orders of magnitude) the amount of information transferred by the UDP protocol, which illustrates the average daily traffic graph of the amount of information transferred by the TCP and UDP protocols. Methods of traffic analysis are considered, the most significant characteristics of traffic are determined.

A log management system is essential for the networks administrator. With a log management tool, we can collect, store, analyze, archive, and finally dispose of the log information. In this paper, we propose the architecture model of a log management system using ELK Stack with Ceph to provide a safe network, good Wi-Fi signal strength, and adequate backup data mechanism. In this case, we use our campus data of Wi-Fi log and NetFlow log. First, we collect and store data of our Wi-Fi log using Filebeats tool, and then, we use Elasticsearch, Logstash, and Kibana Stack to visualize the Wi-Fi log data. Second, we collect and store our NetFlow log using NFDUMP, and then, we also use ELK Stack to visualize the NetFlow log data. Third, we integrate the Wi-Fi log and NetFlow log data in one architecture using a distributed storage Ceph file system (CephFS). Moreover, we also compare the performance of RADOS Gateway and CephFS for better storage mechanism.

Complex science workflows involve very large data demands and resource-intensive computations. These demands need reliable high-speed networks, that can optimize performance for application data flows. Characterizing flows into large flows (elephant) versus small flows (mice) can allow networks to optimize performance by detecting and handling demands in real-time. However, predicting elephant versus mice flows is extremely difficult as their definition varies based on networks. Machine learning techniques can help classify flows into two distinct clusters to identify characteristics of transfers. In this paper, we investigate unsupervised and semi-supervised machine learning approaches to classify flows in real time. We develop a Gaussian Mixture Model combined with an initialization algorithm, to develop a novel general-purpose method to help classification based on network sites (in terms of data transfers, flow rates and durations). Our results show that the proposed algorithm is able to cluster elephants and mice with an accuracy rate of 90%. We analyzed NetFlow reports of 1 month from 3 ESnet site routers to train the model and predict clusters.