This paper proposes a fast and scalable traffic monitoring system called Fast xFlow Proxy. For efficiently provisioning and operating networks, xFlow such as IPFIX and NetFlow is a promising technology for visualizing the detailed traffic matrix in a network. However, internet protocol (IP) packets in a large carrier network are encapsulated with various outer headers, e.g., layer 2 tunneling protocol (L2TP) or multi-protocol label switching (MPLS) labels. As native xFlow technologies are applied to the outer header, the desired inner information cannot be visualized. From this motivation, we propose Fast xFlow Proxy, which explores the complicated carrier's packet, extracts inner information properly, and relays the inner information to a general flow collector. Fast xFlow Proxy should be able to handle various packet processing operations possible (e.g., header analysis, header elimination, and statistics) at a wire rate. To realize the processing speed needed, we implement Fast xFlow Proxy using the data plane development kit (DPDK) and field-programmable gate array (FPGA). By optimizing deployment of processes between DPDK and FPGA, Fast xFlow Proxy achieves wire rate processing. From evaluations, we can achieve over 20 Gbps performance by using a single server and 100 Gbps performance by using scale-out architecture. We also show that this performance is sufficiently practical for monitoring a nationwide carrier network.

Traffic engineering refers to techniques to accommodate traffic efficiently by dynamically configuring traffic routes so as to adjust to changes in traffic. If traffic changes frequently and drastically, the interval of route reconfiguration should be short. However, with shorter intervals, obtaining traffic information is problematic. To calculate a suitable route, accurate traffic information of the whole network must be gathered. This is difficult in short intervals, owing to the overhead incurred to monitor and collect traffic information. In this paper, we propose a framework for traffic engineering in cases where only partial traffic information can be obtained in each time slot. The proposed framework is inspired by the human brain, and uses conditional probability to make decisions. In this framework, a controller is deployed to (1) obtain a limited amount of traffic information, (2) estimate and predict the probability distribution of the traffic, (3) configure routes considering the probability distribution of future predicted traffic, and (4) select traffic that should be monitored during the next period considering the system performance yielded by route reconfiguration. We evaluate our framework with a simulation. The results demonstrate that our framework improves the efficiency of traffic accommodation even when only partial traffic information is monitored during each time slot.

With the ever-growing prevalence of web 2.0, users can access information and resources easily and ubiquitously. It becomes increasingly important to understand the characteristics of user's complex behavior for efficient network management and security monitoring. In this paper, we develop a novel method to visualize and measure user's web-communication-behavior character in large-scale networks. First, we employ the active and passive monitoring methods to collect more than 20,000 IP addresses providing web services, which are divided into 12 types according to the content they provide, e.g. News, music, movie and etc, and then the IP address library is established with elements as (servicetype, IPaddress). User's behaviors are complex as they stay in multiple service types during any specific time period, we propose the behavior spectrum to model this kind of behavior characteristics in an easily understandable way. Secondly, two kinds of user's behavior characters are analyzed: the character at particular time instants and the dynamic changing characters among continuous time points. We then employ Renyi cross entropy to classify the users into different groups with the expectation that users in the same groups have similar behavior profiles. Finally, we demonstrated the application of behavior spectrum in profiling network traffic patterns and finding illegal users. The efficiency and correctness of the proposed methods are verified by the experimental results using the actual traffic traces collected from the Northwest Regional Center of China Education and Research Network (CERNET).

Critical infrastructures are falsely believed to be safe when they are isolated from the Internet. However, the recent appearance of Stuxnet demonstrated that isolated networks are no longer safe. We observe that a better intrusion detection scheme can be established based on the unique features of critical infrastructures. In this paper, we propose a whitelist-based detection system. Network and application-level whitelists are proposed, which are combined to form a novel cross-layer whitelist. Through experiments, we confirm that the proposed whitelists can exactly detect attack packets, which cannot be achieved by existing schemes.

Per-flow traffic measurement is essential for network management; billing, traffic engineering, mitigating denial of service attacks, to mention just a few. In this field, the fundamental problem is that the size of expensive SRAM is too small to hold traffic data from high-speed networks. In this paper, we propose a new method for per-flow traffic measurement, which is based on the virtual vector that was originally designed for the problem of spread estimation. We modify the original virtual vector and show that this simple change yields a highly effective per-flow traffic estimator. Experiments show that our proposed scheme outperforms the state-of-the-art method in terms of both processing time and space requirement.

A new network measurement primitive was recently proposed, known as long duration flows (LDF). LDF deserves special attention for network management and security monitoring. This kind of traffic appears periodically and persistently through a long period, but its total amount of traffic is not necessarily large. This feature makes detection difficult especially when the resources of detection system are limited or the detection should cover high-speed networks. In this paper, we propose a new lightweight data structure and streaming algorithm to detect such traffic.

We consider the problem of fast identification of high-rate flows in backbone links with possibly millions of flows. Accurate identification of high-rate flows is important for active queue management, traffic measurement and network security such as detection of distributed denial of service attacks. It is difficult to directly identify high-rate flows in backbone links because tracking the possible millions of flows needs correspondingly large high speed memories. To reduce the measurement overhead, the deterministic 1-out-of-k sampling technique is adopted which is also implemented in Cisco routers (NetFlow). Ideally, a high-rate flow identification method should have short identification time, low memory cost and processing cost. Most importantly, it should be able to specify the identification accuracy. We develop two such methods. The first method is based on fixed sample size test (FSST) which is able to identify high-rate flows with user-specified identification accuracy. However, since FSST has to record every sampled flow during the measurement period, it is not memory efficient. Therefore the second novel method based on truncated sequential probability ratio test (TSPRT) is proposed. Through sequential sampling, TSPRT is able to remove the low-rate flows and identify the high-rate flows at the early stage which can reduce the memory cost and identification time respectively. According to the way to determine the parameters in TSPRT, two versions of TSPRT are proposed: TSPRT-M which is suitable when low memory cost is preferred and TSPRT-T which is suitable when short identification time is preferred. The experimental results show that TSPRT requires less memory and identification time in identifying high-rate flows while satisfying the accuracy requirement as compared to previously proposed methods.

Distributed denial-of-service attacks on public servers have recently become more serious. More are SYN Flood attacks, since the malicious attackers can easily exploit the TCP specification to generate traffic making public servers unavailable. To assure that network services will not be interrupted, we need faster and more accurate defense mechanisms against malicious traffic, especially SYN Floods. One of the problems in detecting SYN Flood traffic is that server nodes or firewalls cannot distinguish the SYN packets of normal TCP connections from those of SYN Flood attack. Moreover, since the rate of normal network traffic may vary, we cannot use an explicit threshold of SYN arrival rates to detect SYN Flood traffic. In this paper we introduce a mechanism for detecting SYN Flood traffic more accurately by taking into consideration the time variation of arrival traffic. We first investigate the statistics of the arrival rates of both normal TCP SYN packets and SYN Flood attack packets. We then describe our new detection mechanism based on the statistics of SYN arrival rates. Our analytical results show that the arrival rate of normal TCP SYN packets can be modeled by a normal distribution and that our proposed mechanism can detect SYN Flood traffic quickly and accurately regardless of time variance of the traffic.

Various shadows are one of main factors that cause errors in vision based vehicle detection. In this paper, two simple methods, land mark based method and BS & Edge method, are proposed for vehicle detection and shadow rejection. In the experiments, the accuracy of vehicle detection is higher than 98%, during which the shadows arisen from roadside buildings grew considerably. Based on these two methods, vehicle counting, tracking, classification, and speed estimation are achieved so that real-time traffic parameters concerning traffic flow can be extracted to describe the load of each lane.

Internet communication is increasingly becoming an important element in daily life. Keeping this network safe from malicious elements is an urgent task for network management. To maintain the security level networks are generally, monitored for indications of usage with ill-intentions. Such indications are events which need to be collated, correlated and analyzed in real-time to be effective. However, on an average medium to large size network the number of such events are very large. This makes it practically impossible to analyze the information in real-time and provide the necessary security measures. In this paper, we propose a mechanism that keeps the number of events, to be analyzed, low thereby making it possible to provide ample security measures. We discuss a real-time Intrusion Detection System (IDS) for detecting network attacks. The system looks out for TCP ACK/RST packets, which are generally caused by network scans. The system can extract the tendency of network flows in real-time, based on the newly developed time-based clustering and Dynamic Access Tree creation techniques. The algorithm, implemented and deployed on a medium size backbone network using RMON (Remote MONitoring) technology, successfully detected 195 intrusion attempts during a one month period. The results of the pilot deployment are discussed. In this paper, the proposal, implementation and evaluation will be described.

Asynchronous Transfer Mode (ATM) is considered to bo the key technology for realizing B-ISDN. This paper discusses current research on ATM switching nodes for high-speed communication networks. Although some ATM switching nodes have been deployed, much work continues for resolving problems as regards operations and maintainability, such as ATM layer performance evaluation including layered management scheme upon detection of line failure, function test methods regarding channel connectivity for multicasting, and real-time ATM traffic-monitoring mechanism with QoS control. To achieve sufficient ATM node maintainability, the ATM cell transfer quality on the VP and VC levels should be ensured both within the ATM nodes and between adjacent ATM nodes. Since ATM switching nods handle many kinds of virtual paths and virtual channels, each channel's connectivity must be confirmed. This paper proposes ATM layer performance evaluation concept, layered management scheme upon detection of line failure, function test methods for a multicast switch using test cells that periodically pass through pre-determined switching path routes. It also proposes the concept of test cell generation for simulating multiplexed ATM test cells taking ATM truffic characteristics into account. Furthermore, this paper describes a fault diagnosis scheme using test cells that can continually observe the entire ATM connection length in the system. A real-time traffic monitoring hardware configuration and an interface with software control are also discussed and it is clarified that the required functions can be realized by using commercially available DSPs.

Advanced Traveller Information Systems (ATIS) and Advanced Traffic Management Systems (ATMS) require real-time traffic data to observe and control the trafic flow. Still, there is a lack of proficient traffic monitoring systems. One method to collect such data is using particular equipped vehicles, called probes, transmitting experienced travel times to base stations which in turn are connected to a traffic control center. In this paper we analyse the performance of a radio network for collecting real-time traffic data from probes. The results reveal that random transmission of traffic reports is a (spectrum) efficient, inexpensive and flexible method for collecting road traffic data that can provide reliable traffic monitoring.