We propose an algorithm for finding heavy hitters in terms of cardinality (the number of distinct items in a set) in massive traffic data using a small amount of memory. Examples of such cardinality heavy-hitters are hosts that send large numbers of flows, or hosts that communicate with large numbers of other hosts. Finding these hosts is crucial to the provision of good communication quality because they significantly affect the communications of other hosts via either malicious activities such as worm scans, spam distribution, or botnet control or normal activities such as being a member of a flash crowd or performing peer-to-peer (P2P) communication. To precisely determine the cardinality of a host we need tables of previously seen items for each host (e.g., flow tables for every host) and this may infeasible for a high-speed environment with a massive amount of traffic. In this paper, we use a cardinality estimation algorithm that does not require these tables but needs only a little information called the cardinality summary. This is made possible by relaxing the goal from exact counting to estimation of cardinality. In addition, we propose an algorithm that does not need to maintain the cardinality summary for each host, but only for partitioned addresses of a host. As a result, the required number of tables can be significantly decreased. We evaluated our algorithm using actual backbone traffic data to find the heavy-hitters in the number of flows and estimate the number of these flows. We found that while the accuracy degraded when estimating for hosts with few flows, the algorithm could accurately find the top-100 hosts in terms of the number of flows using a limited-sized memory. In addition, we found that the number of tables required to achieve a pre-defined accuracy increased logarithmically with respect to the total number of hosts, which indicates that our method is applicable for large traffic data for a very large number of hosts. We also introduce an application of our algorithm to anomaly detection. With actual traffic data, our method could successfully detect a sudden network scan.

This paper proposes a traffic-adaptive dynamic routing method, which we have named RAG, for connectionless packet networks. Conventional traffic control methods discard the packets which cause congestion. Furthermore, conventional routing methods propagate control messages all over the network for gathering global topology information, and this causes more congestion. In contrast, RAG estimates traffic conditions all over a network without any communication between nodes and makes the best use of free links so that packets make detours to avoid congestive sites. RAG adopts distributed control based on game theory (non-communication, non-zero-sum, two-person). With RAG, nodes play a packet-forwarding game without any communication with each other, and each node controls ordering and routing of the forwarding packets based on the node's individual payoff table which is dynamically reconstructed by observation of surrounding nodes. Nodes cooperate with each other, except for punishment for disloyalty. Repetition of these local operations in nodes aims at the emergence of the gradual network-global traffic balancing. The results of experiments in comparison with the conventional shortest path first (SPF) routing method show that the throughput is about 1.58 times higher with the new method.