MapReduce and its open software implementation Hadoop are now widely deployed for big data analysis. As MapReduce runs over a cluster of massive machines, data transfer often becomes a bottleneck in job processing. In this paper, we explore the influence of data transfer to job processing performance and analyze the mechanism of job performance deterioration caused by data transfer oriented congestion at disk I/O and/or network I/O. Based on this analysis, we update Hadoop's Heartbeat messages to contain the real time system status for each machine, like disk I/O and link usage rate. This enhancement makes Hadoop's scheduler be aware of each machine's workload and make more accurate decision of scheduling. The experiment has been done to evaluate the effectiveness of enhanced scheduling methods and discussions are provided to compare the several proposed scheduling policies.

The current telecommunication network is structured in two layers: The intelligent layer that includes Intelligent Network (IN) nodes and Operation, Administration and Maintenance (OAM) nodes, and the transport layer that includes Network Elements (NEs). The transport layer carries user Information (Iu) from end-users as well as control and OAM Information (Ic&o) from IN/OAM nodes. The quick deployment of new IN services and OAM capabilities that will need (a) flexibility and easy management, and (b) an effective handling method for searching the huge amount of data among distributed databases, will be two requirements to be satisfied. Integrating various types of Ic&o into a unique Ic&o transport network and using ATM technique as a transport technique satisfies partly the requirement (a). To completely meet both requirements, this paper proposes the following solutions:(a) Intelligent layer connections and transport layer connections should be managed independently: The necessary mapping between the Logical Destination Address (LDA) that represents the logical address of the physical entity where data are routed, combined with the Quality Of Service (QOS) type, and the ATM connection IDentifier (ID), that is to say the Virtual Channel Identifier/ Virtual Path Identifier (VCI/VPI), is provided by specific nodes (the Ic&o network Management Nodes (Ic&o MNs)) belonging to an intermediate layer, i.e., the Ic&o network management layer.(b) The widely distributed aspect of the databases also needs a very effective data handling method. This paper proposes to implement a Distributed Directory System (DDS) into both intelligent nodes and Ic&o MNs.In order to apply the DDS function to 2 functional levels, the following items are studied: First, the possible mapping of DDS functions into the intelligent node functions is proposed. Second, this paper gives an interaction scenario between intelligent nodes and Ic&o MNs, to translate the LDA/QOS type into VPI/VCI. Finally, the analysis of the mapping of LDA/QOS type into VCI/VPI at the ATM level shows that the Ic&o network based on VP backbone offers the best compromise between flexibility, complexity and cost.

This paper evaluates the performance of TCP over ATM by simulation studies to clarify its applicability to high-speed WANs. We compared the performance of TCP over ABR with that of TCP over UBR, and TCP over UBR with Early Packet Discard (EPD). As for TCP over UBR, TCP has all responsibilities for end-to-end performance. In this case, cell loss at the ATM layer degrades TCP performance. Optimum tuning of TCP parameters may mitigate this degradation problem, but cannot solve it. Using EPD with UBR can fairly reduce useless transmission of corrupted packets and improve TCP performance, but still have the problem on fairness. As a result, TCP over ABR was proved to be the most effective as long as it suppressed cell loss. It was also proved that, if we want to extract best performance by TCP over ABR, we need to choose TCP parameters such as window size or timer granularity, so that ABR rate control does not interact with TCP window control and retransmission control.