Data parallelism is the dominant method used to train deep learning (DL) models on High-Performance Computing systems such as large-scale GPU clusters. When training a DL model on a large number of nodes, inter-node communication becomes bottle-neck due to its relatively higher latency and lower link bandwidth (than intra-node communication). Although some communication techniques have been proposed to cope with this problem, all of these approaches target to deal with the large message size issue while diminishing the effect of the limitation of the inter-node network. In this study, we investigate the benefit of increasing inter-node link bandwidth by using hybrid switching systems, i.e., Electrical Packet Switching and Optical Circuit Switching. We found that the typical data-transfer of synchronous data-parallelism training is long-lived and rarely changed that can be speed-up with optical switching. Simulation results on the Simgrid simulator show that our approach speed-up the training time of deep learning applications, especially in a large-scale manner.

In parallel computing systems, the interconnection network forms the critical infrastructure which enables robust and scalable communication between hundreds of thousands of nodes. The traditional packet-switched network tends to suffer from long communication time when network congestion occurs. In this context, we explore the use of circuit switching (CS) to replace packet switches with custom hardware that supports circuit-based switching efficiently with low latency. In our target CS network, a certain amount of bandwidth is guaranteed for each communication pair so that the network latency can be predictable when a limited number of node pairs exchange messages. The number of allocated time slots in every switch is a direct factor to affect the end-to-end latency, we thereby improve the slot utilization and develop a network topology generator to minimize the number of time slots optimized to target applications whose communication patterns are predictable. By a quantitative discrete-event simulation, we illustrate that the minimum necessary number of slots can be reduced to a small number in a generated topology by our design methodology while maintaining network cost 50% less than that in standard tori topologies.

We experimentally demonstrate an optical packet and circuit integrated (OPCI) ring network interoperated with a wavelength-switched optical network (WSON) in a network domain. OPCI network and WSON have distinct characteristics from each other: the methods to transfer path control messages and the protocols to set up or delete the optical connections in an optical circuit switch. To interoperate the two types of optical networks, we develop a common path control-plane which can establish or release an end-to-end path by only one autonomous distributed signaling process without stitching. In the common path control-plane, we modify the signaling protocol for OCS so that we can allocate a distinct wavelength to each link on an end-to-end path and also allocate a distinct path route to each of downstream and upstream directions in a bi-directional path. We experimentally show that the common path control-plane can dynamically establish end-to-end paths over the heterogeneous network including the two types of optical networks.

We propose a novel dynamic hierarchical optical path network architecture that achieves efficient optical fast circuit switching. In order to complete wavelength path setup/teardown efficiently, the proposed network adaptively manages waveband paths and bundles of optical paths, which provide virtual mesh connectivity between node pairs for wavelength paths. Numerical experiments show that operational and facility costs are significantly reduced by employing the adaptive virtual waveband connections.

This paper presents recent progress made in the development of an optical packet and circuit integrated network. From the viewpoint of end users, this is a single network that provides both high-speed, inexpensive services and deterministic-delay, low-data-loss services according to the users' usage scenario. From the viewpoint of network service providers, this network provides large switching capacity with low energy requirements, high flexibility, and efficient resource utilization with a simple control mechanism. The network we describe here will contribute to diversification of services, enhanced functional flexibility, and efficient energy consumption, which are included in the twelve design goals of Future Networks announced by ITU-T (International Telecommunication Union - Telecommunication Standardization Sector). We examine the waveband-based network architecture of the optical packet and circuit integrated network. Use of multi-wavelength optical packet increases the switch throughput while minimizing energy consumption. A rank accounting method provides a solution to the problem of inter-domain signaling for end-to-end lightpath establishment. Moving boundary control for packet and circuit services makes for efficient resource utilization. We also describe related advanced technologies such as waveband switching, elastic lightpaths, automatic locator numbering assignment, and biologically-inspired control of optical integrated network.

Integration of optical paths and packets in a switch is a key technique to support ultra-high-speed traffic in the future Internet. However, the question of how to efficiently allocate wavelengths for optical paths and optical packets has not been solved yet due to the lack of a systematic model to evaluate the performance of the integrated switch. In this paper, we model the operation of the integrated switch as a system of two queuing models: M/M/x/x for optical paths and M/M/1/LPS for optical packets. From the model, we find an optimal policy to dynamically allocate wavelength resources in an integrated switch. Simulation results demonstrate that our mechanism achieves better performance than other methods.

This paper presents a hierarchical NoC architecture to support GT (Guaranteed Throughput) signals to process multimedia data in embedded systems. The architecture provides a communication environment that meets the diverse conditions of communication constraints among IPs in power and area. With a system based on packet switching, which requires storage/control circuits to support GT signals, it is hard to satisfy design constraints in area, scalability and power consumption. This paper proposes a hierarchical 444 mesh-type NoC architecture based on circuit switching, which is capable of processing GT signals requiring high throughput. The proposed NoC architecture shows reduction in area by 50.2% and in power consumption by 57.4% compared with the conventional NoC architecture based on circuit switching. These figures amount to by 72.4% and by 86.1%, when compared with an NoC architecture based on packet switching. The proposed NoC architecture operates in the maximum throughput of 19.2 Gb/s.

The performance of parallel computing applications is highly dependent on the efficiency of the underlying communication operations. While often characterized as dynamic, these communication operations frequently exhibit spatial and temporal locality as well as regularity in structure. These characteristics can be exploited to improve communication performance if the correct prediction model is selected to a suitable communication topology. In this paper we describe an entropy based methodology for quantifying and evaluating the success of different prediction models on actual workloads drawn from representative parallel benchmarks. We evaluate two different prediction criteria and combinations thereof: (1) Messages are partitioned by source node. (2) Use of a first order context model. We also describe the threshold for predication designed to largely avoid incorrect predication overheads. Our results show for simple predication models, even on highly dynamic benchmark applications, predictability can be improved by several orders of magnitude. In fact, using simple prediction techniques, over 75% of the communication volume is accurately predictable.

The next generation mobile communications systems must support multimedia communications services as well as conventional voice service. DS-CDMA is regarded as the most promising candidate, because it is indispensable to cope with multimedia. The system capacity of DS-CDMA system is limited by the total interference level. As a result, in DS-CDMA systems many users suffer very poor communication quality if the total interference level exceeds this limit. Therefore, this paper considers smoothing interference fluctuation using the difference between voice and data in a type of QoS (quality of service). In other words, voice communication is suitable for a loss system because the quality of voice communication is delay-sensitive. On the other hand, data communication is suitable for a waiting system because the quality of data communication is non-delay-sensitive. This paper focuses on a system that applies a circuit switching method for voice traffic and a reservation type packet switching method for data traffic and proposes a data traffic control method. In this proposed data traffic control method, a base station controls data transmission from a mobile station to utilize unused voice traffic resources. As a result, the proposed method achieves highly efficient use of the radio spectra by smoothing interference fluctuation in DS-CDMA systems. This paper evaluates the performance level of the proposed method from a system capacity standpoint. It is shown that the proposed method achieves higher system capacity in voice/data integrated transmission.

A processor allocation scheme for mesh computers greatly affects their system utilization. The performance of an allocation scheme is largely dependent on its ability to detect available submeshes. We propose a new type of submesh, called a link-disjoint submesh, for processor allocation in mesh computers. This type of submesh increases the submesh recognition capability of an allocation scheme. A link-disjoint submesh is not a contiguous submesh as in the previous scheme, but this submesh still has no common communication link with any other submesh. When wormhole routing or circuit switching is used, the communication delay caused by non-contiguous processor allocation is minor. Through simulation, the performance of our scheme is measured and compared to the previous schemes in terms of such parameters as finish time and system utilization. It is shown through simulation that the link-disjoint submesh increases the performance of an allocation scheme.

A mechanism of an integrated switching system architecture where PS, CS, and ATM switching functions are integrated based on a hierarchical memory system concept is discussed. A packet buffering control mechanism, and practical random time-slot assignment mechanism for CS traffic, which are composed of multiple bearer rate data traffic are then described. The feasibility of the random time-slot assignment mechanism is also confirmed by a practical experimental system using VLSI technology, particularly, content addressable memory (CAM) technology. The required queuing delay between the nodes for the corresponding call set up procedure is also shown and its application is clarified. For practical digital networks that provide various types of data communications including voice, data, and video services, it is highly desirable to evaluate the transmission efficiency of integrating packet switching (PS) type non-real time traffic and circuit switching (CS) type real time traffic. Transmission line utilization improvement is expected when the random time-slot assignment and the movable boundary scheme on a TDM (Time Division Multiplexing) data frame are adopted. The corresponding control procedure by signaling between switching nodes is also examined.