IEICE TRANSACTIONS on Information
Hybrid Electrical/Optical Switch Architectures for Training Distributed Deep Learning in Large-Scale
Summary :
Data parallelism is the dominant method used to train deep learning (DL) models on High-Performance Computing systems such as large-scale GPU clusters. When training a DL model on a large number of nodes, inter-node communication becomes bottle-neck due to its relatively higher latency and lower link bandwidth (than intra-node communication). Although some communication techniques have been proposed to cope with this problem, all of these approaches target to deal with the large message size issue while diminishing the effect of the limitation of the inter-node network. In this study, we investigate the benefit of increasing inter-node link bandwidth by using hybrid switching systems, i.e., Electrical Packet Switching and Optical Circuit Switching. We found that the typical data-transfer of synchronous data-parallelism training is long-lived and rarely changed that can be speed-up with optical switching. Simulation results on the Simgrid simulator show that our approach speed-up the training time of deep learning applications, especially in a large-scale manner.
- Publication
- IEICE TRANSACTIONS on Information Vol.E104-D No.8 pp.1332-1339
- Publication Date
- 2021/08/01
- Publicized
- 2021/04/23
- Online ISSN
- 1745-1361
- DOI
- 10.1587/transinf.2020EDP7201
- Type of Manuscript
- PAPER
- Category
- Information Network
Authors
Thao-Nguyen TRUONG
National Institute of Advanced Industrial Science and Technology (AIST)
Ryousei TAKANO
National Institute of Advanced Industrial Science and Technology (AIST)
Keyword
Latest Issue
Copyrights notice of machine-translated contents
The copyright of the original papers published on this site belongs to IEICE. Unauthorized use of the original or translated papers is prohibited. See IEICE Provisions on Copyright for details.
Cite this
Copy
Thao-Nguyen TRUONG, Ryousei TAKANO, "Hybrid Electrical/Optical Switch Architectures for Training Distributed Deep Learning in Large-Scale" in IEICE TRANSACTIONS on Information,
vol. E104-D, no. 8, pp. 1332-1339, August 2021, doi: 10.1587/transinf.2020EDP7201.
Abstract: Data parallelism is the dominant method used to train deep learning (DL) models on High-Performance Computing systems such as large-scale GPU clusters. When training a DL model on a large number of nodes, inter-node communication becomes bottle-neck due to its relatively higher latency and lower link bandwidth (than intra-node communication). Although some communication techniques have been proposed to cope with this problem, all of these approaches target to deal with the large message size issue while diminishing the effect of the limitation of the inter-node network. In this study, we investigate the benefit of increasing inter-node link bandwidth by using hybrid switching systems, i.e., Electrical Packet Switching and Optical Circuit Switching. We found that the typical data-transfer of synchronous data-parallelism training is long-lived and rarely changed that can be speed-up with optical switching. Simulation results on the Simgrid simulator show that our approach speed-up the training time of deep learning applications, especially in a large-scale manner.
URL: https://global.ieice.org/en_transactions/information/10.1587/transinf.2020EDP7201/_p
Copy
@ARTICLE{e104-d_8_1332,
author={Thao-Nguyen TRUONG, Ryousei TAKANO, },
journal={IEICE TRANSACTIONS on Information},
title={Hybrid Electrical/Optical Switch Architectures for Training Distributed Deep Learning in Large-Scale},
year={2021},
volume={E104-D},
number={8},
pages={1332-1339},
abstract={Data parallelism is the dominant method used to train deep learning (DL) models on High-Performance Computing systems such as large-scale GPU clusters. When training a DL model on a large number of nodes, inter-node communication becomes bottle-neck due to its relatively higher latency and lower link bandwidth (than intra-node communication). Although some communication techniques have been proposed to cope with this problem, all of these approaches target to deal with the large message size issue while diminishing the effect of the limitation of the inter-node network. In this study, we investigate the benefit of increasing inter-node link bandwidth by using hybrid switching systems, i.e., Electrical Packet Switching and Optical Circuit Switching. We found that the typical data-transfer of synchronous data-parallelism training is long-lived and rarely changed that can be speed-up with optical switching. Simulation results on the Simgrid simulator show that our approach speed-up the training time of deep learning applications, especially in a large-scale manner.},
keywords={},
doi={10.1587/transinf.2020EDP7201},
ISSN={1745-1361},
month={August},}
Copy
TY - JOUR
TI - Hybrid Electrical/Optical Switch Architectures for Training Distributed Deep Learning in Large-Scale
T2 - IEICE TRANSACTIONS on Information
SP - 1332
EP - 1339
AU - Thao-Nguyen TRUONG
AU - Ryousei TAKANO
PY - 2021
DO - 10.1587/transinf.2020EDP7201
JO - IEICE TRANSACTIONS on Information
SN - 1745-1361
VL - E104-D
IS - 8
JA - IEICE TRANSACTIONS on Information
Y1 - August 2021
AB - Data parallelism is the dominant method used to train deep learning (DL) models on High-Performance Computing systems such as large-scale GPU clusters. When training a DL model on a large number of nodes, inter-node communication becomes bottle-neck due to its relatively higher latency and lower link bandwidth (than intra-node communication). Although some communication techniques have been proposed to cope with this problem, all of these approaches target to deal with the large message size issue while diminishing the effect of the limitation of the inter-node network. In this study, we investigate the benefit of increasing inter-node link bandwidth by using hybrid switching systems, i.e., Electrical Packet Switching and Optical Circuit Switching. We found that the typical data-transfer of synchronous data-parallelism training is long-lived and rarely changed that can be speed-up with optical switching. Simulation results on the Simgrid simulator show that our approach speed-up the training time of deep learning applications, especially in a large-scale manner.
ER -
English
