NDCouplingHDFS: A Coupling Architecture for a Power-Proportional Hadoop Distributed File System

Hieu Hanh LE; Satoshi HIKIDA; Haruo YOKOTA

doi:10.1587/transinf.E97.D.213

NDCouplingHDFS: A Coupling Architecture for a Power-Proportional Hadoop Distributed File System

Hieu Hanh LE, Satoshi HIKIDA, Haruo YOKOTA

Full Text Views

0

Cite this

Summary :

Energy-aware distributed file systems are increasingly moving toward power-proportional designs. However, current works have not considered the cost of updating data sets that were modified in a low-power mode, where a subset of nodes were powered off. In detail, when the system moves to a high-power mode, it must internally replicate the updated data to the reactivated nodes. Effectively reflecting the updated data is vital in making a distributed file system, such as the Hadoop Distributed File System (HDFS), power proportional. In the current HDFS design, when the system changes power mode, the block replication process is ineffectively restrained by a single NameNode because of access congestion of the metadata information of blocks. This paper presents a novel architecture, a NameNode and DataNode Coupling Hadoop Distributed File System (NDCouplingHDFS), which effectively reflects the updated blocks when the system goes into high-power mode. This is achieved by coupling metadata management and data management at each node to efficiently localize the range of blocks maintained by the metadata. Experiments using actual machines show that NDCouplingHDFS is able to significantly reduce the execution time required to move updated blocks by 46% relative to the normal HDFS. Moreover, NDCouplingHDFS is capable of increasing the throughput of the system supporting MapReduce by applying an index in metadata management.

Publication: IEICE TRANSACTIONS on Information Vol.E97-D No.2 pp.213-222

Publication Date: 2014/02/01

Publicized

Online ISSN: 1745-1361

DOI: 10.1587/transinf.E97.D.213

Type of Manuscript: PAPER

Category: Data Engineering, Web Information Systems

Authors

Hieu Hanh LE
  Tokyo Institute of Technology
Satoshi HIKIDA
  Tokyo Institute of Technology
Haruo YOKOTA
  Tokyo Institute of Technology

Keyword

power-proportionality, HDFS, metadata management

Cite this

Copy

Hieu Hanh LE, Satoshi HIKIDA, Haruo YOKOTA, "NDCouplingHDFS: A Coupling Architecture for a Power-Proportional Hadoop Distributed File System" in IEICE TRANSACTIONS on Information, vol. E97-D, no. 2, pp. 213-222, February 2014, doi: 10.1587/transinf.E97.D.213.
Abstract: Energy-aware distributed file systems are increasingly moving toward power-proportional designs. However, current works have not considered the cost of updating data sets that were modified in a low-power mode, where a subset of nodes were powered off. In detail, when the system moves to a high-power mode, it must internally replicate the updated data to the reactivated nodes. Effectively reflecting the updated data is vital in making a distributed file system, such as the Hadoop Distributed File System (HDFS), power proportional. In the current HDFS design, when the system changes power mode, the block replication process is ineffectively restrained by a single NameNode because of access congestion of the metadata information of blocks. This paper presents a novel architecture, a NameNode and DataNode Coupling Hadoop Distributed File System (NDCouplingHDFS), which effectively reflects the updated blocks when the system goes into high-power mode. This is achieved by coupling metadata management and data management at each node to efficiently localize the range of blocks maintained by the metadata. Experiments using actual machines show that NDCouplingHDFS is able to significantly reduce the execution time required to move updated blocks by 46% relative to the normal HDFS. Moreover, NDCouplingHDFS is capable of increasing the throughput of the system supporting MapReduce by applying an index in metadata management.
URL: https://global.ieice.org/en_transactions/information/10.1587/transinf.E97.D.213/_p

Copy

@ARTICLE{e97-d_2_213,
author={Hieu Hanh LE, Satoshi HIKIDA, Haruo YOKOTA, },
journal={IEICE TRANSACTIONS on Information},
title={NDCouplingHDFS: A Coupling Architecture for a Power-Proportional Hadoop Distributed File System},
year={2014},
volume={E97-D},
number={2},
pages={213-222},
abstract={Energy-aware distributed file systems are increasingly moving toward power-proportional designs. However, current works have not considered the cost of updating data sets that were modified in a low-power mode, where a subset of nodes were powered off. In detail, when the system moves to a high-power mode, it must internally replicate the updated data to the reactivated nodes. Effectively reflecting the updated data is vital in making a distributed file system, such as the Hadoop Distributed File System (HDFS), power proportional. In the current HDFS design, when the system changes power mode, the block replication process is ineffectively restrained by a single NameNode because of access congestion of the metadata information of blocks. This paper presents a novel architecture, a NameNode and DataNode Coupling Hadoop Distributed File System (NDCouplingHDFS), which effectively reflects the updated blocks when the system goes into high-power mode. This is achieved by coupling metadata management and data management at each node to efficiently localize the range of blocks maintained by the metadata. Experiments using actual machines show that NDCouplingHDFS is able to significantly reduce the execution time required to move updated blocks by 46% relative to the normal HDFS. Moreover, NDCouplingHDFS is capable of increasing the throughput of the system supporting MapReduce by applying an index in metadata management.},
keywords={},
doi={10.1587/transinf.E97.D.213},
ISSN={1745-1361},
month={February},}

Copy

TY - JOUR
TI - NDCouplingHDFS: A Coupling Architecture for a Power-Proportional Hadoop Distributed File System
T2 - IEICE TRANSACTIONS on Information
SP - 213
EP - 222
AU - Hieu Hanh LE
AU - Satoshi HIKIDA
AU - Haruo YOKOTA
PY - 2014
DO - 10.1587/transinf.E97.D.213
JO - IEICE TRANSACTIONS on Information
SN - 1745-1361
VL - E97-D
IS - 2
JA - IEICE TRANSACTIONS on Information
Y1 - February 2014
AB - Energy-aware distributed file systems are increasingly moving toward power-proportional designs. However, current works have not considered the cost of updating data sets that were modified in a low-power mode, where a subset of nodes were powered off. In detail, when the system moves to a high-power mode, it must internally replicate the updated data to the reactivated nodes. Effectively reflecting the updated data is vital in making a distributed file system, such as the Hadoop Distributed File System (HDFS), power proportional. In the current HDFS design, when the system changes power mode, the block replication process is ineffectively restrained by a single NameNode because of access congestion of the metadata information of blocks. This paper presents a novel architecture, a NameNode and DataNode Coupling Hadoop Distributed File System (NDCouplingHDFS), which effectively reflects the updated blocks when the system goes into high-power mode. This is achieved by coupling metadata management and data management at each node to efficiently localize the range of blocks maintained by the metadata. Experiments using actual machines show that NDCouplingHDFS is able to significantly reduce the execution time required to move updated blocks by 46% relative to the normal HDFS. Moreover, NDCouplingHDFS is capable of increasing the throughput of the system supporting MapReduce by applying an index in metadata management.
ER -