The search functionality is under construction.
The search functionality is under construction.

Keyword Search Result

[Keyword] job performance(1hit)

1-1hit
  • System Status Aware Hadoop Scheduling Methods for Job Performance Improvement

    Masatoshi KAWARASAKI  Hyuma WATANABE  

     
    PAPER-Fundamentals of Information Systems

      Pubricized:
    2015/03/26
      Vol:
    E98-D No:7
      Page(s):
    1275-1285

    MapReduce and its open software implementation Hadoop are now widely deployed for big data analysis. As MapReduce runs over a cluster of massive machines, data transfer often becomes a bottleneck in job processing. In this paper, we explore the influence of data transfer to job processing performance and analyze the mechanism of job performance deterioration caused by data transfer oriented congestion at disk I/O and/or network I/O. Based on this analysis, we update Hadoop's Heartbeat messages to contain the real time system status for each machine, like disk I/O and link usage rate. This enhancement makes Hadoop's scheduler be aware of each machine's workload and make more accurate decision of scheduling. The experiment has been done to evaluate the effectiveness of enhanced scheduling methods and discussions are provided to compare the several proposed scheduling policies.