Abstract:

Size-based scheduling with aging has been recognized as an effective approach to guarantee fairness and near optimal system response times.We present HFSP, a scheduler introducing this technique to a real, multi-server, complex and widely used system such as Hadoop.Size based scheduling on Hadoop is an effective approach to avoid starvation and minimize response time using hybrid scheduler.Size-based scheduling requires a priori job size information, which is not available in Hadoop.HFSP builds such knowledge by estimating it on-line during job execution.Size based scheduling in HFSP adopts the idea of giving priority to small jobs that they will not be slowed down by large ones.HFSP is a size based and preemptive scheduler for Hadoop. HFSP is largely fault tolerant and tolerant to job size estimation errors. The Scheduling decisions use the concept of virtual time and cluster resources are focused on jobs according to their priority, computed through aging. This protocol never faces Starvation Problem for small and large jobs.To minimize the response time the least remaining processing time will be calculated for each job.


Keywords — ;