NETWORK AND DATA LOCATION AWARE JOB SCHEDULING IN GRID: IMPROVEMENT TO GRIDWAY

Abstract

Grid technologies have enabled the sharing and aggregation of geographically distributed resources for solving large scale problems in the domain of science, engineering, commerce and business. Nowadays, Grid infrastructure constitutes the core of the computing facilities of High Energy Physics (HEP) experiments. These experiments produce and manage a large amount of data per day and. run thousands of computing jobs to process that data. The applications for these experiments require large data transfers over the network from data sources to computing resources. It is the duty of meta-scheduler to allocate jobs to most appropriate resources, and to use network in an efficient way. In this work, a Network and Data Location Aware job scheduling has been proposed for data intensive jobs. The proposed scheduling algorithm takes into account network characteristics, disk latency of data sources, and data location as well as other computational factors (CPU power, memory, CPU load, e.t.c) when making scheduling decisions. This scheduling algorithm aims to minimize not only file staging (data transfer) time but also turnaround time of the jobs.. The authors have improved the GridWay MetaScheduler with Network and Data Location Aware scheduling algorithm. The improved GridWay MetaScheduler has been tested for compute and data intensive jobs. Results presented here shows that the data transfer time and turnaround time of jobs are reduced when network characteristics, data locations of input files, and disk read speed (disk latency) of storage drive at data sources are considered in the jobs scheduling process