SOURCE STATISTICS BASED MULTI-OBJECTIVE QUERY PROCESSING IN DATA INTEGRATION SYSTEM

Abstract

With the vast number of autonomous information sources available on the internet today, users have access to a large variety of data sources. To make query processing effective in dynamically changing data integration scenarios the integration system needs to be able to gather and use source coverage statistics [2] and source latency statistics of data sources. Given a query, the primary query processing objective in these systems is that of selecting the appropriate subset of sources so as to optimize various user objectives regarding the completeness and the response time. Unfortunately there are no effective approaches for learning the needed statistics for dynamically changing data integration system. Due to the lack of available source statistics, most existing integration frameworks [8] are unable to support flexible query processing that takes conflicting user objectives into account. Recent work in data integration systems [2] has shown the importance of statistical information about the coverage and overlap of data sources for efficient query processing. To solve the above problems, in this dissertation we implement a framework which gathers the required statistics. Data mining techniques are used to optimize the coverage and