DESIGN AND PERFORMANCE STUDY OF SCALABLE AND EFFICIENT PARALLEL ARCHITECTURE FOR IN-LINE DATA DE-DUPLICATION

Abstract

As data have been growing rapidly in data centers, data de-duplication, a form of compression, has received a broad attention from both academia and industry. Some researches focus on the approach by which more redundant data can be reduced and others investigate how to do data de-duplication at high speed. In - this report, we show the importance of data de-duplication . in the current, digital world and aim at reducing the time and space requirement for data de-duplication. Currently, data de-duplication storage systems continuously facing challenges - in providing the required scalability and capacities necessary to move -backup,-data within certain backup and recovery time limits. So, we are proposing a distributed architecture for in-line data dc-duplication with two data distribution strategies, in which, one node designated as server and multiple storage nodes. In architecture, we proposed an Intelligent Storage Balancing Strategy to distribute the data among the storage nodes - to improve- the de-duplication efficiency. All the nodes, including the server can do block level dc-duplication., in parallel. Proposed architecture can de-duplicate - with high- efficiency,. support de-duplication ratio comparable to that of a single system. In our thesis work, we are also proposed two solutions - for resource- reclamation purpose named as Block Referencing List and Disk-Indexing Method for removing the problems associated with reference counting technique: And in--the- last,iwe.-proposed a Hierarchical Indexing Mechanism for maintaining hash index in the storage nodes to reduce the searching time of hash signature of incoming. data block in the maintained -hash index.