DATA MINING TECHNIQUES FOR HIGH DIMENSIONAL DATASETS

Abstract

Real world datasets are complex due to high dimensionality. In addition to many dimensions these datasets may have other problems like missing values, limited labels etc. Hence there is scope to explore data mining techniques for such complex high dimensional datasets. Clustering methods which find clusters in full dimensional space are not appropriate to find clusters in complex data with many dimensions due to problems associated with high dimensional data. In this work we proposed projected clustering methods based on optimization methods like Genetic Algorithm (GA) to find subspace clusters present in high dimensional datasets. The optimization technique finds optimal centers of subspace clusters by optimizing a subspace cluster validation index. Full dimensional clustering methods were used in literature in the pre-processing step to classification stage to solve problems associated with dataset. There is scope to explore usage of clustering methods which find clusters in high dimensional datasets in the pre-processing step to aid classification methods. In this work we proposed several classification methods based on this idea. We have proposed a framework for classification using projected clustering where subspace clusters found are used for s.everal kinds of pre-processing steps depending on the problem associated with the dataset. The proposed methods were applied on benchmark real and synthetic datasets.