Prediction of Lung Cancer Subtypes with Integrated Parallelized Clustering and Classification on Spark

Abstract

Data Mining techniques were used to mine unknown knowledge from Microarray gene expression data. Microarray gene data is huge in volume, so applying traditional data mining algorithms on Microarray gene data is time consuming. Microarray gene expression data has many genes and very few training instances. Feature selection helps to eliminate the unnecessary genes and selects only the prominent genes for diagnosing a disease. Parallel programming framework called Apache Spark was exploited to parallelize data mining algorithms. Spark is well suited for iterative processing. Parallelized computational methods namely feature selection, classification and integrated clustering with classification were used to reduce the time complexity and improve prediction accuracy. Chi-Square gene selection algorithm was used to select the optimal number of important genes. Clustering and classification algorithms were applied separately to the selected set of optimal genes and accuracy was calculated. Integrated clustering and classification algorithms were applied to selected set of optimal genes and accuracy was calculated. The accuracy obtained from clustering and classification applied separately was compared with the accuracy obtained from combined clustering and classification algorithms. Individual algorithms provided better accuracy than integrated algorithms for Lung Cancer data set.