Classification of Micro-array Data in Apache Spark Framework

Abstract

Abstract Apache Spark is an emerging huge information analytics technology. Machine learning (ML) frameworks engineered on Spark are more ascendible compared with traditional ML frameworks. We tend to build SVMwithSGD(SVM with Stochastic Gradient Descent) and LinearRegressionWithSGD models by using Spark Python API (PySpark) to classify normal and tumor microarray samples. Microarray measures expression levels of thousands of genes in a very tissue or cell kind. Feature extraction and cross-validation are used to make sure effectiveness. The SVMwithSGD and LinearRegressionWithSGD models achieve associate degrees accuracies quite eightieths. This paper presents a study of feature selection methods effect, using a filter approach, on the accuracy and time consumed of supervised classification of cancer. A comparative evaluation among different selection methods: Principal Component Analysis (PCA), Independent Component Analysis (ICA) and Locally Linear Embedding (LLE) is carried out with SVMWithSGD or LogisticRegressionWithSGD classifier, using the datasets of prostate, cancer, lung and Huntington’s Disease samples. The classification results using SVMWithSGD and LogisticRegressionWithSGD (LGWithSGD) classifiers show that the SVMWithSGD classifier can present the highest accuracy and much time when compared with LGWithSGD. The results show that when we have classified with SVMWithSGD, PCA and SVMWithSGD is the best combination for analyzing the Borovecki, Gordon, and Chowdary datasets. While ICA and SVMWithSGD in the Singh and Chin datasets. Moreover, the results illustrate that when we have classified with LGWithSGD, PCA and LGWithSGD is the best combination for analyzing the Borovecki and Gordon datasets. While ICA and LGWithSGD in the Chowdary and Singh datasets. LLE and LGWithSGD is the best for analyzing Chin dataset.