Research on Collaborative Filtering Algorithm Based on Hadoop Architecture for Matrix Dimension Reduction in E-commerce Environment

Abstract

Abstract The rapid popularization and expansion of the Internet have catalyzed the growth of diverse e-commerce platforms. To mitigate information overload and enhance consumer shopping experiences, recommender systems have been developed. Our proposed algorithm, grounded in the Hadoop architecture, employs a refined cosine similarity method to calculate the average distance between users and rated items. This method involves the application of the Singular Value Decomposition (SVD) model to reduce the dimensionality of the user-item rating matrix, facilitating the extraction of item feature vectors. Subsequently, these vectors are clustered and segmented using the Matrix Factorization (MF) algorithm, addressing the challenge of data sparsity effectively. Experimental evaluations demonstrate that our enhanced algorithm outperforms five conventional collaborative filtering recommendation algorithms across varying matrix densities (from 0.05 to 0.25) on a public dataset. This results in a significant reduction in prediction error, thereby offering users more precise item recommendations.