A Distributed Recommendation Platform for Big Data

Daniel Valcarce (University of A Coruña, Spain)

Javier Parapar (University of A Coruña, Spain)

Álvaro Barreiro (University of A Coruña, Spain)

Abstract: The vast amount of information that recommenders manage these days has reached a point where scalability has become a critical factor. In this work, we propose a scalable architecture designed for computing Collaborative Filtering recommendations in a Big Data scenario. In order to build a highly scalable and fault-tolerant platform, we employ fully distributed systems without any single point of failure. We study the use of data replication and data distribution technologies. Additionally, we consider different caching techniques. Taking into account these requirements, we propose particular technologies for each component of the platform. Next, we evaluate the response times of storing, generating and serving recommendations using MySQL Cluster and Cassandra showing that the latter technology is much more adequate for that purpose. Finally, we conduct a simulation for evaluating the impact of a memory caching system.

Keywords: NoSQL, architecture, big data, cache, recommender systems, scalability

Categories: H.3.3