Understanding Recommendation Systems: From SVD to Neural Collaborative Filtering

Introduction:

In today’s digital age, we’re constantly bombarded with choices. What movie should I watch next? Which book should I read? What product should I buy? Recommendation systems have become an integral part of our online experiences, helping us navigate through the vast sea of options. These systems analyze our past behaviors, preferences, and the behaviors of similar users to suggest items we might like.

In this blog post, we’ll dive deep into the world of recommendation systems. We’ll explore what they are, how they work, and examine three popular approaches: Singular Value Decomposition (SVD), Autoencoders (AE), and Neural Collaborative Filtering (NCF). Along the way, we’ll provide code examples to illustrate these concepts.

What is a Recommendation System?

A recommendation system is an information filtering system that predicts the preferences or ratings a user might give to an item. These systems are used in a variety of areas, including movie streaming services, e-commerce platforms, social media, and news aggregators.

The primary goal of a recommendation system is to provide personalized suggestions to users, enhancing their experience and potentially increasing engagement or sales for the platform.

How Does a Recommendation System Work?

Let’s consider a simple example to understand the basic principle behind recommendation systems.

Imagine we have a small movie rating database:

User Movie A Movie B Movie C Movie D Movie E Alice 5 4 3 ? 2 Bob 4 ? 3 5 1 Carol ? 2 4 4 ? David 3 4 ? 3 5

In this matrix:

• Each row represents a user
• Each column represents a movie
• The numbers represent the ratings given by users to movies
• The question marks (?) represent movies that users haven’t rated yet

The goal of a recommendation system is to predict these missing ratings. Once we have predictions for all the missing ratings, we can recommend the highest-rated unwatched movies to each user.

Different Approaches to Recommendations:

There are several approaches to building recommendation systems. In this blog, we’ll focus on three popular methods:

1. Singular Value Decomposition (SVD)
2. Autoencoders (AE)
3. Neural Collaborative Filtering (NCF)

Let’s start by examining SVD in detail.

Singular Value Decomposition (SVD):

SVD is a matrix factorization technique that is widely used in recommendation systems. It works by decomposing the user-item interaction matrix into the product of three matrices: U, Σ, and V^T.

Mathematically, for a matrix A, SVD is expressed as:

A = U Σ V^T

Where:

• U is an m x m orthogonal matrix
• Σ (Sigma) is an m x n diagonal matrix with non-negative real numbers on the diagonal
• V^T is the transpose of an n x n orthogonal matrix V

In the context of recommendation systems:

• A is our user-item interaction matrix
• U can be thought of as the “user features” matrix
• V can be thought of as the “item features” matrix
• Σ represents the strength of each latent factor

By reducing the dimensionality of Σ (keeping only the k largest singular values), we can approximate the original matrix and fill in the missing values.

Here’s a Python implementation of SVD for our recommendation system:

import numpy as np from sklearn.decomposition import TruncatedSVD

Our user-item matrix

ratings_matrix = np.array([ [5, 4, 3, 0, 2], [4, 0, 3, 5, 1], [0, 2, 4, 4, 0], [3, 4, 0, 3, 5] ])

Initialize TruncatedSVD

svd = TruncatedSVD(n_components=2, random_state=42)

Fit and transform the ratings matrix

user_features = svd.fit_transform(ratings_matrix) itemfeatures = svd.components

Reconstruct the matrix

predicted_ratings = np.dot(user_features, item_features)

print(“Original Ratings Matrix:”) print(ratings_matrix) print(“\nPredicted Ratings Matrix:”) print(np.round(predicted_ratings, 2))

In this code:

1. We start with our ratings matrix, where 0 represents missing ratings.
2. We use TruncatedSVD from scikit-learn to perform the decomposition. We set n_components=2, meaning we’ll use 2 latent factors.
3. We fit and transform the ratings matrix to get the user features.
4. The item features are stored in the components_ attribute of the SVD object.
5. We reconstruct the matrix by taking the dot product of user_features and item_features.

The output will show the original ratings matrix and the predicted ratings matrix. The predicted matrix will have values filled in for all the previously missing ratings.

Advantages of SVD:

• It can handle sparsity in the data well.
• It’s computationally efficient for large datasets.
• It can capture latent relationships in the data.

Disadvantages of SVD:

• It assumes a linear relationship between users and items.
• It may not capture complex, non-linear interactions.
• The resulting matrix is dense, which can be memory-intensive for very large datasets.

Autoencoders (AE) for Recommendation Systems:

Autoencoders are a type of neural network that aim to copy their inputs to their outputs. They work by compressing the input into a lower-dimensional code and then reconstructing the output from this representation. The network is forced to learn a compressed representation of the input, which can be useful for recommendation systems.

In the context of recommendation systems, autoencoders can be used to learn a compressed representation of user preferences or item characteristics. The network tries to reconstruct the user-item interaction matrix, filling in the missing values in the process.

Here’s how an autoencoder works for recommendations:

1. Input Layer: The user-item interaction vector for a user (or item).
2. Encoder: Compresses the input into a lower-dimensional representation.
3. Bottleneck Layer: The compressed representation of the input.
4. Decoder: Reconstructs the original input from the compressed representation.
5. Output Layer: The reconstructed user-item interaction vector, including predictions for missing ratings.

Let’s implement a simple autoencoder for our recommendation system:

import numpy as np import torch import torch.nn as nn import torch.optim as optim

Our user-item matrix

ratings_matrix = np.array([ [5, 4, 3, 0, 2], [4, 0, 3, 5, 1], [0, 2, 4, 4, 0], [3, 4, 0, 3, 5] ])

Normalize ratings to [0, 1] range

max_rating = 5.0 ratings_matrix_norm = ratings_matrix / max_rating

Convert to PyTorch tensor

ratings_tensor = torch.FloatTensor(ratings_matrix_norm)

Define the autoencoder architecture

class RecommenderAutoencoder(nn.Module): def init(self, input_dim, encoding_dim): super(RecommenderAutoencoder, self).init() self.encoder = nn.Sequential( nn.Linear(input_dim, 8), nn.ReLU(), nn.Linear(8, encoding_dim), nn.ReLU() ) self.decoder = nn.Sequential( nn.Linear(encoding_dim, 8), nn.ReLU(), nn.Linear(8, input_dim), nn.Sigmoid() # Ensure output is between 0 and 1 )

def forward(self, x): encoded = self.encoder(x) decoded = self.decoder(encoded) return decoded

Instantiate the model

model = RecommenderAutoencoder(input_dim=5, encoding_dim=3)

Define loss function and optimizer

criterion = nn.MSELoss() optimizer = optim.Adam(model.parameters(), lr=0.01)

Training loop

num_epochs = 5000 for epoch in range(num_epochs):

Forward pass

outputs = model(ratings_tensor) loss = criterion(outputs, ratings_tensor)

Backward pass and optimize

optimizer.zero_grad() loss.backward() optimizer.step()

if (epoch+1) % 1000 == 0: print(f’Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.4f}’)

Generate predictions

model.eval() with torch.no_grad(): predicted_ratings = model(ratings_tensor).numpy() * max_rating

print(“\nOriginal Ratings Matrix:”) print(ratings_matrix) print(“\nPredicted Ratings Matrix:”) print(np.round(predicted_ratings, 2))

In this implementation:

1. We normalize the ratings to the range [0, 1] to make it easier for the neural network to learn.
2. We define an autoencoder with an encoder that compresses the input to a 3-dimensional encoding, and a decoder that reconstructs the original input.
3. We use Mean Squared Error (MSE) as our loss function and Adam as our optimizer.
4. We train the model for 5000 epochs, printing the loss every 1000 epochs.
5. Finally, we use the trained model to generate predictions for all users and items.

Advantages of Autoencoders:

• Can capture non-linear relationships in the data.
• Can handle missing data naturally.
• Can potentially learn more complex patterns than linear methods like SVD.

Disadvantages of Autoencoders:

• May require more data to train effectively compared to simpler methods.
• Can be computationally intensive to train, especially for large datasets.
• The choice of architecture (number of layers, neurons per layer) can significantly impact performance.

Neural Collaborative Filtering (NCF):

Neural Collaborative Filtering is a framework that combines the linearity of matrix factorization with the non-linearity of neural networks. It aims to learn the complex user-item interactions using neural architectures.

The basic idea behind NCF is to learn separate embeddings for users and items, and then use these embeddings as input to a neural network that predicts the user’s preference for an item.

Here’s a simple implementation of NCF:

import numpy as np import torch import torch.nn as nn import torch.optim as optim

Our user-item matrix

ratings_matrix = np.array([ [5, 4, 3, 0, 2], [4, 0, 3, 5, 1], [0, 2, 4, 4, 0], [3, 4, 0, 3, 5] ])

num_users, num_items = ratings_matrix.shape

Create user-item pairs and corresponding ratings

user_item_pairs = [] ratings = [] for user in range(num_users): for item in range(num_items): if ratings_matrix[user, item] != 0: user_item_pairs.append([user, item]) ratings.append(ratings_matrix[user, item])

user_item_pairs = torch.LongTensor(user_item_pairs) ratings = torch.FloatTensor(ratings)

Define the NCF model

class NCF(nn.Module): def init(self, num_users, num_items, embedding_size, layers): super(NCF, self).init() self.user_embedding = nn.Embedding(num_users, embedding_size) self.item_embedding = nn.Embedding(num_items, embedding_size) self.fc_layers = nn.ModuleList() for i in range(len(layers) — 1): self.fc_layers.append(nn.Linear(layers[i], layers[i+1])) self.output_layer = nn.Linear(layers[-1], 1) self.activation = nn.ReLU()

def forward(self, user_indices, item_indices): user_embedded = self.user_embedding(user_indices) item_embedded = self.item_embedding(item_indices) x = torch.cat([user_embedded, item_embedded], dim=-1) for layer in self.fc_layers: x = self.activation(layer(x)) return self.output_layer(x).squeeze()

Instantiate the model

model = NCF(num_users, num_items, embedding_size=8, layers=[16, 8, 4])

Define loss function and optimizer

criterion = nn.MSELoss() optimizer = optim.Adam(model.parameters(), lr=0.01)

Training loop

num_epochs = 1000 for epoch in range(num_epochs):

Forward pass

outputs = model(user_item_pairs[:, 0], user_item_pairs[:, 1]) loss = criterion(outputs, ratings)

Backward pass and optimize

optimizer.zero_grad() loss.backward() optimizer.step()

if (epoch+1) % 100 == 0: print(f’Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.4f}’)

Generate predictions

model.eval() predicted_ratings = np.zeros_like(ratings_matrix) with torch.no_grad(): for user in range(num_users): for item in range(num_items): predicted_ratings[user, item] = model(torch.LongTensor([user]), torch.LongTensor([item])).item()

print(“\nOriginal Ratings Matrix:”) print(ratings_matrix) print(“\nPredicted Ratings Matrix:”) print(np.round(predicted_ratings, 2))

In this implementation:

1. We create user-item pairs and corresponding ratings from our ratings matrix.
2. We define an NCF model that learns embeddings for users and items, and then passes these through a series of fully connected layers.
3. We train the model using Mean Squared Error as our loss function and Adam as our optimizer.
4. Finally, we use the trained model to generate predictions for all user-item pairs.

Advantages of Neural Collaborative Filtering:

• Can capture both linear and non-linear relationships in the data.
• Flexible architecture that can be adapted to different types of input data.
• Can potentially outperform both traditional matrix factorization and pure neural network approaches.

Disadvantages of Neural Collaborative Filtering:

• May require more data and computational resources to train effectively.
• Can be prone to overfitting if not properly regularized.
• The choice of architecture and hyperparameters can significantly impact performance.

Results

Mean Squared Error Comparison: SVD: 2.0737 Autoencoder: 0.1820 Neural Collaborative Filtering: 0.6614

Overall Statistics: Original Ratings — Mean: 3.01, Std: 1.42 SVD Predictions — Mean: 1.74, Std: 1.13 Autoencoder Predictions — Mean: 2.85, Std: 1.61 NCF Predictions — Mean: 2.49, Std: 1.33

Conclusion:

In this blog post, we’ve explored three different approaches to building recommendation systems: Singular Value Decomposition (SVD), Autoencoders (AE), and Neural Collaborative Filtering (NCF). Each method has its own strengths and weaknesses, and the choice of which to use depends on the specific requirements of your recommendation task, the size and nature of your dataset, and the computational resources available.

SVD provides a simple and efficient method for matrix factorization, while Autoencoders and NCF offer more flexibility in modeling complex relationships between users and items. As we move from SVD to NCF, we generally see an increase in model complexity and potential performance, but also an increase in the amount of data and computational power required for effective training.

In practice, it’s often beneficial to experiment with multiple approaches and evaluate their performance on your specific dataset. Remember that the effectiveness of a recommendation system isn’t just about predictive accuracy — factors like diversity of recommendations, explanation of recommendations, and computational efficiency are also important considerations in many real-world applications.