Neural Movie Recommenders with Small & Large MovieLens Data

ML Project #4

In this post, we will walk through a project that builds neural-network-based movie recommenders on MovieLens data: one using the 100K rating set and one using the full ~33M rating set. Both implement latent‐factor models (using Keras embeddings and dense layers) to predict explicit ratings, but they differ in scale and engineering.

We will discuss the project goals, data schemas, preprocessing, model architectures (embeddings followed by multi-layer perceptron), training regimes, evaluation metrics (rating prediction and top‐K recommendations), and deployment in detail. We will also consider the trade‐offs of small vs large data (memory, speed, sampling strategies, batch feeding).

We will be explaining the project which is spread over two notebooks available in the following github repo:

[embed]GitHub - pjc-codes/movie-lens-recsys An end-to-end neural collaborative filtering pipeline for movie recommendations. - pjc-codes/movie-lens-recsysgithub.com

The notebooks are:

• movielens-small-nn-recsys.ipynb for the MovieLens small dataset (100k+ ratings).
• movielens-large-nn-recsys.ipynb for the MovieLens large dataset (33M+ ratings).

Let’s get into the weeds of it now.

Project Goals & Environment

We aim to demonstrate a neural collaborative-filtering (CF) recommender. The 100K dataset is a toy example; the 33M dataset scales the same idea to a more “realistic” and “full” dataset. Both solve the explicit-feedback recommendation problem: predict a user’s movie rating (in the range 0.5–5 stars). In CF terms, this is classic matrix-factorization/regression on a sparse user-item rating matrix.

In terms of the Python environment used, both use NumPy and Pandas with TensorFlow (Keras). Common imports include tensorflow.keras for building models, and scikit-learn utilities for preprocessing.

Both notebooks have used keras.layers.Embeddingto build the models as we ll see later. In the large dataset NB, we have also usedtf.keras.utils.Sequence and defined data generators to handle large batches to efficiently handle the huge dataset. Both notebooks use an early-stopping callback to prevent overfitting.

Dataset

1. Small dataset (100K): Uses MovieLens small ratings. According to GroupLens, the latest small set has 100k ratings from ~600 users on ~9,000 movies. (The classic ml-100k subset has 1,000 users and 1,700 movies, but here the notebook data folder suggests data-small/ and resembles ml-latest-small.)
2. Large dataset (33M): Uses the full latest MovieLens ratings. The dataset describes ~33 million ratings by ~330k users on ~86k movies. The notebooks refer to data-large/ratings.csv and movies.csv, which likely come from the ml-latest full dump.

Each ratings.csv file contains columns (userId, movieId, rating, timestamp). However, timestamps are ignored in this project. The movies.csv file contains (movieId, title, genres). Both notebooks merge or reference these to compute recommendations but mainly use the numeric IDs and ratings.

Users and movie IDs in MovieLens are not 0-based or contiguous. So we map the original IDs to contiguous indices 0…N–1. This is needed for Keras embeddings (which index by [0,num_items)). We save the bidirectional mapping so that one can convert predicted indices back to original IDs for recommendation output.

Data Preprocessing & Feature Engineering

We follow similar preprocessing in both the datasets, just scaled for dataset size:

• ID Mapping: Build dictionaries to convert userId→user_idx and movieId→movie_idx (where the _idx objects are 0-based).
• Normalizing: To train more stably, raw ratings (in the range 0.5–5 stars) are scaled to [0,1]. To achieve this we use min-max scaling (MinMaxScaler).
• Sampling (Only for large dataset): To speed up training on 33M rows, the large notebook subsamples “active” users. We are only choosing users that have atleast rated 100 movies. Only these users’ ratings are kept in ratings_df_sampled. This reduced set is then used for model training. This is a form of implicit feature engineering – focusing on higher-activity users gives the model more reliable patterns and avoids noisey data from less active users.
• Train/Test Split: In the case of the small dataset, we split the 100K ratings into training and test sets (80/20 split using train_test_split). The large notebook also splits into train/val/test (80/10/10), but only after (optionally) filtering active users.
• Data Generators: (Only for large dataset): We define a custom data generator by subclassing tf.keras.utils.Sequence, which enables efficient and scalable batch-wise data loading during model training. Instead of loading the entire dataset into memory, the RatingDataGenerator dynamically yields mini-batches of user–movie interaction data, making it suitable for large recommendation datasets such as the one we are us The __len__ method specifies how many batches constitute one epoch, while __getitem__ retrieves a specific batch by slicing shuffled indices and splitting it into user IDs, movie IDs, and corresponding ratings. These inputs are reshaped and returned as a dictionary with named keys (user_input, movie_input) to align with the model’s input layers, ensuring compatibility during training. You can inspect the notebook for clarity. Additionally, the on_epoch_end method reshuffles the data after each epoch to prevent the model from learning any unintended ordering patterns, thereby improving generalization. The code is :

class RatingDataGenerator(tf.keras.utils.Sequence):

    def __init__(self, data, batch_size=256, shuffle=True, **kwargs):
        super().__init__(**kwargs)
        self.data = data
        self.batch_size = batch_size
        self.shuffle = shuffle
        self.indices = np.arange(len(self.data))
        self.on_epoch_end()

    def __len__(self):  # Number of batches per epoch
        return int(np.ceil(len(self.data) / self.batch_size))

    def __getitem__(self, index):
        batch_indices = self.indices[index * self.batch_size:(index + 1) * self.batch_size]
        batch_data = self.data[batch_indices]

        X_user = batch_data[:, 0].astype(np.int32).reshape(-1, 1)
        X_movie = batch_data[:, 1].astype(np.int32).reshape(-1, 1)
        y = batch_data[:, 2].astype(np.float32)

        return {"user_input": X_user, "movie_input": X_movie}, y
        # using named inputs to match model's input layers 

    def on_epoch_end(self):  # shuffle after each epoch
        if self.shuffle:
            np.random.shuffle(self.indices)

We create a 80/10/10 train/test/val split of the data and then create data generators for each using the RatingDataGenerator class as shown in the code below:

# create train/val/test split (80/10/10)
SEED = 69
train_indices, temp_indices = train_test_split(np.arange(len(data_array)), test_size=0.2, random_state=SEED)
val_indices, test_indices = train_test_split(temp_indices, test_size=0.5, random_state=SEED)
train_data = data_array[train_indices]
val_data = data_array[val_indices]
test_data = data_array[test_indices]

# creating generators
batch_size = 2048*2
train_generator = RatingDataGenerator(train_data, batch_size=batch_size)
val_generator = RatingDataGenerator(val_data, batch_size=batch_size, shuffle=False)
test_generator = RatingDataGenerator(test_data, batch_size=batch_size, shuffle=False)

It is these RatingDataGenerator objects that are then passed onto for training and testing the model.

Model Architectures

We implement neural collaborative filtering via embedding layers followed by dense layers. In matrix-factorization terms, each user and movie is embedded into a latent vector, and their interaction predicts the rating. Roughly speaking:

• Embeddings: Each user idx and movie idx (both 0-indexed and contiguous) goes into its own Embedding layer. The embedding dimensions (embedding_dim) differ: the small model uses 64-dimensional embeddings, the large model uses 128. The large model also adds L2 regularization on the embeddings.
• Combination: The user and movie embeddings are flattened and combined through concatenation (using tf.keras.layers.Concatenate).
• Hidden Layers: Both small and large models use a few fully connected dense layers after the embeddings. The small model has 4 hidden layers of sizes 128,64,32 and 16 whereas the large model has 4 of sizes 256, 128,64 and 32. We use ReLU as the activation for each of the hidden layers. We can optionally use Dropout layers for regularization effects.
• Output: In both cases, a single neuron with sigmoid activation produces a normalized rating as the output. Sigmoid is chosen because inputs were scaled to [0,1]. Training loss is Mean Squared Error (MSE); we also include MAE as a metric.

To summarize, both the small and large models are multi-layer perceptrons taking two integer inputs. They are essentially performing matrix factorization with learnable user and item vectors.

As in standard CF, each user u has a vector ***p_u and each item i has a vector `qi***associated with them, and the model approximates ratingr{ui}via an interaction ofp_u.q_i`.) Here, the “interaction” is learned via the neural network instead of a direct dot product of vectors.

Model Hyperparameters

Model Training

• Small dataset (in-memory): We usemodel.fit on NumPy arrays: model.fit([X_user, X_movie], y, batch_size=32, epochs=50, validation_split=0.2, callbacks=[early_stop]). Early stopping halts training if val_loss does not improve for 5 epochs. Because the dataset is small, all data fits in memory easily.
• Large dataset (generator-based): The large dataset cannot fit in a reasonable batch. So we use the custom RatingDataGenerator class that inherits tf.keras.utils.Sequence. This generator reads data in chunks and yields minibatches of (user, movie) inputs with ratings as labels. For example, its __getitem__ returns a batch of size 4096. Keras can then train with model.fit(train_generator, validation_data=val_generator, epochs=10, callbacks=[early_stop]). This approach streams data in batches, shuffling each epoch. It is memory-efficient (only one batch at a time) and uses Keras’ guaranteed-epoch semantics.
• Early stopping: Both models use early stopping on validation loss, which avoids overfitting. The small model splits off 20% of training setfor validation. The large model as mentioned before uses separate generators for train/val.

# train using generators 
history = model.fit(
    train_generator,
    validation_data=val_generator,
    epochs=10,
    callbacks=[early_stop],
    verbose=1
)

Visualization of the training of the large model (which took almost 2.5 hours).

Evaluation Metrics

Given that this boils down to a standard regression problem, we calculate the common regression metrics (RMSE and MAE) on the held-out test set. The model’s normalized output is inverse-transformed to original rating scale to get the rating in the expected range of 0.5–5.

We also also evaluate ranking metrics by treating the trained model as a recommender and generate top-K movie recommendations for each test user. We compute Precision@K, Recall@K, HitRate@K, and NDCG@K for K=5,10.

We also demonstrate a few inference examples by generating top-10 recommendations for sample users in the large model NB. Also, for a new user (ID not seen in training) ie a cold start problem, we simply recommends globally popular or highly-rated movies using a Bayesian average of mean rating value and number of ratings.

We also use various plots to inspect the results such as scatter plot of predicted vs actual ratings, histogram of residuals etc, to note bias or systemic errors. For example:

From the small dataset model.

Inference

Generating recommendations: For both models, we generate user recommendations by taking a user ID and outputting the top-N recommended movies. Typically, this means:

1. Map the original user ID to user_idx.
2. Score every movie by calling model.predict([user_idx, all_movie_idx]).
3. Sort by predicted score (highest first) for unseen movies (so we filter out seen movies).
4. Map the top indices back to original movieId and look up movie metadata like title, genres etc.

Cold-start: In the large model NB, we explicitly handle a cold-start example by checking if a user ID is new (ie not present in the bidirectional embedding map). If so, it falls back to recommending movies by global popularity.

Scalability & Performance

Memory usage: The small model loads all data at once since it requires very less memory. The large dataset (~33M rows) is huge and therefore is very expensive to load in one shot. By sampling and using generators, memory use stays under control.

Training time: The small model (100K ratings, batch_size = 32, 50 epochs) trains in seconds to minutes on a typical CPU. The large model (tens of millions of ratings, batch size 4096, 10 epochs) takes hours (on CPU).

Trade-offs: The small model NB is simple and fast to run, making it good for illustration and experimentation. However, it does not reflect production challenges like data volume.

The large model NB is more realistic but at the cost of complexity (requiring custom-defined data generators etc) and resource usage. For instance, larger embeddings and lower learning rates were needed for stability. Also, hyperparameter tuning on 33M examples is very costly (which we have not performed in this project).

I hope this project helps you get an understanding of how to use neural networks for building recommendation systems (collaborative filtering using embeddings). Until next time:)