How to Handle Large Files in Node.js: Uploading, Storing, and Managing Files with MongoDB

Photo by Tim Mossholder on Unsplash

Handling large files can be a daunting task for any developer, especially when you’re building a Node.js application. Uploading, storing, and managing large file, whether they’re images, videos, or documents, demands an efficient strategy to ensure fast performance, secure storage, and easy retrieval. With the right approach, you can make your application robust enough to handle large file uploads seamlessly, keeping your users happy and your app running smoothly.

In this guide, we’ll walk through the steps of managing large files in Node.js using MongoDB, covering best practices for uploading, storing, and managing these files efficiently.

Why File Handling is Challenging in Node.js

Node.js is known for its single-threaded, event-driven nature, which makes it great for handling asynchronous operations. However, working with large files can strain the server’s memory if not managed correctly, potentially causing performance issues or even crashes. Efficiently handling large files means implementing strategies that balance performance with storage needs. Here’s where MongoDB becomes an ideal solution, offering specialized tools for handling large files, particularly with GridFS.

Step 1: Setting Up Your Project

To get started, let’s set up a basic Node.js and Express application. If you haven’t already, install Node.js and MongoDB. Next, create a new directory and initialize your project with npm:

mkdir file-upload-app
cd file-upload-app
npm init -y

//install package
npm install express multer mongoose mongodb

Step 2: Configuring Multer for File Uploads

Multer is a popular middleware for handling multipart/form-data, which is primarily used for file uploads. To handle large files, Multer allows you to set limits and configurations to keep file sizes manageable.

In your project directory, crate an upload.js file to configure Multer:

const multer = require('multer');

// Set up storage destination and file naming convention
const storage = multer.memoryStorage();

const upload = multer({
  storage: storage,
  limits: { fileSize: 100 * 1024 * 1024 } // Limit file size to 100MB
});

module.exports = upload;

Here, we set up memory storage to process files in memory before moving them to MongoDB. Setting limits ensures your server only accepts files within a manageable size.

Step 3: Configuring MongoDB for Large File Storage with GridFS

MongoDB is not designed to store large files directly within its document collections, so MongoDB provides GridFS — a system for storing and retrieving large files (larger than 16 MB) in smaller chunks.

Let’s set up GridFS in our server.js file:

const express = require('express');
const mongoose = require('mongoose');
const { GridFsStorage } = require('multer-gridfs-storage');
const upload = require('./upload'); // Multer configuration

const app = express();

// MongoDB URI
const mongoURI = 'mongodb://localhost:27017/fileUploadDB';

// Connect to MongoDB
mongoose.connect(mongoURI, {
  useNewUrlParser: true,
  useUnifiedTopology: true
});

const conn = mongoose.connection;

// GridFS storage setup
const storage = new GridFsStorage({
  url: mongoURI,
  file: (req, file) => {
    return {
      filename: `${Date.now()}-${file.originalname}`,
      bucketName: 'uploads'
    };
  }
});

const uploadMiddleware = multer({ storage });

conn.once('open', () => {
  console.log('MongoDB connected and GridFS ready!');
});

app.listen(3000, () => {
  console.log('Server running on http://localhost:3000');
});

In this setup:

• We define a GridFsStorage with bucketName: ‘uploads’, specifying the collection where files will be stored.
• Each file is assigned a unique filename using a timestamp to prevent overwriting.

Step 4: Creating the File Upload Endpoint

Now, let’s create an endpoint for uploading files to the server, which will then be stored in MongoDB using GridFS.

Add this route to server.js:

// POST route for file upload
app.post('/upload', uploadMiddleware.single('file'), (req, res) => {
  if (!req.file) {
    return res.status(400).send('No file uploaded.');
  }
  res.status(201).send({ fileId: req.file.id, message: 'File uploaded successfully!' });
});

Here’s what’s happening:

• Endpoint: POST /upload accepts a single file named file in the form data.
• Response: Once the file is successfully stored in GridFS, it returns a fileId for easy retrieval.

Step 5: Retrieving and Streaming Large Files

For large files, retrieving the entire file data might be inefficient. Streaming files is a better approach, allowing you to send the file in chunks rather than loading it entirely into memory. Let’s set up a route for streaming files from MongoDB:

app.get('/files/:fileId', async (req, res) => {
  try {
    const { fileId } = req.params;
    const bucket = new mongoose.mongo.GridFSBucket(conn.db, {
      bucketName: 'uploads'
    });

    const downloadStream = bucket.openDownloadStream(mongoose.Types.ObjectId(fileId));

    downloadStream.on('data', (chunk) => {
      res.write(chunk);
    });

    downloadStream.on('end', () => {
      res.end();
    });

    downloadStream.on('error', (err) => {
      console.error(err);
      res.status(500).send('File could not be retrieved');
    });
  } catch (err) {
    res.status(400).send('Invalid file ID');
  }
});

This endpoint:

• Streams file in chunks: Using MongoDB’s GridFSBucket openDownloadStream, it retrieves files in chunks, optimizing memory usage.
• Handles large files: This method prevents large files from overwhelming memory, ensuring efficient and stable performance.

Step 6: Deleting Files from MongoDB

Finally, let’s add functionality to delete files, ensuring we keep storage optimized and free up space when files are no longer needed.

app.delete('/files/:fileId', async (req, res) => {
  try {
    const { fileId } = req.params;
    const bucket = new mongoose.mongo.GridFSBucket(conn.db, {
      bucketName: 'uploads'
    });

    bucket.delete(mongoose.Types.ObjectId(fileId), (err) => {
      if (err) return res.status(500).send('File deletion failed');
      res.status(200).send('File deleted successfully');
    });
  } catch (err) {
    res.status(400).send('Invalid file ID');
  }
});

This endpoint deletes a file from MongoDB, freeing up space and ensuring only necessary files are stored.

Best Practices for Handling Large Files in Node.js

1. Use Streaming: Always stream files when possible. Loading large files directly into memory can slow down or crash app.
2. Limit File Sizes: Use Multer’s limits option to control the maximum upload size and prevent overly large files from overwhelming your server.
3. Monitor Storage: With GridFS, MongoDB stores files in chunks. Monitor your MongoDB storage to ensure that large files are managed appropriately.
4. Cache Metadata: For frequently accessed files, consider caching metadata or basic file information in a faster storage layer (like Redis) for quick lookups.

Handling large files in Node.js becomes manageable when you pair with MongoDB’s GridFS. From uploading and storing to streaming and deleting, each step plays a role in ensuring that your application remains responsive and efficient, even with large file handling. By following best practices and leveraging tools like Multer and GridFS, you can build a file handling system that’s robust, scalable, and ready for production.

With these techniques, you’re now equipped to handle large files in Node.js, keeping your app running smoothly while providing an optimal experience for users.

Stackademic 🎓

Thank you for reading until the end. Before you go:

• Please consider clapping and following the writer! 👏
• Follow us **X | [LinkedIn](https://www.linkedin.com/company/stackademic) | [YouTube](https://www.youtube.com/c/stackademic) | [Discord](https://discord.gg/in-plain-english-709094664682340443) | [Newsletter](https://newsletter.plainenglish.io/) | [Podcast](https://open.spotify.com/show/7qxylRWKhvZwMz2WuEoua0)**
• **Create a free AI-powered blog on Differ.**
• More content at **Stackademic.com**