Efficient Image Handling: Streamlined Multi-File Uploads to S3 with Next.js

Uploading multiple images to a web application can be a performance bottleneck. While storing entire files on the server can be inefficient, directly uploading them to S3 without any backend processing isn’t always the ideal solution for web applications that need to associate image data with other user information.

This article explores a more optimized approach for Next.js applications: receiving image files as part of a POST request in the backend, efficiently streaming them to S3, and saving the image URLs along with other user data in the database. This method leverages Next.js’s ability to handle FormData while minimizing resource usage on the backend. We’ll delve into the steps involved in building this robust image upload pipeline, including handling file selection and preparation on the frontend, creating efficient backend API endpoints to receive and process the image data, and securely uploading images to S3.

Prerequisites:-

To effectively implement this image upload solution, you’ll need the following:

Development Environment

• Node.js and npm (or yarn): Ensure you have a recent version of Node.js and a package manager installed.
• Next.js: Create a new Next.js project or use an existing one.
• Code Editor: Choose a suitable code editor or IDE for development.

AWS Account and Configuration

• AWS Account: Create an AWS account if you don’t have one already.
• S3 Bucket: Set up an S3 bucket to store the uploaded images.
• IAM User: Create an IAM user with appropriate permissions to access the S3 bucket.
• AWS Credentials: Configure your Next.js application with the IAM user’s access key and secret key.

Additional Libraries

• @aws-sdk/client-s3
• @aws-sdk/lib-storage

Handling Text Fields and File Uploads in the Frontend Form

<form onSubmit={handleSubmit}>
   <input type="text" placeholder="Title" value={title} onChange={(e) => setTitle(e.target.value)} />
   <input placeholder="Description" value={description} onChange={(e) => setDescription(e.target.value)} />
   <input type="file" multiple onChange={handleFileChange} />
   <button type="submit">Upload</button>
</form>

The provided code defines an asynchronous function handleCreate that handles the process of preparing and sending form data to the backend. This function begins by constructing a FormData object, a suitable format for handling form data, including file uploads. It iterates over an array of files, appending each file's originFileObj property to the FormData object. Additionally, it includes the title and description fields in the FormData object. Subsequently, the function utilizes Axios to send a POST request to the specified API endpoint, incorporating the constructed FormData as the request body. To ensure correct handling of file uploads, the request headers are set to specify the Content-Type as multipart/form-data. Finally, a basic error-handling mechanism is implemented using a try-catch block to capture potential exceptions during the process.

const handleCreate = async () => {
  try {
    const formData = new FormData();
    files.forEach(file => {
      formData.append('files', file.originFileObj);
    });
    formData.append('title', title)
    formData.append('description', description)

    const response = await axios.post('http://localhost:3000/api/post', formData, {
      headers: {
        'Content-Type': 'multipart/form-data',
       },
     });
   } catch (error) {
     console.log(error)
   }
 }

The provided code defines a Next.js API route handler responsible for processing image uploads and storing associated data. The function begins by extracting form data and separating image files from other form fields. Subsequently, it iterates through the uploaded files, converting them into readable streams and uploading them to an S3 bucket using the AWS SDK. The uploaded image URLs are then collected and combined with the remaining form data. This combined data is then used to create a new record in the database using Prisma. The code incorporates error handling mechanisms to gracefully manage potential issues during the S3 upload and database operations, providing informative error messages to the client.

import prisma from '@/database'
import { NextResponse } from 'next/server'
import { S3Client } from '@aws-sdk/client-s3'
import { Readable } from 'stream'
import { Upload } from '@aws-sdk/lib-storage'

const s3Client = new S3Client({
  region: process.env.AWS_REGION,
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
  },
})

export async function POST(req) {
  const formadata = await req.formData()
  let data = {}
  const files = []
  let fileUrls = []
  for (const [key, value] of formadata.entries()) {
    if (key === 'files') {
      files.push(value)
    } else {
      data[key] = value
    }
  }

  try {
    fileUrls = await Promise.all(
      files.map(async (file) => {
        const fileStream = Readable.from(file.stream())

        const uploadTos3 = new Upload({
          client: s3Client,
          params: {
            Bucket: process.env.BUCKET_NAME,
            Key: `images/${Date.now()}-${file.name}`,
            Body: fileStream,
          },
        })
        const res = await uploadTos3.done()
        return res.Location
      }),
    )
  } catch (err) {
    NextResponse.json(
      { message: 'Error uploading image to s3.' },
      { status: 500 },
    )
  }

  data = { ...data, files: fileUrls}

  try {
    const post = await prisma.post.create({
      data,
    })
    NextResponse.json({ post: post }, { status: 400 })
  } catch (err) {
    NextResponse.json(
      { message: 'Error uploading data to the database.' },
      { status: 500 },
    )
  }
}

Replace <AWS_REGION>, <AWS_ACCESS_KEY_ID>, <AWS_SECRET_ACCESS_KEY>, and <BUCKET_NAME> with your actual AWS credentials and S3 bucket information.

Conclusion

This article has demonstrated a streamlined approach to handling multi-file image uploads within Next.js applications. By directly streaming images to S3 and integrating additional form data, we’ve optimized the upload process, reducing server load and enhancing performance. This method provides a flexible foundation for applications requiring image storage and associated data, ensuring scalability and security through S3’s robust features. By following the outlined steps and tailoring them to specific requirements, developers can implement efficient image-handling solutions within their Next.js projects.