How to Mount S3 bucket to EC2 on AWS

S3fs is a FUSE-based file system that enables the mounting of an Amazon S3 bucket as a local file system in an AWS instance. This file system behaves like a network attached drive, allowing users to access data stored on S3 directly from the EC2 instance.

Filesystem in Userspace (FUSE) provides a straightforward interface for userspace programs to create and export virtual file systems to the Linux kernel. It also offers a secure method for non-privileged users to develop and mount their own file system implementations.

Let’s Mount Your S3 Bucket To Your Linux Instance

The S3fs-fuse project, implemented in Python, utilizes Amazon’s Simple Storage Service as its backend. Amazon provides an open API for building applications on top of this service, which has been leveraged by various companies through different interfaces such as web, rsync, and FUSE.

To mount your S3 bucket to your Linux instance, please follow the steps outlined below. This tutorial assumes that you already have a Linux EC2 instance running on AWS with root access, as well as an existing S3 bucket that you want to mount on your Linux instance. Additionally, you will need an Access and Secret key pair with appropriate S3 permissions, or an IAM access to generate or create these credentials.

Please note that the following steps will be performed as a root user. If you are a regular user with sudo access, you can use the sudo command accordingly. Let’s begin the process.

1. Always update the system before starting:

For CentOS or Red Hat

yum update all

For Ubuntu

apt-get update

After sucessful update we are ready to rock’n’roll!

2. Install the dependencies

CentOS or Red Hat

sudo yum install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel

Ubuntu or Debian

sudo apt-get install automake autotools-dev fuse g++ git libcurl4-gnutls-dev libfuse-dev libssl-dev libxml2-dev make pkg-config

3. Get s3fs source code from git

git clone https://github.com/s3fs-fuse/s3fs-fuse.git

4. Compile and install the code

cd s3fs-fuse
./autogen.sh
./configure --prefix=/usr --with-openssl
make
sudo make install

Check if the installation went succesfully:

which s3fs

5. Create a config file

Create a config file in home folder with the with the name .passwd-s3fs and Paste the access key and secret key in the below format (In order to get your access and secret key read star section below)

nano ${PWD}/.passwd-s3fs

Paste your credentials to the file

Your_accesskey:Your_secretkey

Change the permission to the file

sudo chmod 640 ${PWD}/.passwd-s3fs

6. Create a directory and mount S3bucket

or provide the path of an existing directory

There are 2 types of bucket names — without dot(.) in the bucket name or with a dot(.) in a bucket name.

Mount bucket without a dot(.):

mkdir /mys3bucket
s3fs your_bucketname -o use_cache=/tmp -o allow_other -o uid=1001 -o mp_umask=002 -o multireq_max=5 /mys3bucket

where, “your_bucketname” = the name of your S3 bucket that you have created on AWS S3, use_cache = to use a directory for its cache purpose, allow_other= to allow other users to write to the mount-point, uid= uid of the user/owner of the mountpoint (can also add “-o gid=1001” for group), mp_umask= to remove other users permission. multireq_max= parameter to send request to s3 bucket, /mys3bucket= mountpoint where the bucket will be mounted.

Mount bucket with a dot(.):

mkdir /mys3bucket
s3fs your_bucketname /mys3bucket -o use_cache=/tmp -o allow_other -o uid=1001 -o mp_umask=002 -o multireq_max=5 -o use_path_request_style -o url=https://s3-{{aws_region}}.amazonaws.com

where, “your_bucketname” = the name of your S3 bucket that you have created on AWS S3, use_cache = to use a directory for its cache purpose, allow_other= to allow other users to write to the mount-point, uid= uid of the user/owner of the mountpoint (can also add “-o gid=1001” for group), mp_umask= to remove other users permission. multireq_max= parameter to send request to s3 bucket, /mys3bucket= mountpoint where the bucket will be mounted .

Remember to replace “{{aws_region}}” with your bucket region (example: eu-west-1).

That’s it!

Now just check if bucket is mounted. You should be able to see s3fs mounted volume.

df -h

* Getting the access key and secret key

To access your s3 bucket from your EC2 instance, you will require AWS Access key and Secret key with the appropriate permissions. Managing user permissions can be easily done through the AWS IAM (Identity and Access Management) Service. You can create an IAM user with S3 full access or a role with enough permissions. Alternatively, you can utilize the root credentials of your AWS Account for simplicity.

To begin, navigate to the AWS Menu, select your AWS Account Name, and click on “My Security Credentials.” This action will open the IAM console. From there, go to Users and select your Account name. Under the permissions tab, verify if you possess adequate access to the S3 bucket. If not, you can manually assign an existing “S3 Full-Access” policy or create a new policy with sufficient permissions.

Next, proceed to the Security Credentials Tab and create an Access Key. This will generate a new Access Key and Secret Key pair. In this section, you can view the access key and the secret key (the latter is visible when you click on the “show” tab) and also have the option to download them. Make sure to copy both keys separately.

Remember, you can always use an existing access and secret key pair or alternatively create a new IAM user and assign the necessary permissions to generate the access and secret key.