Use AWS EC2 DataSync to Synchronize Between NFS and S3

1. AWS EC2 (Elastic Compute Cloud):

• AWS EC2 is the foundational service for deploying virtual servers in the cloud.
• In this scenario, EC2 instances are used to host both the DataSync agent and the Storage Gateway File Gateway.
• EC2 instances provide the computing power and network connectivity required for running these services.

2. AWS DataSync:

• AWS DataSync is a data transfer service that simplifies and accelerates moving large amounts of data between on-premises storage systems and AWS storage services.
• It facilitates data migration, replication, and synchronization tasks.
• The DataSync agent, deployed as an EC2 instance, acts as a bridge between the on-premises NFS file server and Amazon S3.
• It handles the transfer of data from the NFS file server to S3, ensuring efficient and secure synchronization.

3.AWS Storage Gateway

• AWS Storage Gateway is a hybrid storage service that enables seamless integration between on-premises environments and AWS cloud storage.
• It supports various storage protocols, including NFS, SMB, and iSCSI.
• In this scenario, we focus on the File Gateway type of Storage Gateway, which provides NFS access to data stored in Amazon S3.
• The File Gateway appliance, also deployed as an EC2 instance, serves as a virtual NFS file server that hosts the migrated data from the on-premises NFS server.
• It allows applications and users to access the data stored in S3 as if it were stored on a local NFS file share.

Integration between Services:

• EC2 instances host both the DataSync agent and the Storage Gateway File Gateway, providing the necessary compute resources for their operation.
• The DataSync agent, running on an EC2 instance, connects to the on-premises NFS file server and initiates the data transfer process to Amazon S3.
• Once the data is migrated to S3, the File Gateway appliance, running on another EC2 instance, provides NFS access to the data stored in S3, enabling seamless integration with on-premises applications and workflows.
• Together, these services form a cohesive solution for migrating and accessing data between on-premises environments and AWS cloud storage, leveraging the scalability, reliability, and flexibility of the AWS platform.

In summary, AWS EC2 provides the infrastructure for hosting essential components such as the DataSync agent and Storage Gateway File Gateway, while AWS DataSync and Storage Gateway enable efficient data transfer and seamless integration between on-premises storage systems and AWS cloud storage.

With the completion of your proof of concept (POC) environment, it’s time to move forward with deploying and activating a DataSync agent as an EC2 instance. This process will ensure smooth data synchronization between your on-premises NFS file server and Amazon S3.

Deployment of DataSync Agent:

Access AWS Management Console:

• Open your web browser and navigate to the AWS Management Console.
• In the search bar at the top, type “EC2” and select it from the search results.

Launch EC2 Instance:

• In the EC2 Management Console, click on “Launch instance” and then choose “Launch instance” again.

Configure Instance:

• In the “Names and tags” section, name your instance as “DataSync agent.”
• Add additional tags if needed.
• In the “Application and OS Images” section, search for “aws-datasync” and select the latest DataSync agent AMI from the Community AMIs tab.
• Choose the instance type as “t3.xlarge.”
• Select your key pair for login.
• In the “Network settings” section, choose your VPC and subnet.
• Enable auto-assign public IP.
• Select the existing security group named “DataSyncAccess” which allows necessary traffic.

Launch Instance:

• Review your configuration and click “Launch instance.”
• Once the launch is successful, navigate to “View all instances” at the bottom of the page.
• Monitor the deployment status until it shows “2/2 checks passed” indicating readiness.

Retrieve Instance Details:

• Select the DataSync agent instance and navigate to the “Details” tab.
• Copy both the Public IPv4 address and Private IPv4 addresses for later use.

Activate DataSync Agent:

• Return to the AWS Management Console and search for “DataSync.”
• Choose “Get started” and navigate to the “Create agent” page.
• In the “Deploy agent” section, select “Amazon EC2” as the hypervisor.
• Choose “Automatically get the activation key from your agent.”
• Enter the Public IPv4 address of the DataSync agent instance and click “Get key.”

Complete Activation:

• Enter a name for the agent (e.g., “NFS DataSync agent”).
• Click “Create agent” and verify the successful activation message.

Task 1: Creating and Running a DataSync Task:

1. Modify NFS Server Configuration:

• Access your Server Linux instance via SSH.
• Use the command sudo vi /etc/exports to modify the exports file.
• Add a new line with /var/nfs DATA_SYNC_AGENT_IP(rw,fsid=2,sync,no_subtree_check).
• Save and exit the text editor, then run sudo exportfs -a to activate the changes.

Create DataSync Task:

• Return to the AWS DataSync Console.
• Navigate to the “Tasks” section and choose “Create task.”
• Configure the source location as NFS and enter necessary details.
• Configure the destination location as Amazon S3 and enter relevant information.
• Proceed through the configuration steps, reviewing and confirming your choices.

Start DataSync Task:

• Once the task is created, wait for it to become available.
• Select the task and choose “Start with defaults” from the Start dropdown menu.
• Monitor the execution status until it shows “Success.”

Task 2: Deploying and Activating an AWS Storage Gateway File Gateway:

1. Access Storage Gateway Console:

• Search for “Storage Gateway” in the AWS Management Console and select it.

Create Gateway:

• Click on “Create gateway” and proceed to set up a new gateway.

Configure File Gateway:

• Enter details such as Gateway name and time zone.
• Select “Amazon S3 File Gateway” as the Gateway type.
• Choose “Amazon EC2” as the Host platform and launch a new instance with specified settings.
• Configure storage settings as needed.

Activate Gateway:

• Follow the activation steps, ensuring correct connection options and IP addresses.
• Complete the activation process and wait for the gateway status to show as “Running.”

By following these steps, you’ll successfully deploy and activate both the DataSync agent and the AWS Storage Gateway File Gateway, enabling efficient data synchronization between your NFS file server and Amazon S3.