Migrate from AWS S3 storage to Zata.ai S3 storage

Prerequisites

• Select any Linux operating system to perform the migration process.

• In this setup, we have chosen Ubuntu 22.04 as the operating system.

• Click on the provided link to Launch the Instance Instance.

AWS Setup

For AWS configuration:

• Log in to the AWS Management Console.

• This console will be used to create and manage the S3 bucket and generate access credentials required for migration.

Create a Bucket in AWS S3

• After logging in to the AWS Management Console, search for S3 in the services search bar.

• Open the S3 service and click on Create bucket.

• Provide the required details such as bucket name and region, and complete the bucket creation process.

Set Bucket Policy and Permissions

• After creating the bucket, apply the required bucket permissions and bucket policy.

• Click on the created bucket, then navigate to Permissions → Bucket Policy → Edit.

• Update the policy by modifying the bucket name, user name, and user account ID according to your requirements before saving it.

When applying a bucket policy, ensure you customize the values based on your requirements. Do not copy this policy directly.

Create Access and Secret Key

• Navigate to the IAM service from the AWS Management Console.

• Go to the Users section and select an existing user to create an access and secret key.

• If the user does not exist, create a new user and assign AdministratorAccess permissions.

Verify that the selected user has sufficient permissions to access and manage the S3 buckets. Then, navigate to the Security Credentials section to generate the Access Key ID and Secret Access Key.

Now create the security credentials to click on the security credentials to create the access key.

Before clicking on "Done," first copy the access and secret keys to your notepad.

After performing these processes in AWS, configure the Zata.ai settings.

Setup Zata.ai

For Zata.ai configuration:

• Open the Zata.ai login page and sign in to your account login page.

• Create a bucket by following the Zata Bucket Creation Guide Bucket Creation Guide

• Generate the Access Key ID and Secret Access Key by following the Access Keys Guide Access Keys Guide

• Refer to the Zata.ai documentation for more detailed information and features zata.ai

Guide to Move Data from AWS to Zata.ai

Ubuntu 22.04 is used as an intermediary system to facilitate the data migration from AWS S3 to Zata.ai S3. This system acts as a bridge between both storage platforms during the transfer process.

For the first process (AWS side setup):

• Update and upgrade the Ubuntu system to ensure all packages are up to date.

sudo apt-get update -y
sudo apt-get upgrade -y

For migrating data from AWS to Zata.ai, use rclone

Install the configure the rclone

Install and configure the rclone.

sudo apt-get install rclone -y

• To migrate data, rclone must be configured for both AWS and Zata.ai.

• First, configure rclone for AWS, and then configure it for Zata.ai.

• Run the following command to start the rclone configuration:

sudo rclone config

• When the rclone configuration starts, choose n to create a new remote.

• Enter AWS as the name of the remote.

For the storage type, choose option 4 to select the AWS S3–compliant storage provider.

Select your S3 provider by choosing option 1 for AWS S3.

• When prompted for env_auth, select false (press Enter to use the default value).

• This option is used only if credentials are taken from environment variables or EC2/ECS metadata.

• Since the Access Key ID and Secret Access Key were already created in the AWS section, manual entry will be used.

• Enter the AWS Access Key ID and AWS Secret Access Key when prompted.

• Set the endpoint according to your AWS region.

• In our case, we are using the Mumbai region, so the endpoint is:

  s3.ap-south-1.amazonaws.com

• When asked for the region, choose option 17 for Asia Pacific (Mumbai).

• When prompted for the S3 API endpoint, enter the following value:

  s3.ap-south-1.amazonaws.com

• This sets the S3 API endpoint for the Mumbai (Asia Pacific) region.

• Press Enter to accept all default values.

• When prompted for Storage Class, change the value and choose option 2.

• For the ACL option, choose 4.

• This setting applies only when creating buckets.

• Keep all remaining options as default by pressing Enter until the configuration is completed.

This is the remote configuration completed after following these steps:

Second process for Zata.ai

Configure rclone for Zata.ai

• Choose n to create a new remote.

• Set the remote name as zata.

• Choose option 4 for S3 Compliant Storage Provider.

• Choose option 13 for S3 Compatible Provider.

• Set env_auth to false to use manual credentials.

• Enter the Zata Access Key ID and Zata Secret Access Key (refer to the Access Keys Guide).Access Keys Guide

• Set the endpoint as:

  idr01.zata.ai

Ensure you have created buckets in both AWS and Zata.ai before running these commands.

Copy Data from AWS S3 to Zata.ai S3

• After completing the rclone configuration for both AWS and Zata.ai, you can start the data transfer.

• Use the following command to copy data from the AWS bucket to the Zata.ai bucket:

rclone copy AWS:<aws_bucket_name> zata:<zata_bucket_name> --s3-region ap-south-1 --progress

• Replace <aws_bucket_name> with your AWS S3 bucket name.

• Replace <zata_bucket_name> with your Zata.ai bucket name.

• The --progress option shows real-time transfer status.

• This command securely copies all objects from AWS S3 to Zata.ai S3.

After completing the data transfer, return to both the AWS S3 console and the Zata.ai dashboard to verify the status of the buckets and confirm that all data has been successfully copied.