Transfer files from one S3 bucket to another

With the open web service S3 sync in Eyevinn Open Source Cloud you can synchronize the contents of one S3 bucket to another S3 bucket without having to download and upload the files first. This service can be used to for example:

• Migrate from cloud storage in AWS to MinIO storage service in Eyevinn Open Source Cloud
• Migrate from MinIO storage service in Eyevinn Open Source Cloud to MinIO storage service on own infrastructure
• Backup and provide redundancy for critical files

In this blog post we will give an example of how you can synchronize files on an S3 bucket in AWS to a bucket in a MinIO server instance in Eyevinn Open Source Cloud.

Create an account for free at app.osaas.io and create your tenant. If you already have access to Eyevinn Open Source Cloud you can skip this step.

Step 1: Create the destination bucket

Login and navigate to the MinIO service in the catalog of open web services. Follow the MinIO getting started guide in a previous blog post here. Once completed you should have MinIO server and a bucket called “tutorial” and a web user interface to access it.

Step 2: Setup access to source bucket

Setup and obtain the access credentials from AWS for the source bucket. Ensure that the access credentials can only access the source bucket.

Navigate to the S3 Sync service in the Eyevinn Open Source Cloud web console and select the tab titled “Service Secrets”. Create secrets called “sourceaccesskey” and “sourcesecretkey” where “sourceaccesskey” stores the AWS_ACCESS_KEY_ID and “sourcesecretkey” stores the AWS_SECRET_ACCESS_KEY.

Step 3: Setup access to destination bucket

Add two new secrets called “destaccesskey” and “destsecretkey” where “destaccesskey” is the MinIO RootUser and “destsecretkey” is the RootPassword that you configured in step 1. You should now have 4 secrets created that you will use in the next step.

Step 4: Create an S3 sync job

In this example we want to synchronize the content on an S3 bucket in AWS called “lab-testcontent-output” and available in the “eu-north-1” AWS region to the bucket called “tutorial” on a MinIO server instance in Eyevinn Open Source Cloud. Choosing a folder on this bucket will migrate all the files and the subfolders in this folder.

• Name: A unique name for the S3 sync job.
• CmdLineArgs: We enter here the S3 URL for the source and destination.
• SourceAccessKey: We reference the “sourceaccesskey” secret we created.
• SourceSecretKey: We reference the “sourcesecretkey” secret we created.
• SourceRegion: We can enter “eu-north-1” here but in this example it was not needed to specify.
• DestAccessKey: We reference the “destaccesskey” secret we created.
• DestSecretKey: We reference the “destaccesskey” secret we created.
• DestEndpoint: The URL to the MinIO storage instance in Eyevinn Open Source Cloud.

Press the button “Create” to create and start the job.

Now we have a job running that starts to migrate the files from the AWS S3 bucket to a bucket in Eyevinn Open Source Cloud.

Command Line Tool

You can also use the OSC command line tool to create this job.

osc create eyevinn-s3-sync guidecli \
  -o cmdLineArgs="s3://lab-testcontent-output/osc/VINN-11/ s3://tutorial/" \
  -o SourceAccessKey="{{secrets.sourceaccesskey}}" \
  -o SourceSecretKey="{{secrets.sourcesecretkey}}" \
  -o DestAccessKey="{{secrets.destaccesskey}}" \
  -o DestSecretKey="{{secrets.destsecretkey}}" \
  -o DestEndpoint="https://eyevinnlab-jonas.minio-minio.auto.prod.osaas.io"

Conclusion

With the open web service providing storage functionality in Eyevinn Open Source Cloud you always have the option to run the same solution on your own premises as it is based on open source, and with this tooling you can easily migrate data to Eyevinn Open Source Cloud as well as from Eyevinn OSC to a self-hosted solution.