Category: Storage

With the open web service S3 sync in Eyevinn Open Source Cloud you can synchronize the contents of one S3 bucket to another S3 bucket without having to download and upload the files first. This service can be used to for example:

• Migrate from cloud storage in AWS to MinIO storage service in Eyevinn Open Source Cloud
• Migrate from MinIO storage service in Eyevinn Open Source Cloud to MinIO storage service on own infrastructure
• Backup and provide redundancy for critical files

In this blog post we will give an example of how you can synchronize files on an S3 bucket in AWS to a bucket in a MinIO server instance in Eyevinn Open Source Cloud.

Create an account for free at app.osaas.io and create your tenant. If you already have access to Eyevinn Open Source Cloud you can skip this step.

Step 1: Create the destination bucket

Login and navigate to the MinIO service in the catalog of open web services. Follow the MinIO getting started guide in a previous blog post here. Once completed you should have MinIO server and a bucket called “tutorial” and a web user interface to access it.

Step 2: Setup access to source bucket

Setup and obtain the access credentials from AWS for the source bucket. Ensure that the access credentials can only access the source bucket.

Navigate to the S3 Sync service in the Eyevinn Open Source Cloud web console and select the tab titled “Service Secrets”. Create secrets called “sourceaccesskey” and “sourcesecretkey” where “sourceaccesskey” stores the AWS_ACCESS_KEY_ID and “sourcesecretkey” stores the AWS_SECRET_ACCESS_KEY.

Step 3: Setup access to destination bucket

Add two new secrets called “destaccesskey” and “destsecretkey” where “destaccesskey” is the MinIO RootUser and “destsecretkey” is the RootPassword that you configured in step 1. You should now have 4 secrets created that you will use in the next step.

Step 4: Create an S3 sync job

In this example we want to synchronize the content on an S3 bucket in AWS called “lab-testcontent-output” and available in the “eu-north-1” AWS region to the bucket called “tutorial” on a MinIO server instance in Eyevinn Open Source Cloud. Choosing a folder on this bucket will migrate all the files and the subfolders in this folder.

• Name: A unique name for the S3 sync job.
• CmdLineArgs: We enter here the S3 URL for the source and destination.
• SourceAccessKey: We reference the “sourceaccesskey” secret we created.
• SourceSecretKey: We reference the “sourcesecretkey” secret we created.
• SourceRegion: We can enter “eu-north-1” here but in this example it was not needed to specify.
• DestAccessKey: We reference the “destaccesskey” secret we created.
• DestSecretKey: We reference the “destaccesskey” secret we created.
• DestEndpoint: The URL to the MinIO storage instance in Eyevinn Open Source Cloud.

Press the button “Create” to create and start the job.

Now we have a job running that starts to migrate the files from the AWS S3 bucket to a bucket in Eyevinn Open Source Cloud.

Command Line Tool

You can also use the OSC command line tool to create this job.

osc create eyevinn-s3-sync guidecli \
  -o cmdLineArgs="s3://lab-testcontent-output/osc/VINN-11/ s3://tutorial/" \
  -o SourceAccessKey="{{secrets.sourceaccesskey}}" \
  -o SourceSecretKey="{{secrets.sourcesecretkey}}" \
  -o DestAccessKey="{{secrets.destaccesskey}}" \
  -o DestSecretKey="{{secrets.destsecretkey}}" \
  -o DestEndpoint="https://eyevinnlab-jonas.minio-minio.auto.prod.osaas.io"

Conclusion

With the open web service providing storage functionality in Eyevinn Open Source Cloud you always have the option to run the same solution on your own premises as it is based on open source, and with this tooling you can easily migrate data to Eyevinn Open Source Cloud as well as from Eyevinn OSC to a self-hosted solution.

As a follow-up to our last post how you can simplify your file storage without being locked in with a specific vendor we will in this post walk you through step-by-step how to get started with file storage in Eyevinn Open Source Cloud.

Why Open Source Cloud with MinIO?

Using an open web service based on open source you are not locked in with a specific vendor and you have the option to run the very same code in your own infrastructure or cloud. To reduce the barrier to get started we have included in the free tier one MinIO server instance with 50 GB of storage. Storage interface is compatible with AWS S3 tools offering a wide range of options to access the storage.

Create an account for free at https://app.osaas.io and create your tenant. If you already have access to Eyevinn Open Source Cloud you can skip this step.

Step 1: Create a MinIO server instance

Login and navigate to the MinIO service in the catalog of open web services. Follow the MinIO getting started guide in the Open Source Cloud documentation. Once completed you should have MinIO server and a bucket called “tutorial”.

Step 2: Provide a web user interface to the storage

To manage the storage we have a couple of options. Either use the AWS S3 command line tool for uploading and downloading files, the AWS S3 SDK or a desktop application. If you want to provide your users with a web based user interface to the buckets on this MinIO server instance you can use another open web service available in Eyevinn Open Source Cloud.

Navigate to the Filestash service in the catalog of services in Eyevinn Open Source Cloud. Create a new Filestash instance by pressing “Create filestash” button.

Follow the steps below to configure and connect this manager with the MinIO instance you created earlier.

• 1. Click on the instance card once it is in state running. A new page will open in a new tab or browser window.
• 2. Enter an administrator password for this Filestash storage manager instance.
• 3. In the navigation sidebar on the left click on the item “Backend”. Select S3 as storage backend. You may remove the others as we will be only be using S3 in this example.
• 4. Choose the authentication middleware ADMIN. This means that you will login with the admin password you just created. You might at least want to use HTPASSWD for more granular access control in practice.
• 5. Select S3 backend
• 6. Enter the access key id and secret key. This is the RootUser and RootPassword that you set for your MinIO instance. The endpoint is the URL to the MinIO server instance that you created.

Step 3: Upload a file

Now go back to the start page by clicking on the instance card and login with the admin password that you created. Use drag-and-drop to upload a file.

Now you have a storage based on open web services in Eyevinn Open Source Cloud and a web based user interface to access it. A storage for storing poster and image objects for a streaming service, backing up project files, sharing large media files with team members or creating scalable media libraries to mention a few examples of use cases.

Ever struggled with managing digital files for your project? Whether you’re a developer, content creator, or just someone who needs reliable file storage, I’ve got a game-changing solution that’s both powerful and surprisingly easy to use.

What Exactly is Object Storage?

Think of object storage like a super-smart, infinitely expandable digital filing cabinet. Instead of saving files on your local computer or a single hard drive, you can store them in the cloud, access them from anywhere, and scale your storage as your needs grow. The best part? You don’t need to be a tech wizard to use it.

Why Open Source Cloud with MinIO?

Traditional cloud storage can be expensive and complicated. Open Source Cloud offers a refreshing alternative:

– Free tier with 50 GB of storage
– Completely compatible with popular tools like AWS S3
– No complex setup or massive technical knowledge required
– Supports media, images, documents, and more

Your Quick Start Guide

Getting Set Up

– Create an Account: Sign up for Open Source Cloud (it’s free!)
– Install the Basics: You’ll need NodeJS and a few simple command-line tools
– Follow the simple instructions: https://docs.osaas.io/osaas.wiki/Service%3A-MinIO.html
– Create Your Storage: Set up a MinIO storage service in just a few clicks

Creating Your First Bucket

A “bucket” is just a fancy term for a folder in the cloud. You can create as many as you need for different projects – one for photos, another for documents, another for backups.

Uploading and Accessing Files:

– Upload files using simple commands
– Generate shareable links
– Access your files from any device
– Integrate with existing tools and applications

Real-World Use Cases:

– Storing poster and image objects for a streaming service
– Backing up project files
– Sharing large media files with team members
– Creating scalable media libraries

The Open Source Difference

What makes this special isn’t just the technology – it’s the philosophy. Open Source Cloud shares revenue with the open-source authors, which means you’re supporting the community of developers who create these amazing tools.

Getting Started is Easier Than You Think

Don’t let technical jargon intimidate you. With Open Source Cloud’s MinIO, you can have a professional-grade storage solution up and running in minutes, without spending a fortune or getting a computer science degree.

Ready to simplify your file storage?

– Visit Open Source Cloud
– Sign up for a free account
– Start storing your files in minutes

Additional Resources:

– Open Source Cloud: https://app.osaas.io
– API Documentation: https://api.osaas.io

Tip: Start small, explore, and scale as you grow!