Category: Media Solutions

There are several options on how to store files in the cloud today and in this blog post we will show how you with an open source project made available as a service in Eyevinn Open Source Cloud can simplify the access to the storage. In this blog we will as an example use Akamai S3 compatible Object Storage as the cloud storage.

Create storage bucket

Ref: https://techdocs.akamai.com/cloud-computing/docs/create-and-manage-buckets

1. Log in to Cloud Manager and select Object Storage from the left menu. If you currently have buckets on your account, they are listed on this page, along with their URL, region, size, and the number of objects (files) they contain.

2. One of the first steps to using Object Storage is to create a bucket. Here’s how to create a bucket using Cloud Manager, though you can also use the Linode CLI, s3cmd, and s4cmd.

3. Navigate to the Object Storage page in Cloud Manager (see View buckets).

4. Click the Create Bucket button to open the Create Bucket panel. If you have not created an access key or a bucket on this account, you are prompted to enable Object Storage.

5. Within the Create Bucket form, add a Label for the new bucket. This label must be unique and should not be used by any other bucket (from any customer) in the selected data center.

6. Choose a Region for the bucket to reside. See the Availability section on the Object Storage Overview page for a list of available regions.

7. Click Submit to create the bucket.

In this example we have created a bucket called “osc-blog” in the data center in Stockholm.

To be able to access this bucket we have created we need to create an access key. Navigate to Access Keys tab and press Create Access key. Give the access key a name and in this case we will limit the access to only the bucket we created.

Copy and store the generated “access key id” and “secret key” as you will use these later.

Setup Cloud Storage Manager

In Eyevinn Open Source Cloud web console navigate to the service called Filestash and press “Create filestash”.

Give the service a name for example “blog” in this case. Click on the instance card once it is in state running. A new page will open in a new tab or browser window. Then enter an administrator password for this Filestash storage manager instance.

In the navigation sidebar on the left click on the item “Backend”. Select S3 as storage backend.

You may remove the others as we will be only be using S3 in this example.

For simplicity we will be using the ADMIN authentication middleware. This means that you will login with the admin password you just created. You might at least want to use HTPASSWD for more granular access control in practice.

Enter the access key id and secret key.

The endpoint in this case is https://se-sto-1.linodeobjects.com as the bucket is located in region se-sto-1.

Upload a file

Now go back to the start page by clicking on the instance card and login with the admin password that you created.

Now you can upload a file by using drag-and-drop.

Conclusion

With this open source project now made available as a service in Eyevinn Open Source Cloud you can give your users a simpler and consistent user interface independent from what cloud storage provider you are using. Using Eyevinn Open Source Cloud you contribute back to a sustainable business model for open source as a share of the revenue is shared with the open source creator.

In a previous blog post we provided a walk-through on how to setup video file transcoding using Open Source Cloud based on SVT Encore and supporting backend services. In this blog post we are extending the setup by adding the creation of video-on-demand streaming files to the pipeline.

In this solution we will add another open source project made available as a service. The Encore Packager is a backend service that creates the VOD file package. It consumes jobs from a Redis queue and creates the VOD file package and uploads the package to an S3 bucket. For the creation of the VOD file package the open source packager Shaka Packager is used. The red box in the diagram below shows what we will add to our solution.

Step 1: Create another Valkey queue

Valkey provides a Redis compatible key / value store and we will create another queue for the packaging jobs. Navigate to the Valkey service in Open Source Cloud and press “Create valkey”. Give the instance a name and press Create.

Note down the IP and port to the Valkey instance card and this is what will be the Redis URL that we will refer to later in this blog. In this example it would be redis://172.232.131.169:10511.

Step 2: Launch another Encore Callback Listener

We will now create a separate service that can be used to monitor a transcoding job in SVT Encore so we know when the file is ready to be packaged. Navigate to the Encore Callback Listener in the web user interface. Click on button “Create callback” and enter the name of the instance, Redis URL (above), URL to the SVT Encore instance that we created last time and the name of the queue. We will call this queue for “package” now.

Important the URL to the SVT Encore instance is without the trailing slash.

Step 3: Create Encore Packager service

We can now move on with creating the Encore Packager service. Enter the name of the instance, Redis URL, name of queue in Redis (Valkey), output S3 URL, OSC token and the AWS credentials for the output S3 bucket. In this example we will have the following values:

Then press Create and wait for the instance to be ready.

Step 4: Submit a job

Now we are ready to try transcoding and creating a VOD package that we have available on an S3 compatible storage. We will create signed URL to the video file we want to transcode. For example:

https://lab-testcontent-input.s3.eu-north-1.amazonaws.com/NO_TIME_TO_DIE_short_Trailer_2021.mp4?SIGNURLSTUFF

Navigate back to the SVT Encore service and press the menu item to open API docs again. Click on the POST /encoreJobs bar and button “Try it out” and enter the following JSON. Here we have changed the progressCallbackUri to point to our Encore Callback Listener for VOD packaging.

{
  "externalId": "blog",
  "profile": "program",
  "outputFolder": "/usercontent/",
  "baseName": "blog",
  "progressCallbackUri": "https://demo-vod.eyevinn-encore-callback-listener.auto.prod.osaas.io/encoreCallback",
  "inputs": [
    {
      "uri": "https://lab-testcontent-input.s3.eu-north-1.amazonaws.com/NO_TIME_TO_DIE_short_Trailer_2021.mp4?SIGNURL",
      "seekTo": 0,
      "copyTs": true,
      "type": "AudioVideo"
    }
  ]
}

And then press button Execute. Now a job is submitted and if you want to see the progress you can go to the Encore Callback Listener service and open the instance logs to check that it is receiving the callbacks.

When the transcoding process is completed it will place a job on the packaging queue that will be picked up by the Encore Packager service. And when the packaging job is completed you will in this example find a VOD package ready for streaming: https://lab.cdn.eyevinn.technology/osc/NO_TIME_TO_DIE_short_Trailer_2021/bb347d8e-c095-43dc-ba5f-914c7e74f13d/index.m3u8.

Conclusion

You now have a fully fledged video transcoding and packaging pipeline for preparing video files for streaming using ´SVT Encore with some supporting services. All based on open source and you don’t have to setup your own infrastructure for this to get started. If you later choose to do so you are free to do it as the code and everything demonstrated here is available as open source.

In this blog post I will describe how to trim a video file on an S3 compatible bucket using ffmpeg without having to download it first, process it and then upload the result.

For trimming the video we will use the open source tool ffmpeg and a script that handles uploading the result to an S3 bucket. This open source script is available as a service in Open Source Cloud.

Step 1: Login to Eyevinn Open Source Cloud

Go to www.osaas.io and login. Sign up for an account if you don’t already have one. It is free to get started and you don’t even have to enter a credit card to try this out.

Step 2: Setup access to S3 bucket

Go to the service in Open Source Cloud called “FFmpeg to S3” using the search bar on the browse page. Click on the card “FFmpeg to S3 to go to the service.

Then click on the tab named “Service secrets”

Get the S3 access key credentials from your administrator of your S3 buckets. You need at minimum the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. Create a service secret for each of these credentials.

Step 3: Generate signed URL to the video to trim

Now we need to generate a signed URL for the video that you want to trim.

Copy the presigned URL to the clipboard

Step 4: Create a ffmpeg trim job

As an example we will extract the first 30 seconds of the video file and the ffmpeg command for that is:

ffmpeg -ss 0 -t 30 -c:v copy -c:a copy

Go back to the FFmpeg to S3 service page in Open Source Cloud and click on button “Create job”.

Enter the following in the settings dialog:

Name: “tutorial”
CmdLineArgs:
Replace [SIGNED_URL] from clipboard and lab-testcontent-input with the name of your bucket:

-i [SIGNED-URL] -d s3://lab-testcontent-input/tutorial-30sec.mp4 "-ss 0 -t 30 -c:v copy -c:a copy"

AwsAccessKeyId and AwsSecretAccessKey: reference to the service secrets created
Region: Location of the S3 bucket

Now press Create and wait for the job to be completed. When the job is completed you should have a file called tutorial-30sec.mp4 on the bucket you provided and 30 seconds duration.

Create a job from command line

You might want to automate or script the creation of these ffmpeg jobs and to facilitate that there is an open source SDK and command line tool for Eyevinn OSC. The command line tool is a Node.js script.

Follow the instructions on how to install Node.js if you don’t already have it installed.

Then install the CLI


% npm install -g @osaas/cli


Obtain the personal access token by going to Settings in OSC and the tab API. Here you find the personal access token that you copy to your clipboard. Set this token as an environment variable in your shell.


% export OSC_ACCESS_TOKEN=token


Now you can create the same job with the following command (replace [SIGNED-URL] and s3 bucket):


% osc create eyevinn-ffmpeg-s3 tutorialcli -o awsAccessKeyId="{{secrets.eyevinnawskeyid}}" -o awsSecretAccessKey="{{secrets.eyevinnawssecret}}" -o cmdLineArgs='-i [SIGNED-URL] -d s3://lab-testcontent-input/tutorial-30sec.mp4 "-ss 0 -t 30 -c:v copy -c:a copy"'


Conclusion

This was an example of how you can run ffmpeg to process a video file on an S3 bucket and output the result back to an S3 bucket without having to develop your own script for it as a script already existed that is open source and made available as a service in Eyevinn Open Source Cloud.

SVT Encore is a powerful open-source video transcoder specifically designed for the cloud. It forms the backbone of the transcoding process in the media supply chain, taking raw video inputs and converting them into multiple formats and bitrates suitable for adaptive streaming. The transcoding process involves breaking down video files into different resolutions and bitrates, allowing viewers to receive the best possible quality based on their device and network conditions.

To reduce the barrier to get started with SVT Encore we have added their project to Open Source Cloud together with some supporting backend services that we have added. This blog gives you a walk-through on how to setup video file transcoding using Open Source Cloud.

Prerequisites

• If you have not already done so, sign up for an OSC account.
• 5 remaining services on your subscription plan or individually purchased the services included in this solution.
• S3 compatible object storage solution

This solution is based on the following open source projects made available as a service:

• SVT Encore
• Valkey
• Encore Callback Listener
• Encore Transfer
• Retransfer

After completed this tutorial you will be able to transcode a video file on an S3 compatible storage and the output is placed on another S3 compatible storage when the processing is completed.

Step 1: Create Encore Queue

Go to the web user interface and navigate to the service called SVT Encore. Click on the button “Create queue” and give the queue a name.

You can leave the Profiles URL empty for now and then press Create.

Now you have an instance of SVT Encore running with one single queue and ready to receive transcoding jobs for processing. You can try this out by clicking on the menu item Open API docs to access the online REST API documentation and submit a job.

However, to automatically get transcoded files out from SVT Encore and transferred to the output storage we need a few more help services. So that we will setup now. Start by take a note of the URL to the SVT Encore instance.

Remove the trailing slash an keep it for later use. In this case it is https://demo-blog.encore.prod.osaas.io.

Step 2: Create Valkey queue

Valkey provides a Redis compatible key / value store and this i what we will use to manage the queue for transferring files out from Encore and to out output bucket.

Navigate to the Valkey service in Open Source Cloud and press “Create valkey”. Give the instance a name and press Create.

Note down the IP and port to the Valkey instance card and this is what will be the Redis URL that we will refer to later in this blog. In this example it would be redis://172.232.131.169:10507.

Step 3: Launch Encore Callback Listener

Now we need something that monitors a transcoding job in SVT Encore so we know when the file is ready to be transferred. For that you navigate to the Encore Callback Listener in the web user interface. Click on button “Create callback” and enter the name of the instance, Redis URL (above), URL to the SVT Encore instance and the name of the transfer queue. We call it “transfer” in this example.

Important the URL to the SVT Encore instance is without the trailing slash.

Press Create and you are done with this step for now.

Step 4: Setup secrets

Now we have the Callback Listener service running that will monitor transcoding job and place completed jobs in the transfer queue. Now we need a service that picks up a job from the transfer queue and actually transfers the file out from SVT Encore and to our destination bucket.

First we need to configure the transfer job service with API secrets needed for the access to the S3 bucket. Navigate to the Retransfer service in Open Source Cloud and click on the tab Secrets.

Create the secrets containing the Access Key Id and Secret Access Key for the destination storage access. Note down the name of these secrets as you will be using it later.


awsaccesskeyid
awssecretaccesskey


Now navigate to the Encore Transfer service in the web user interface and click on the tab Secrets. Add a secret with your personal access token (OSC token) that you find under Settings and the tab API.

Step 5: Create Encore Transfer service

When the service is created and saved we can now move on with creating the Encore Transfer service. Enter the name of the instance, Redis URL, name of queue in Redis (Valkey), output URL, OSC token and the name of the access key secrets in the Retransfer service. In this example we will have the following values:

Then press Create and wait for the instance to be ready.

Step 6: Submit a job

Now we are ready to try transcoding a video file that we have available on an S3 compatible storage. We will create signed URL to the video file we want to transcode. For example:


https://lab-testcontent-input.s3.eu-north-1.amazonaws.com/NO_TIME_TO_DIE_short_Trailer_2021.mp4?SIGNURLSTUFF


Navigate back to the SVT Encore service and press the menu item to open API docs again.

Click on the POST /encoreJobs bar and button Try it out and enter the following JSON

{
  "externalId": "blog",
  "profile": "program",
  "outputFolder": "/usercontent/blog",
  "baseName": "blog",
  "progressCallbackUri": "https://demo-blog.eyevinn-encore-callback-listener.auto.prod.osaas.io/encoreCallback",
  "inputs": [
    {
      "uri": "https://lab-testcontent-input.s3.eu-north-1.amazonaws.com/NO_TIME_TO_DIE_short_Trailer_2021.mp4?SIGNURL",
      "seekTo": 0,
      "copyTs": true,
      "type": "AudioVideo"
    }
  ]
}

And then press button Execute. Now a job is submitted and if you want to see the progress you can go to the Encore Callback Listener service and open the instance logs to check that it is receiving the callbacks.

When the transcoding process is completed it will place a job on the transfer queue that will be picked up by the Encore Transfer service. And when all the transfer jobs are completed you will in this example find a set of files in your output bucket where you have set of different variants with different resolutions and bitrates.

Conclusion

You now have a fully fledged video transcoding pipeline for preparing video files for streaming using ´SVT Encore with some supporting services. All based on open source and you don’t have to setup your own infrastructure for this to get started. If you later choose to do so you are free to do it as the code and everything demonstrated here is available as open source.