Open Source Cloud Blog

In this blog post I will describe how to trim a video file on an S3 compatible bucket using ffmpeg without having to download it first, process it and then upload the result.

For trimming the video we will use the open source tool ffmpeg and a script that handles uploading the result to an S3 bucket. This open source script is available as a service in Open Source Cloud.

Step 1: Login to Eyevinn Open Source Cloud

Go to www.osaas.io and login. Sign up for an account if you don’t already have one. It is free to get started and you don’t even have to enter a credit card to try this out.

Step 2: Setup access to S3 bucket

Go to the service in Open Source Cloud called “FFmpeg to S3” using the search bar on the browse page. Click on the card “FFmpeg to S3 to go to the service.

Then click on the tab named “Service secrets”

Get the S3 access key credentials from your administrator of your S3 buckets. You need at minimum the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. Create a service secret for each of these credentials.

Step 3: Generate signed URL to the video to trim

Now we need to generate a signed URL for the video that you want to trim.

Copy the presigned URL to the clipboard

Step 4: Create a ffmpeg trim job

As an example we will extract the first 30 seconds of the video file and the ffmpeg command for that is:

ffmpeg -ss 0 -t 30 -c:v copy -c:a copy

Go back to the FFmpeg to S3 service page in Open Source Cloud and click on button “Create job”.

Enter the following in the settings dialog:

Name: “tutorial”
CmdLineArgs:
Replace [SIGNED_URL] from clipboard and lab-testcontent-input with the name of your bucket:

-i [SIGNED-URL] -d s3://lab-testcontent-input/tutorial-30sec.mp4 "-ss 0 -t 30 -c:v copy -c:a copy"

AwsAccessKeyId and AwsSecretAccessKey: reference to the service secrets created
Region: Location of the S3 bucket

Now press Create and wait for the job to be completed. When the job is completed you should have a file called tutorial-30sec.mp4 on the bucket you provided and 30 seconds duration.

Create a job from command line

You might want to automate or script the creation of these ffmpeg jobs and to facilitate that there is an open source SDK and command line tool for Eyevinn OSC. The command line tool is a Node.js script.

Follow the instructions on how to install Node.js if you don’t already have it installed.

Then install the CLI


% npm install -g @osaas/cli


Obtain the personal access token by going to Settings in OSC and the tab API. Here you find the personal access token that you copy to your clipboard. Set this token as an environment variable in your shell.


% export OSC_ACCESS_TOKEN=token


Now you can create the same job with the following command (replace [SIGNED-URL] and s3 bucket):


% osc create eyevinn-ffmpeg-s3 tutorialcli -o awsAccessKeyId="{{secrets.eyevinnawskeyid}}" -o awsSecretAccessKey="{{secrets.eyevinnawssecret}}" -o cmdLineArgs='-i [SIGNED-URL] -d s3://lab-testcontent-input/tutorial-30sec.mp4 "-ss 0 -t 30 -c:v copy -c:a copy"'


Conclusion

This was an example of how you can run ffmpeg to process a video file on an S3 bucket and output the result back to an S3 bucket without having to develop your own script for it as a script already existed that is open source and made available as a service in Eyevinn Open Source Cloud.

SVT Encore is a powerful open-source video transcoder specifically designed for the cloud. It forms the backbone of the transcoding process in the media supply chain, taking raw video inputs and converting them into multiple formats and bitrates suitable for adaptive streaming. The transcoding process involves breaking down video files into different resolutions and bitrates, allowing viewers to receive the best possible quality based on their device and network conditions.

To reduce the barrier to get started with SVT Encore we have added their project to Open Source Cloud together with some supporting backend services that we have added. This blog gives you a walk-through on how to setup video file transcoding using Open Source Cloud.

Prerequisites

• If you have not already done so, sign up for an OSC account.
• 5 remaining services on your subscription plan or individually purchased the services included in this solution.
• S3 compatible object storage solution

This solution is based on the following open source projects made available as a service:

• SVT Encore
• Valkey
• Encore Callback Listener
• Encore Transfer
• Retransfer

After completed this tutorial you will be able to transcode a video file on an S3 compatible storage and the output is placed on another S3 compatible storage when the processing is completed.

Step 1: Create Encore Queue

Go to the web user interface and navigate to the service called SVT Encore. Click on the button “Create queue” and give the queue a name.

You can leave the Profiles URL empty for now and then press Create.

Now you have an instance of SVT Encore running with one single queue and ready to receive transcoding jobs for processing. You can try this out by clicking on the menu item Open API docs to access the online REST API documentation and submit a job.

However, to automatically get transcoded files out from SVT Encore and transferred to the output storage we need a few more help services. So that we will setup now. Start by take a note of the URL to the SVT Encore instance.

Remove the trailing slash an keep it for later use. In this case it is https://demo-blog.encore.prod.osaas.io.

Step 2: Create Valkey queue

Valkey provides a Redis compatible key / value store and this i what we will use to manage the queue for transferring files out from Encore and to out output bucket.

Navigate to the Valkey service in Open Source Cloud and press “Create valkey”. Give the instance a name and press Create.

Note down the IP and port to the Valkey instance card and this is what will be the Redis URL that we will refer to later in this blog. In this example it would be redis://172.232.131.169:10507.

Step 3: Launch Encore Callback Listener

Now we need something that monitors a transcoding job in SVT Encore so we know when the file is ready to be transferred. For that you navigate to the Encore Callback Listener in the web user interface. Click on button “Create callback” and enter the name of the instance, Redis URL (above), URL to the SVT Encore instance and the name of the transfer queue. We call it “transfer” in this example.

Important the URL to the SVT Encore instance is without the trailing slash.

Press Create and you are done with this step for now.

Step 4: Setup secrets

Now we have the Callback Listener service running that will monitor transcoding job and place completed jobs in the transfer queue. Now we need a service that picks up a job from the transfer queue and actually transfers the file out from SVT Encore and to our destination bucket.

First we need to configure the transfer job service with API secrets needed for the access to the S3 bucket. Navigate to the Retransfer service in Open Source Cloud and click on the tab Secrets.

Create the secrets containing the Access Key Id and Secret Access Key for the destination storage access. Note down the name of these secrets as you will be using it later.


awsaccesskeyid
awssecretaccesskey


Now navigate to the Encore Transfer service in the web user interface and click on the tab Secrets. Add a secret with your personal access token (OSC token) that you find under Settings and the tab API.

Step 5: Create Encore Transfer service

When the service is created and saved we can now move on with creating the Encore Transfer service. Enter the name of the instance, Redis URL, name of queue in Redis (Valkey), output URL, OSC token and the name of the access key secrets in the Retransfer service. In this example we will have the following values:

Then press Create and wait for the instance to be ready.

Step 6: Submit a job

Now we are ready to try transcoding a video file that we have available on an S3 compatible storage. We will create signed URL to the video file we want to transcode. For example:


https://lab-testcontent-input.s3.eu-north-1.amazonaws.com/NO_TIME_TO_DIE_short_Trailer_2021.mp4?SIGNURLSTUFF


Navigate back to the SVT Encore service and press the menu item to open API docs again.

Click on the POST /encoreJobs bar and button Try it out and enter the following JSON

{
  "externalId": "blog",
  "profile": "program",
  "outputFolder": "/usercontent/blog",
  "baseName": "blog",
  "progressCallbackUri": "https://demo-blog.eyevinn-encore-callback-listener.auto.prod.osaas.io/encoreCallback",
  "inputs": [
    {
      "uri": "https://lab-testcontent-input.s3.eu-north-1.amazonaws.com/NO_TIME_TO_DIE_short_Trailer_2021.mp4?SIGNURL",
      "seekTo": 0,
      "copyTs": true,
      "type": "AudioVideo"
    }
  ]
}

And then press button Execute. Now a job is submitted and if you want to see the progress you can go to the Encore Callback Listener service and open the instance logs to check that it is receiving the callbacks.

When the transcoding process is completed it will place a job on the transfer queue that will be picked up by the Encore Transfer service. And when all the transfer jobs are completed you will in this example find a set of files in your output bucket where you have set of different variants with different resolutions and bitrates.

Conclusion

You now have a fully fledged video transcoding pipeline for preparing video files for streaming using ´SVT Encore with some supporting services. All based on open source and you don’t have to setup your own infrastructure for this to get started. If you later choose to do so you are free to do it as the code and everything demonstrated here is available as open source.

As live streaming continues to grow in popularity across various platforms, building a scalable, resilient, and cost-effective architecture is essential for broadcasters and content providers. Whether you’re streaming live sports, news, or entertainment, ensuring that your content reaches audiences globally with minimal latency and interruptions is critical.

With Open Source Cloud, you can build a live streaming architecture that leverages open-source components to offer scalability, resilience, and customization — all at a lower cost.

Live Transcoding and CDN pull stream

In this scenario the CDN pulls the stream from the encoder instance in Open Source Cloud. We strongly recommend to place a CDN origin shield between the CDN distribution nodes and the encoder instance acting as origin.

This tutorial walks you through how to setup live transcoding using Open Source Cloud.

Prerequisites

• If you have not already done so, sign up for an OSC account.
• Be on a paid subscription plan.

For best guaranteed performance we recommend purchasing a reservation for at least 1 single premium node (200 EUR per month) but not required to getting started.

Step 1: Setup secrets

Go to the web user interface and navigate to the service called Eyevinn Live Encoding. Click on the tab Service Secrets and the button New Secret.

Create a secret called streamkeya containing mysecretkey. This will be the stream key the live production software will need to be able to push the stream to the encoder.

Step 2: Create encoder instance

In the web user interface click on the button “Create encoder +” and enter the following in the instance creation dialog.

• Name: guide
• HlsOnly: true
• StreamKey: {{secrets.streamkeya}}

You will create one encoder instance for each live stream you want to distribute.

Step 3: Start encoder

Once the instance is up and running you can use the provided web user interface to start and stop the encoder.

Click on the instance card to open the web user interface and for automation and integration with API you find the live API documentation by clicking on the menu item Open API Docs.There you are presented with a web based REST API client ready to use.

To start the encoder click on the button START ENCODER. When the encoder is in status “starting” it is ready to receive the stream.

On the service card you find an IP and port in the format IP:PORT and these values are used when configuring your live production software (for example OBS). For example you enter rtmp://IP:PORT/live/mysecretkey as the stream destination in OBS.

As soon as the stream is being received by the encoder you will find a video playing the encoded stream.

Step 4: Watch the stream

You can play the HLS stream directly from the origin by opening an HLS capable video player to the address returned by the OSC tool. In this example it was https://demo-guide.eyevinn-live-encoding.auto.prod.osaas.io/origin/hls/index.m3u8.

Step 5: Stop the stream

When the live event has been completed you stop the stream in the live production software and when that is stopped you can stop the encoder by clicking on the button STOP ENCODER.

Step 6: Configure CDN

Configure your CDN and CDN origin shield to fetch the stream from the URL that you tested with your video player as a media origin.