Author: Jonas Birmé

Posted on

AI Assisted Code Review

Based on open source made available as open web service in Eyevinn Open Source Cloud we can have AI to assist with code reviewing of submitted pull requests to a GitHub repository.

The AI Code Reviewer is an open source project that provides an API and user interface to review code in a public GitHub repository. It analyzes code for quality, best practices and potential improvements, providing actionable feedback to improve a code base. This is achieved by carefully prompting a GPT4 model to perform the task and return a structured response with scores and suggested improvements.

This project has been made available as an open web service in Eyevinn Open Source Cloud which means that you can instantly start to integrate this into your solution.

AI Code Review GitHub Action

For example, we might want to incorporate this AI Code Reviewer in our Pull Request workflow, and use this to provide an automated first review of all opened pull requests. It could add a comment on overall score and suggested improvements.

In order to add this to our GitHub workflow we need a GitHub action to perform this task based on this open web service in Eyevinn Open Source Cloud. We will create a GitHub action that uses the client libraries for Eyevinn OSC to create an AI Code Reviewer instance.

core.info('Setting up Code Reviewer');
const ctx = new Context();
let reviewer = await getEyevinnAiCodeReviewerInstance(ctx, 'ghaction');
if (!reviewer) {
  reviewer = await createEyevinnAiCodeReviewerInstance(ctx, {
    name: 'ghaction',
    OpenAiApiKey: '{{secrets.openaikey}}'
  });
  await delay(1000);
}
core.info(`Reviewer available, requesting review of ${gitHubUrl.toString()}`);

These lines of code will create an instance called “ghaction” if it does not already exists. When that is available we can use the API that this service provides to perform the actual code review. The following lines of codes takes care of this.

const reviewRequestUrl = new URL('/api/v1/review', reviewer.url);
const sat = await ctx.getServiceAccessToken('eyevinn-ai-code-reviewer');
const response = await fetch(reviewRequestUrl, {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${sat}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    githubUrl: gitHubUrl.toString()
  })
});
if (response.ok) {
  const review = await response.json();
  return review;
} else {
  throw new Error('Failed to get review');
}

We package this together into a GitHub action and makes it available on the GitHub action marketplace.

Add review to pull request workflow

Now it is time to add this to our pull request workflow in GitHub. We add a step to review the branch for the pull request using the GitHub action we created. The input variable “repo_url” contains the URL to this branch and in addition we need to provide the access token to Eyevinn OSC as an environment variable.

  - name: Review branch
    id: review
    uses: EyevinnOSC/code-review-action@v1
    with:
      repo_url: ${{ github.server_url }}/${{ github.repository }}/tree/${{ github.head_ref}}
    env:
      OSC_ACCESS_TOKEN: ${{ secrets.OSC_ACCESS_TOKEN }}

Next step is to take the outputs “score” and “suggestions” of this step and add this as a comment to the pull request. 

  - name: comment
    uses: actions/github-script@v7
    with:
      github-token: ${{secrets.GITHUB_TOKEN}}
      script: |
        github.rest.issues.createComment({
          issue_number: context.issue.number,
          owner: context.repo.owner,
          repo: context.repo.repo,
          body: 'Code review score: ${{ steps.review.outputs.score }}\n${{ join(fromJSON(steps.review.outputs.suggestions), ', ') }}'
        })

When a pull request is open the workflow is run and result of the code review is posted as a comment to the pull request.

Conclusion

This is an example on how you can integrate open web services to enhance your software development processes. We are continuing our journey to advance and democratize web services through open source and a sustainable business model for creators.

Posted on

Empowers Developers to Integrate Open Web Services into their Applications

This blog post serves as an example of how our platform empowers developers and businesses to seamlessly integrate open source as web services into their applications and services. Build applications and solutions on these open web services to avoid being locked in to a single web services vendor.

In this example we will build a NodeJS application that uses an open web service to store application configurations. Before we begin and to follow this guide you will need to signup with Eyevinn Open Source Cloud to get the personal access token to access available web services. Navigate to the Web Console and register with your email. Signup is free and on the free plan you have access to create one open web service at the time. Upgrade to startup or business plan gives you access to use more open web services at the same time. Create a tenant and you are good to go.

You can now obtain the access token by navigating to Settings / API in the web console. Copy this and store in your shell’s environment in the environment variable OSC_ACCESS_TOKEN.

% export OSC_ACCESS_TOKEN=access-token-copied-above

Setup your Node/Typescript project and install the Typescript client SDK.

% npm install --save @osaas/client-services

Create a main function that will read a config value and if not existing it will save a default.

async function main() {
  const ctx = new Context();
  const service = await setup(ctx);
  console.log('Configuration UI available at:', service.url);
  let value = await readConfigVariable(service, 'foo');
  if (!value) {
    await saveConfigVariable(service, 'foo', 'default');
    value = await readConfigVariable(service, 'foo');
  }
  console.log(`Config value: ${value}`);
}

Let us go through in more detail what above does.

This line will read the OSC_ACCESS_TOKEN environment variable from your shell and create a context for accessing the Eyevinn Open Source Cloud platform.

const ctx = new Context();

Then we will setup the open web services that we will need.

const service = await setup(ctx);

In return we will get the open web service handling the configuration variables and a URL to the configuration service web user interface. In this web interface you can manage the values of the configurations.

  let value = await readConfigVariable(service, 'foo');
  if (!value) {
    await saveConfigVariable(service, 'foo', 'default');
    value = await readConfigVariable(service, 'foo');
  }
  console.log(`Config value: ${value}`);

We try to read a variable called foo and if it does not exists we will create a variable with a default value. We then print the value to the console.

Now let us take a closer look at the function setup().

In this function we create the open web service for managing configuration variables. An open web service created from the open source project App Config Svc available on GitHub. This service requires a Redis database for storing and accessing the values.

For the database we will then create an instance of the Valkey.io open web service that provides a Redis compatible API. We obtain the IP and ports to this instance and creates a Redis URL that we provide the application config service we want to create.

async function setup(ctx: Context) {
  const configServiceAccessToken = await ctx.getServiceAccessToken(
    'eyevinn-app-config-svc'
  );
  let configService: EyevinnAppConfigSvc = await getInstance(
    ctx,
    'eyevinn-app-config-svc',
    'example',
    configServiceAccessToken
  );
  if (!configService) {
    const valkeyInstance = await createValkeyIoValkeyInstance(ctx, {
      name: 'configstore'
    });
    const redisUrl = await getRedisUrlFromValkeyInstance(
      ctx,
      valkeyInstance.name
    );
    configService = await createEyevinnAppConfigSvcInstance(ctx, {
      name: 'example',
      RedisUrl: redisUrl
    });
  }
  return configService;
}

The functions to save and read configuration variables are written as followed. Basically just using the HTTP API that the configuration service provides for writing and reading variables.

async function saveConfigVariable(service: EyevinnAppConfigSvc, key: string, value: string) {
  const url = new URL('/api/v1/config', service.url);
  const response = await fetch(url.toString(), {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({ key, value })
  });
  if (!response.ok) {
    throw new Error(`Failed to save config: ${response.statusText}`);
  }
}

async function readConfigVariable(service: EyevinnAppConfigSvc, key: string) {
  const url = new URL(`/api/v1/config/${key}`, service.url);
  const response = await fetch(url.toString(), {
    method: 'GET',
    headers: {
      'Content-Type': 'application/json'
    }
  });
  if (!response.ok) {
    return undefined;
  }
  const data = await response.json();
  return data.value;
}

This example shows you how you can integrate open web services easily in your applications and services. More examples are available in this GitHub repository.

Posted on

Can Claude create a VOD streaming package for you?

The question in the title is of course a bit rhetorical. Of course Claude can. In this post I am going to describe how that works and how you can try this out.

Claude is an AI assistant built by Anthropic that is trained to have natural, text-based conversations, and first model was released in March 2023. Anthropic released in November 2024 a specification for the Model Context Protocol (MCP) that is an open protocol to enable seamless integration between LLM applications and external data sources and tools. MCP provides a standardized way to connect LLMs with the context they need.

MCP is a protocol that enables secure connections between host applications, such as Claude Desktop, and local services. Programs like Claude Desktop, IDEs or AI tools access MCP servers that are lightweight programs that exposes specific capabilities through the standardized Model Context Protocol.

We have developed and open sourced an MCP server for Eyevinn Open Source Cloud. An MCP server provides tools and resources and we currently provide tools for video on-demand streaming but more will be added by us our hopefully the open source community.

In the demonstration video below I show how I can have Claude to setup a video on-demand preparation pipeline and create a video on-demand file for streaming from a video file available online.


Install

If you want to try this out yourself you can follow these steps. A prerequisite is that you have an account on Eyevinn Open Source Cloud and at least 6 services available on your plan.

1. Download and install Claude Desktop.
2. In the Eyevinn OSC web console go to API settings (in Settings > API settings)
3. Copy the Personal Access Token
4. add the following to your claude_desktop_config.json:

{
  "mcpServers": {
    "eyevinn-osc": {
      "command": "npx",
      "args": ["-y", "@osaas/mcp-server"],
      "env": {
        "OSC_ACCESS_TOKEN": "YOUR_PERSONAL_ACCESS_TOKEN"
      }
    }
  }
}

5. Restart Claude

If everything is correctly installed you should see an icon of a hammer in the bottom of the chat input.

Now you can ask Claude to create a VOD from a file that you have available online as shown in the video above.

Client SDK

This MCP server uses the Typescript client SDK for Eyevinn Open Source Cloud. With this SDK you can create and remove instances and automate what you can do in the web console. Here is an example of how to create a VOD package using the client SDK which is basically what one of the tools currently can do.

import { Context, Log } from '@osaas/client-core';
import { createVod, createVodPipeline } from '@osaas/client-transcode';

async function main() {
  const ctx = new Context();

  try {
    const ctx = new Context({ environment });
    Log().info('Creating VOD pipeline');
    const pipeline = await createVodPipeline(name, ctx);
    Log().info('VOD pipeline created, starting job to create VOD');
    const job = await createVod(pipeline, source, ctx);
    if (job) {
      Log().info('Created VOD will be available at: ' + job.vodUrl);
    }
  } catch (err) {
    Log().error(err);
  }
}

main();

This gives you an example of what you can do and the possibilities are “endless”. It feels as it is only creativity that stands in the way of what you can do.

Share your ideas either in the comments below or with a contribution to the Eyevinn OSC MCP server that is open source. Be creative!

Posted on

Setup a Bluesky Personal Data Server in Open Source Cloud

Bluesky is a decentralized microblogging social media service based on open standards (AT Protocol) and open source infrastructure so that social communication can be as open and interoperable as the web itself. The AT Protocol (Authenticated Transfer Protocol aka atproto) is a federated protocol for large-scale distributed social applications.

The three core services in a network are Personal Data Server (PDS), Relays and App Views. A personal data server is your home in the cloud. This is the server that hosts your data, distribute it, manage your identity and orchestrate requests to other services to give you your views. However, the goal of the AT protocol is to ensure that a user on one PDS can move and migrate their account to a new PDS without the server’s involvement.

In this blog we will describe how you can setup your own Personal Data Server based on open source made available as a service all for free.

Step 1: Create an account in Eyevinn Open Source Cloud

Navigate to www.osaas.io and click on Login/Signup. Enter your email to create an account and enter the login code you receive in your inbox. If this is the first time you logged in you need to create a tenant first.

Step 2: Create your own PDS

Navigate to Bluesky Personal Data Server by entering this text in the search bar in the top bar.

Click on the tab Service secrets and click on New Secret to create a secret for your administration password.

Click on the button “Create pds” and enter the name of your PDS and a reference to the secret you created.

Leave the input DnsName empty for now. This will be used when you add a CDN in front of the server and use a custom domain name. Press create and wait for the indicator on the instance card to turn green.

Step 3: Create an invitation code

Now you have your own PDS up and running. To create an account on the server you need to first create an invitation code. This is done by sending an HTTP request to the PDS API. In this example we will use an HTTP API client available online.

Use Basic auth as authentication method and admin as user and the password is the administration password that you created above. As URL you enter the URL available on the instance card and add /xrpc/com.atproto.server.createInviteCode

In the body you enter the following JSON:

{ "useCount": 1 }

The code returned in the response is the invitation code, in this case demo-blog-bluesky-social-pds-auto-prod-osaas-io-5ito3-t5umt. This is the code you are using when creating an account on this server.

Step 4: Create an account

Download the Bluesky social app on your appstore. When registering a new account select a custom hosting provider and enter the URL to the PDS created. Use the invitation code and enter email and password. Now you will have an account created with a handle @.demo-blog.bluesky-social-pds.auto.prod.osaas.io and you are ready to go!

Advanced: Custom domain and CDN

To use a custom domain name for your service you need to be able to administer a DNS domain and CDN. We will not go through this in detail in this blog post. What you need to setup is the following:

  • 1. Decide and register a root domain name, e.g. my.org
  • 2. Decide what domain name you will use for the PDS, e.g. pds.my.org
  • 3. Create an SSL certificate for *.pds.my.org and pds.my.org
  • 4. Create a PDS in OSC as before with the addition that you set DNS_NAME to pds.my.org
  • 5. Setup a CDN property / distribution where origin is the URL to the PDS created above, e.g. demo-blog.bluesky-social-pds.auto.prod.osaas.io and use the SSL cert created in 3. It is important that the CDN uses the origin host in the request to the origin. Consult your CDN provider for how to configure this.
  • 6. Create DNS records *.pds.my.org and pds.my.org to point to the CDN distribution created in 5.

Conclusion

Creating your own Bluesky Personal Data Server based on open source is achievable with only a few click of a button and a quick way to get your own self-hosted account to join the conversation in this open social media network.

Posted on

Simplified access to cloud storages with Eyevinn OSC

There are several options on how to store files in the cloud today and in this blog post we will show how you with an open source project made available as a service in Eyevinn Open Source Cloud can simplify the access to the storage. In this blog we will as an example use Akamai S3 compatible Object Storage as the cloud storage.

Create storage bucket

Ref: https://techdocs.akamai.com/cloud-computing/docs/create-and-manage-buckets

1. Log in to Cloud Manager and select Object Storage from the left menu. If you currently have buckets on your account, they are listed on this page, along with their URL, region, size, and the number of objects (files) they contain.

2. One of the first steps to using Object Storage is to create a bucket. Here’s how to create a bucket using Cloud Manager, though you can also use the Linode CLI, s3cmd, and s4cmd.

3. Navigate to the Object Storage page in Cloud Manager (see View buckets).

4. Click the Create Bucket button to open the Create Bucket panel. If you have not created an access key or a bucket on this account, you are prompted to enable Object Storage.

5. Within the Create Bucket form, add a Label for the new bucket. This label must be unique and should not be used by any other bucket (from any customer) in the selected data center.

6. Choose a Region for the bucket to reside. See the Availability section on the Object Storage Overview page for a list of available regions.

7. Click Submit to create the bucket.

In this example we have created a bucket called “osc-blog” in the data center in Stockholm.

To be able to access this bucket we have created we need to create an access key. Navigate to Access Keys tab and press Create Access key. Give the access key a name and in this case we will limit the access to only the bucket we created.

Copy and store the generated “access key id” and “secret key” as you will use these later.

Setup Cloud Storage Manager

In Eyevinn Open Source Cloud web console navigate to the service called Filestash and press “Create filestash”.

Give the service a name for example “blog” in this case. Click on the instance card once it is in state running. A new page will open in a new tab or browser window. Then enter an administrator password for this Filestash storage manager instance.

In the navigation sidebar on the left click on the item “Backend”. Select S3 as storage backend.

You may remove the others as we will be only be using S3 in this example.

For simplicity we will be using the ADMIN authentication middleware. This means that you will login with the admin password you just created. You might at least want to use HTPASSWD for more granular access control in practice.

Enter the access key id and secret key.

The endpoint in this case is https://se-sto-1.linodeobjects.com as the bucket is located in region se-sto-1.

Upload a file

Now go back to the start page by clicking on the instance card and login with the admin password that you created.

Now you can upload a file by using drag-and-drop.

Conclusion

With this open source project now made available as a service in Eyevinn Open Source Cloud you can give your users a simpler and consistent user interface independent from what cloud storage provider you are using. Using Eyevinn Open Source Cloud you contribute back to a sustainable business model for open source as a share of the revenue is shared with the open source creator.

Posted on

Open source databases available as a service

In this blog post we will go through the open source databases that are available as a service in Eyevinn Open Source Cloud.

Databases are fundamental in many solutions and some of the open source projects that are available as a service in Eyevinn Open Source Cloud depends on a database for storing data and states. There are a great number of databases to choose from and recent years a lot of open source alternatives have emerged. With open source you are not locked in with a single vendor but it requires you to host and manage it yourself. To reduce this barrier we have a few of these open source databases already made available as a service and more will be added. This enables you to run open source services in our platform that depends on a database without having to host and manage the database server yourself. It is of course possible to run the databases from another cloud provider if available. That choice is entirely up to you. And as with any other service in this platform we give a share of the revenue back to the original creators.

Let us go through what is available in Eyevinn Open Source Cloud today.

Valkey

Valkey is a Redis-compatible high-performance key-value store that can serve many purposes where simplicity and performance is of most importance. It can be used as a processing queue in a VOD transcoding and packaging solution as well as a store for application config service.

To create a Valkey instance simply navigate to the Valkey service in the Eyevinn Open Source Cloud web console and press Create. When the instance is created you obtain the IP and port to use on the instance card, e.g. redis://[IP]:[PORT].

MariaDB

MariaDB is a relational databases made by the original developers of MySQL and guaranteed to stay open source. It can be used as the database for a WordPress blog for example the blog you are currently reading from. This blog is powered by WordPress and MariaDB in Eyevinn Open Source Cloud (dogfooding). Another example is the database for Suite CRM available here.

To create a MariaDB database instance navigate to the MariaDB service in the web console and press Create. Enter the root password and database users you want to setup. These credentials are then used to connect to the database from the application. Obtain the IP and port on the instance card when constructing the connection URL.

PostgreSQL

Another object-relational database that is open source is PostgreSQL. The origins of PostgreSQL date back to 1986 as part of the POSTGRES project at the University of California at Berkeley and has more than 35 years of active development on the core platform. Navigate to PostgreSQL service in the Open Source Cloud web console and press the button Create to launch a new database instance.

Enter the credentials and name of the database and press create.

Couch DB

Apache CouchDB is an open source NoSQL document database that collects and stores data in JSON-based document formats and works well with modern web and mobile applications. Access your document with your web browser via HTTP. The CouchDB API is the primary method of interfacing to a CouchDB instance. Requests are made using HTTP and requests are used to request information from the database, store new data, and perform views and formatting of the information stored within the documents. Simple and easy to use with any HTTP client.

Navigate to the CouchDB service in Open Source Cloud and press Create to start a new instance.

When the instance is up and running you can click on the instance card to go to the web user interface to Couch DB.

Create a new database by clicking the Create database button in the top right corner. Then you can create your first document that you want to store.

There are client libraries available and the offical Apache CouchDB library for Node.js is called nano.

Conclusion

These are the open source databases that are available as service in Eyevinn Open Source Cloud today. If you have a suggestion of another open source database that can be made available as a service go to www.osaas.io and submit it there, or write a comment to this post.

Posted on

VOD File Creation with Open Source Cloud

In a previous blog post we provided a walk-through on how to setup video file transcoding using Open Source Cloud based on SVT Encore and supporting backend services. In this blog post we are extending the setup by adding the creation of video-on-demand streaming files to the pipeline.

In this solution we will add another open source project made available as a service. The Encore Packager is a backend service that creates the VOD file package. It consumes jobs from a Redis queue and creates the VOD file package and uploads the package to an S3 bucket. For the creation of the VOD file package the open source packager Shaka Packager is used. The red box in the diagram below shows what we will add to our solution.

Step 1: Create another Valkey queue

Valkey provides a Redis compatible key / value store and we will create another queue for the packaging jobs. Navigate to the Valkey service in Open Source Cloud and press “Create valkey”. Give the instance a name and press Create.

Note down the IP and port to the Valkey instance card and this is what will be the Redis URL that we will refer to later in this blog. In this example it would be redis://172.232.131.169:10511.

Step 2: Launch another Encore Callback Listener

We will now create a separate service that can be used to monitor a transcoding job in SVT Encore so we know when the file is ready to be packaged. Navigate to the Encore Callback Listener in the web user interface. Click on button “Create callback” and enter the name of the instance, Redis URL (above), URL to the SVT Encore instance that we created last time and the name of the queue. We will call this queue for “package” now.

Important the URL to the SVT Encore instance is without the trailing slash.

Step 3: Create Encore Packager service

We can now move on with creating the Encore Packager service. Enter the name of the instance, Redis URL, name of queue in Redis (Valkey), output S3 URL, OSC token and the AWS credentials for the output S3 bucket. In this example we will have the following values:

Then press Create and wait for the instance to be ready.

Step 4: Submit a job

Now we are ready to try transcoding and creating a VOD package that we have available on an S3 compatible storage. We will create signed URL to the video file we want to transcode. For example:

https://lab-testcontent-input.s3.eu-north-1.amazonaws.com/NO_TIME_TO_DIE_short_Trailer_2021.mp4?SIGNURLSTUFF

Navigate back to the SVT Encore service and press the menu item to open API docs again. Click on the POST /encoreJobs bar and button “Try it out” and enter the following JSON. Here we have changed the progressCallbackUri to point to our Encore Callback Listener for VOD packaging.

{
  "externalId": "blog",
  "profile": "program",
  "outputFolder": "/usercontent/",
  "baseName": "blog",
  "progressCallbackUri": "https://demo-vod.eyevinn-encore-callback-listener.auto.prod.osaas.io/encoreCallback",
  "inputs": [
    {
      "uri": "https://lab-testcontent-input.s3.eu-north-1.amazonaws.com/NO_TIME_TO_DIE_short_Trailer_2021.mp4?SIGNURL",
      "seekTo": 0,
      "copyTs": true,
      "type": "AudioVideo"
    }
  ]
}

And then press button Execute. Now a job is submitted and if you want to see the progress you can go to the Encore Callback Listener service and open the instance logs to check that it is receiving the callbacks.

When the transcoding process is completed it will place a job on the packaging queue that will be picked up by the Encore Packager service. And when the packaging job is completed you will in this example find a VOD package ready for streaming: https://lab.cdn.eyevinn.technology/osc/NO_TIME_TO_DIE_short_Trailer_2021/bb347d8e-c095-43dc-ba5f-914c7e74f13d/index.m3u8.

Conclusion

You now have a fully fledged video transcoding and packaging pipeline for preparing video files for streaming using ´SVT Encore with some supporting services. All based on open source and you don’t have to setup your own infrastructure for this to get started. If you later choose to do so you are free to do it as the code and everything demonstrated here is available as open source.

Posted on

Scheduled MariaDB backup using GitHub action and job in Eyevinn OSC

This blog gives an example on how to run regular database backups of your MariaDB database where the result is uploaded to an S3 compatible bucket.

For the task to perform the database backup and upload the result to S3 we will be using an open source script that is made available in Eyevinn Open Source Cloud. To launch this on a regular basis we will use a scheduled GitHub workflow that uses Eyevinn OSC action to create the jobs in OSC.

Step 1: setup secrets

Navigate to the service called “MariaDB backup to S3” and select the tab “Service Secrets”.

Create a secret for the URL to the database you want to backup. In the screenshot above we have a secret for the MariaDB url that this blog is running on. A URL is in the form mariadb://root:[rootpassword]@[host]:[ip]/[database]

Then we need secrets for the credentials to the AWS S3 bucket where we will place the backup. In this example they are called “eyevinnawskeyid” and “eyevinnawssecret”.

Step 2: create a test job

To test that everything is setup correctly we will manually create a test job.

Press Create and verify that a backup is taken and the result ends up on the S3 bucket.

Step 3: create a GitHub workflow file

In this example we will be using a GitHub workflow schedule to create a backup job in Open Source Cloud.

This workflow uses the GitHub action Eyevinn OSC action available on the GitHub action marketplace.

Conclusion

This provided an example for how you can launch jobs for an open source project that is made available as a service in Eyevinn Open Source Cloud as a step in a GitHub workflow.

Posted on

Trim video file on an S3 compatible bucket using open source

In this blog post I will describe how to trim a video file on an S3 compatible bucket using ffmpeg without having to download it first, process it and then upload the result.

For trimming the video we will use the open source tool ffmpeg and a script that handles uploading the result to an S3 bucket. This open source script is available as a service in Open Source Cloud.

Step 1: Login to Eyevinn Open Source Cloud

Go to www.osaas.io and login. Sign up for an account if you don’t already have one. It is free to get started and you don’t even have to enter a credit card to try this out.

Step 2: Setup access to S3 bucket

Go to the service in Open Source Cloud called “FFmpeg to S3” using the search bar on the browse page. Click on the card “FFmpeg to S3 to go to the service.

Then click on the tab named “Service secrets”

Get the S3 access key credentials from your administrator of your S3 buckets. You need at minimum the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. Create a service secret for each of these credentials.

Step 3: Generate signed URL to the video to trim

Now we need to generate a signed URL for the video that you want to trim.

Copy the presigned URL to the clipboard

Step 4: Create a ffmpeg trim job

As an example we will extract the first 30 seconds of the video file and the ffmpeg command for that is:

ffmpeg -ss 0 -t 30 -c:v copy -c:a copy

Go back to the FFmpeg to S3 service page in Open Source Cloud and click on button “Create job”.

Enter the following in the settings dialog:

Name: “tutorial”
CmdLineArgs:
Replace [SIGNED_URL] from clipboard and lab-testcontent-input with the name of your bucket:

-i [SIGNED-URL] -d s3://lab-testcontent-input/tutorial-30sec.mp4 "-ss 0 -t 30 -c:v copy -c:a copy"

AwsAccessKeyId and AwsSecretAccessKey: reference to the service secrets created
Region: Location of the S3 bucket

Now press Create and wait for the job to be completed. When the job is completed you should have a file called tutorial-30sec.mp4 on the bucket you provided and 30 seconds duration.

Create a job from command line

You might want to automate or script the creation of these ffmpeg jobs and to facilitate that there is an open source SDK and command line tool for Eyevinn OSC. The command line tool is a Node.js script.

Follow the instructions on how to install Node.js if you don’t already have it installed.

Then install the CLI


% npm install -g @osaas/cli

Obtain the personal access token by going to Settings in OSC and the tab API. Here you find the personal access token that you copy to your clipboard. Set this token as an environment variable in your shell.


% export OSC_ACCESS_TOKEN=token

Now you can create the same job with the following command (replace [SIGNED-URL] and s3 bucket):


% osc create eyevinn-ffmpeg-s3 tutorialcli -o awsAccessKeyId="{{secrets.eyevinnawskeyid}}" -o awsSecretAccessKey="{{secrets.eyevinnawssecret}}" -o cmdLineArgs='-i [SIGNED-URL] -d s3://lab-testcontent-input/tutorial-30sec.mp4 "-ss 0 -t 30 -c:v copy -c:a copy"'

Conclusion

This was an example of how you can run ffmpeg to process a video file on an S3 bucket and output the result back to an S3 bucket without having to develop your own script for it as a script already existed that is open source and made available as a service in Eyevinn Open Source Cloud.

Posted on

Video File Transcoding with Open Source Cloud

SVT Encore is a powerful open-source video transcoder specifically designed for the cloud. It forms the backbone of the transcoding process in the media supply chain, taking raw video inputs and converting them into multiple formats and bitrates suitable for adaptive streaming. The transcoding process involves breaking down video files into different resolutions and bitrates, allowing viewers to receive the best possible quality based on their device and network conditions.

To reduce the barrier to get started with SVT Encore we have added their project to Open Source Cloud together with some supporting backend services that we have added. This blog gives you a walk-through on how to setup video file transcoding using Open Source Cloud.

Prerequisites

  • If you have not already done so, sign up for an OSC account.
  • 5 remaining services on your subscription plan or individually purchased the services included in this solution.
  • S3 compatible object storage solution

This solution is based on the following open source projects made available as a service:

  • SVT Encore
  • Valkey
  • Encore Callback Listener
  • Encore Transfer
  • Retransfer

After completed this tutorial you will be able to transcode a video file on an S3 compatible storage and the output is placed on another S3 compatible storage when the processing is completed.

Step 1: Create Encore Queue

Go to the web user interface and navigate to the service called SVT Encore. Click on the button “Create queue” and give the queue a name.

You can leave the Profiles URL empty for now and then press Create.

Now you have an instance of SVT Encore running with one single queue and ready to receive transcoding jobs for processing. You can try this out by clicking on the menu item Open API docs to access the online REST API documentation and submit a job.

However, to automatically get transcoded files out from SVT Encore and transferred to the output storage we need a few more help services. So that we will setup now. Start by take a note of the URL to the SVT Encore instance.

Remove the trailing slash an keep it for later use. In this case it is https://demo-blog.encore.prod.osaas.io.

Step 2: Create Valkey queue

Valkey provides a Redis compatible key / value store and this i what we will use to manage the queue for transferring files out from Encore and to out output bucket.

Navigate to the Valkey service in Open Source Cloud and press “Create valkey”. Give the instance a name and press Create.

Note down the IP and port to the Valkey instance card and this is what will be the Redis URL that we will refer to later in this blog. In this example it would be redis://172.232.131.169:10507.

Step 3: Launch Encore Callback Listener

Now we need something that monitors a transcoding job in SVT Encore so we know when the file is ready to be transferred. For that you navigate to the Encore Callback Listener in the web user interface. Click on button “Create callback” and enter the name of the instance, Redis URL (above), URL to the SVT Encore instance and the name of the transfer queue. We call it “transfer” in this example.

Important the URL to the SVT Encore instance is without the trailing slash.

Press Create and you are done with this step for now.

Step 4: Setup secrets

Now we have the Callback Listener service running that will monitor transcoding job and place completed jobs in the transfer queue. Now we need a service that picks up a job from the transfer queue and actually transfers the file out from SVT Encore and to our destination bucket.

First we need to configure the transfer job service with API secrets needed for the access to the S3 bucket. Navigate to the Retransfer service in Open Source Cloud and click on the tab Secrets.

Create the secrets containing the Access Key Id and Secret Access Key for the destination storage access. Note down the name of these secrets as you will be using it later.


awsaccesskeyid
awssecretaccesskey

Now navigate to the Encore Transfer service in the web user interface and click on the tab Secrets. Add a secret with your personal access token (OSC token) that you find under Settings and the tab API.

Step 5: Create Encore Transfer service

When the service is created and saved we can now move on with creating the Encore Transfer service. Enter the name of the instance, Redis URL, name of queue in Redis (Valkey), output URL, OSC token and the name of the access key secrets in the Retransfer service. In this example we will have the following values:

Then press Create and wait for the instance to be ready.

Step 6: Submit a job

Now we are ready to try transcoding a video file that we have available on an S3 compatible storage. We will create signed URL to the video file we want to transcode. For example:


https://lab-testcontent-input.s3.eu-north-1.amazonaws.com/NO_TIME_TO_DIE_short_Trailer_2021.mp4?SIGNURLSTUFF

Navigate back to the SVT Encore service and press the menu item to open API docs again.

Click on the POST /encoreJobs bar and button Try it out and enter the following JSON

{
  "externalId": "blog",
  "profile": "program",
  "outputFolder": "/usercontent/blog",
  "baseName": "blog",
  "progressCallbackUri": "https://demo-blog.eyevinn-encore-callback-listener.auto.prod.osaas.io/encoreCallback",
  "inputs": [
    {
      "uri": "https://lab-testcontent-input.s3.eu-north-1.amazonaws.com/NO_TIME_TO_DIE_short_Trailer_2021.mp4?SIGNURL",
      "seekTo": 0,
      "copyTs": true,
      "type": "AudioVideo"
    }
  ]
}


And then press button Execute. Now a job is submitted and if you want to see the progress you can go to the Encore Callback Listener service and open the instance logs to check that it is receiving the callbacks.

When the transcoding process is completed it will place a job on the transfer queue that will be picked up by the Encore Transfer service. And when all the transfer jobs are completed you will in this example find a set of files in your output bucket where you have set of different variants with different resolutions and bitrates.

Conclusion

You now have a fully fledged video transcoding pipeline for preparing video files for streaming using ´SVT Encore with some supporting services. All based on open source and you don’t have to setup your own infrastructure for this to get started. If you later choose to do so you are free to do it as the code and everything demonstrated here is available as open source.

Proudly powered by WordPress