Category: Automation

Posted on

AI Assisted Code Review

Based on open source made available as open web service in Eyevinn Open Source Cloud we can have AI to assist with code reviewing of submitted pull requests to a GitHub repository.

The AI Code Reviewer is an open source project that provides an API and user interface to review code in a public GitHub repository. It analyzes code for quality, best practices and potential improvements, providing actionable feedback to improve a code base. This is achieved by carefully prompting a GPT4 model to perform the task and return a structured response with scores and suggested improvements.

This project has been made available as an open web service in Eyevinn Open Source Cloud which means that you can instantly start to integrate this into your solution.

AI Code Review GitHub Action

For example, we might want to incorporate this AI Code Reviewer in our Pull Request workflow, and use this to provide an automated first review of all opened pull requests. It could add a comment on overall score and suggested improvements.

In order to add this to our GitHub workflow we need a GitHub action to perform this task based on this open web service in Eyevinn Open Source Cloud. We will create a GitHub action that uses the client libraries for Eyevinn OSC to create an AI Code Reviewer instance.

core.info('Setting up Code Reviewer');
const ctx = new Context();
let reviewer = await getEyevinnAiCodeReviewerInstance(ctx, 'ghaction');
if (!reviewer) {
  reviewer = await createEyevinnAiCodeReviewerInstance(ctx, {
    name: 'ghaction',
    OpenAiApiKey: '{{secrets.openaikey}}'
  });
  await delay(1000);
}
core.info(`Reviewer available, requesting review of ${gitHubUrl.toString()}`);

These lines of code will create an instance called “ghaction” if it does not already exists. When that is available we can use the API that this service provides to perform the actual code review. The following lines of codes takes care of this.

const reviewRequestUrl = new URL('/api/v1/review', reviewer.url);
const sat = await ctx.getServiceAccessToken('eyevinn-ai-code-reviewer');
const response = await fetch(reviewRequestUrl, {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${sat}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    githubUrl: gitHubUrl.toString()
  })
});
if (response.ok) {
  const review = await response.json();
  return review;
} else {
  throw new Error('Failed to get review');
}

We package this together into a GitHub action and makes it available on the GitHub action marketplace.

Add review to pull request workflow

Now it is time to add this to our pull request workflow in GitHub. We add a step to review the branch for the pull request using the GitHub action we created. The input variable “repo_url” contains the URL to this branch and in addition we need to provide the access token to Eyevinn OSC as an environment variable.

  - name: Review branch
    id: review
    uses: EyevinnOSC/code-review-action@v1
    with:
      repo_url: ${{ github.server_url }}/${{ github.repository }}/tree/${{ github.head_ref}}
    env:
      OSC_ACCESS_TOKEN: ${{ secrets.OSC_ACCESS_TOKEN }}

Next step is to take the outputs “score” and “suggestions” of this step and add this as a comment to the pull request. 

  - name: comment
    uses: actions/github-script@v7
    with:
      github-token: ${{secrets.GITHUB_TOKEN}}
      script: |
        github.rest.issues.createComment({
          issue_number: context.issue.number,
          owner: context.repo.owner,
          repo: context.repo.repo,
          body: 'Code review score: ${{ steps.review.outputs.score }}\n${{ join(fromJSON(steps.review.outputs.suggestions), ', ') }}'
        })

When a pull request is open the workflow is run and result of the code review is posted as a comment to the pull request.

Conclusion

This is an example on how you can integrate open web services to enhance your software development processes. We are continuing our journey to advance and democratize web services through open source and a sustainable business model for creators.

Posted on

Video File Transcoding with Open Source Cloud using Terraform

In a previous post, Video File Transcoding with Open Source Cloud, we discussed how to set up a fully-fledged video transcoding pipeline using SVT Encore. In this follow-up, we will walk through the process of recreating that setup using Terraform, an open-source infrastructure-as-code (IaC) tool from HashiCorp, or its open-source alternative, OpenTofu.

Terraform enables you to automate the creation, management, and maintenance of infrastructure by defining it in code. By using HCL (HashiCorp Configuration Language), you can declaratively describe resources like servers, databases, and networks, and Terraform will handle their provisioning across a variety of platforms.

OSC Terraform Provider

To simplify interaction with Open Source Cloud (OSC), we’ve created a Terraform provider. This allows you to easily spin up or tear down OSC resources directly through Terraform.

In this post, we’ll define a Terraform configuration file to describe the video file transcoding pipeline.

Prerequisites

The Terraform Configuration

Create a new Terraform file named main.tf. We’ll begin by defining the required provider for OSC.

terraform {
  required_providers {
    osc = {
      source = "EyevinnOSC/osc"
      version = "0.1.3"
    }
  }
}

provider "osc" {
  pat        = <PERSONAL_ACCESS_TOKEN>
  environment = "prod"
}

Using Variables for Credentials

To avoid accidentally exposing sensitive credentials, we’ll use a variable for the Personal Access Token (PAT). This approach makes it easier to manage credentials securely. We also define the OSC environment as a variable to easily switch between devand prod.

variable "osc_pat" {
  type = string
}
variable "osc_env" {
  value = string
  environment = "dev"

You can then reference this variable in the provider block like so:

provider "osc" {
  pat        = var.osc_pat
  environment = var.osc_env
}

Setting the Token

To set the osc_pat variable, you can either pass it via the command line using the -var flag or set it as an environment variable:

export TF_VAR_osc_pat=<PERSONAL_ACCESS_TOKEN>

Define the SVT Encore Resource

The first step in the transcoding pipeline is setting up an SVT Encore instance. This resource requires a name and, optionally, a profiles_url where transcoding profiles are stored.

Define the osc_encore_instance resource like this:

resource "osc_encore_instance" "example" {
  name          = "ggexample"
  profiles_url  = "https://raw.githubusercontent.com/Eyevinn/encore-test-profiles/refs/heads/main/profiles.yml"
}

Define the Valkey Instance

Next, we define a Valkey instance, which is a required component in the pipeline.

resource "osc_valkey_instance" "example" {
  name = "ggexample"
}

Define the Callback Listener

The Encore Callback Listener connects to both the Valkey and Encore instances. It requires the redis_url and encore_url, which are derived from the earlier resources.

The redis_url should be in the Redis-compatible format, and the encore_url should be formatted without a trailing slash:

resource "osc_encore_callback_instance" "example" {
  name         = "ggexample"
  redis_url    = format("redis://%s:%s", osc_valkey_instance.example.external_ip, osc_valkey_instance.example.external_port)
  encore_url   = trimsuffix(osc_encore_instance.example.url, "/")
  redis_queue  = "transfer"
}

Define the Retransfer Service

We also need to set up AWS secrets for the Retransfer service. To manage this securely, we’ll use variables for the AWS access key and secret, and local variables to store the names of the credentials.

variable "aws_keyid" {
  type = string
}

variable "aws_secret" {
  type = string
}

variable "aws_output" {
  type    = string
  default = "s3://path/to/bucket"
}

resource "osc_secret" "keyid" {
  service_ids  = ["eyevinn-docker-retransfer"]
  secret_name  = "awsaccesskeyid"
  secret_value = var.aws_keyid
}

resource "osc_secret" "secret" {
  service_ids  = ["eyevinn-docker-retransfer"]
  secret_name  = "awssecretaccesskey"
  secret_value = var.aws_secret
}

Define the Encore Transfer Service

Finally, we define the Encore Transfer service, which will manage the transfer of transcoded media files to the specified AWS S3 bucket.

resource "osc_encore_transfer_instance" "example" {
  name        = "ggexample"
  redis_url   = osc_encore_callback_instance.example.redis_url
  redis_queue = osc_encore_callback_instance.example.redis_queue
  output      = var.aws_output
  aws_keyid   = osc_secret.keyid.secret_name
  aws_secret  = osc_secret.secret.secret_name
  osc_token   = var.osc_pat
}

Define Outputs

To easily access dynamic endpoints, we can define outputs in Terraform. These outputs can be used in scripts or other automation tasks.

output "encore_url" {
  value = trimsuffix(osc_encore_instance.example.url, "/")
}
}

output "encore_name" {
  value = osc_encore_instance.example.name
}

output "callback_url" {
  value = trimsuffix(osc_encore_callback_instance.example.url, "/")
}

Full Configuration File (main.tf)

Here’s the complete main.tf file:

terraform {
  required_providers {
    osc = {
      source = "EyevinnOSC/osc"
      version = "0.1.3"
    }
  }
}

variable "osc_pat" {
  type      = string
  sensitive = true
}

variable "osc_environment" {
  type    = string
  default = "prod"
}

variable "aws_keyid" {
  type      = string
  sensitive = true
}

variable "aws_secret" {
  type      = string
  sensitive = true
}

variable "aws_output" {
  type = string
}

provider "osc" {
  pat         = var.osc_pat
  environment = var.osc_environment
}

resource "osc_encore_instance" "example" {
  name         = "ggexample"
  profiles_url = "https://raw.githubusercontent.com/Eyevinn/encore-test-profiles/refs/heads/main/profiles.yml"
}

resource "osc_valkey_instance" "example" {
  name = "ggexample"
}

resource "osc_encore_callback_instance" "example" {
  name        = "ggexample"
  redis_url   = format("redis://%s:%s", osc_valkey_instance.example.external_ip, osc_valkey_instance.example.external_port)
  encore_url  = trimsuffix(osc_encore_instance.example.url, "/")
  redis_queue = "transfer"
}

resource "osc_secret" "keyid" {
  service_ids  = ["eyevinn-docker-retransfer"]
  secret_name  = "awsaccesskeyid"
  secret_value = var.aws_keyid
}

resource "osc_secret" "secret" {
  service_ids  = ["eyevinn-docker-retransfer"]
  secret_name  = "awssecretaccesskey"
  secret_value = var.aws_secret
}

resource "osc_encore_transfer_instance" "example" {
  name        = "ggexample"
  redis_url   = osc_encore_callback_instance.example.redis_url
  redis_queue = osc_encore_callback_instance.example.redis_queue
  output      = var.aws_output
  aws_keyid   = osc_secret.keyid.secret_name
  aws_secret  = osc_secret.secret.secret_name
  osc_token   = var.osc_pat
}

output "encore_url" {
  value = trimsuffix(osc_encore_instance.example.url, "/")
}

output "encore_name" {
  value = osc_encore_instance.example.name
}

output "callback_url" {
  value = trimsuffix(osc_encore_callback_instance.example.url, "/")
}

Running the Pipeline

Before running Terraform, ensure your environment variables are set:

export TF_VAR_osc_pat=<PERSONAL_ACCESS_TOKEN>
export TF_VAR_aws_keyid=<AWS_KEYID>
export TF_VAR_aws_secret=<AWS_SECRET>

Once the environment variables are configured, you can run the pipeline with:

terraform init
terraform apply

Follow the prompts to create the infrastructure. Once complete, the instances should be successfully provisioned.

Viewing Outputs

To view the outputs from Terraform, run:

terraform output

Example output:

callback_url = "https://eyevinnlab-ggexample.eyevinn-encore-callback-listener.auto.prod.osaas.io"
encore_url = "https://eyevinnlab-ggexample.encore.prod.osaas.io"
encore_name = "ggexample"

To view a specific output, such as the encore_url:

terraform output encore_url
"https://eyevinnlab-ggexample.encore.prod.osaas.io"

Encode Job

To initiate a transcoding job, you can either use the Swagger UI, as described in the previous post, or run the script provided below. The script accepts the URL of the media you want to encode as an input argument.

encoreJob.sh

#!/bin/bash

# Ensure MEDIA_URL argument is provided
if [ -z "$1" ]; then
  echo "Usage: $0 <MEDIA_URL>"
  exit 1
fi

# Assign the first argument to MEDIA_URL
MEDIA_URL="$1"

# Retrieve values from Terraform output
ENCORE_URL=$(terraform output -raw encore_url)
EXTERNAL_ID=$(terraform output -raw encore_name)
EXTERNAL_BASENAME=$(terraform output -raw encore_name)
CALLBACK_URL=$(terraform output -raw callback_url)
OSC_PAT=$TF_VAR_osc_pat
OSC_ENV=$TF_VAR_osc_env

# Validate required variables
if [ -z "$ENCORE_URL" ]; then
  echo "Error: Terraform output 'encore_url' is missing."
  exit 1
fi

if [ -z "$EXTERNAL_ID" ]; then
  echo "Error: Terraform output 'encore_name' is missing."
  exit 1
fi

if [ -z "$EXTERNAL_BASENAME" ]; then
  echo "Error: Terraform output 'encore_name' is missing."
  exit 1
fi

if [ -z "$CALLBACK_URL" ]; then
  echo "Error: Terraform output 'callback_url' is missing."
  exit 1
fi

if [ -z "$OSC_PAT" ]; then
  echo "Error: Environment variable 'OSC_PAT' (TF_VAR_osc_pat) is not set."
  exit 1
fi

if [ -z "$OSC_ENV" ]; then
  echo "Error: Environment variable 'OSC_ENV' (TF_VAR_osc_env) is not set."
  exit 1
fi

TOKEN_URL="https://token.svc.$OSC_ENV.osaas.io/servicetoken"
ENCORE_TOKEN=$(curl -X 'POST' \
    $TOKEN_URL \
    -H 'Content-Type: application/json' \
    -H "x-pat-jwt: Bearer $OSC_PAT"  \
    -d '{"serviceId": "encore"}' | jq -r '.token')

curl -X 'POST' \
  "$ENCORE_URL/encoreJobs" \
  -H "x-jwt: Bearer $ENCORE_TOKEN" \
  -H 'accept: application/hal+json' \
  -H 'Content-Type: application/json' \
  -d '{
  "externalId": "'"$EXTERNAL_ID"'",
  "profile": "program",
  "outputFolder": "/usercontent/",
  "baseName": "'"$EXTERNAL_BASENAME"'",
  "progressCallbackUri": "'"$CALLBACK_URL/encoreCallback"'",
  "inputs": [
    {
      "uri": "'"$MEDIA_URL"'",
      "seekTo": 0,
      "copyTs": true,
      "type": "AudioVideo"
    }
  ]
}'

Example Usage

To run the script with an example media URL, use the following command:

./encoreJob.sh http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/WeAreGoingOnBullrun.mp4

This script will trigger a transcoding job for the specified media file and send progress updates to the provided callback URL. Make sure that all Terraform outputs are correctly set before running the script.

When a job has been submitted and if you want to see the progress you can go to the Encore Callback Listener service and open the instance logs to check that it is receiving the callbacks.

When the transcoding process is completed it will place a job on the transfer queue that will be picked up by the Encore Transfer service. And when all the transfer jobs are completed you will in this example find a set of files in your output bucket where you have set of different variants with different resolutions and bitrates.

Destroy

When the transcoding is finished and no more jobs are required we can take down the instances running with the simple command:

terraform destroy

Conclusion

You now have a fully-fledged video transcoding pipeline for preparing video files for streaming using SVT Encore, along with supporting services. The entire setup is based on open-source software, and you don’t need to set up your own infrastructure to get started. Additionally, the pipeline leverages Terraform for managing and deploying the infrastructure, making it easy to get up and running. Should you later choose to host everything yourself, you’re free to do so, as all the code and resources demonstrated here are available as open source.

Posted on

Scheduled MariaDB backup using GitHub action and job in Eyevinn OSC

This blog gives an example on how to run regular database backups of your MariaDB database where the result is uploaded to an S3 compatible bucket.

For the task to perform the database backup and upload the result to S3 we will be using an open source script that is made available in Eyevinn Open Source Cloud. To launch this on a regular basis we will use a scheduled GitHub workflow that uses Eyevinn OSC action to create the jobs in OSC.

Step 1: setup secrets

Navigate to the service called “MariaDB backup to S3” and select the tab “Service Secrets”.

Create a secret for the URL to the database you want to backup. In the screenshot above we have a secret for the MariaDB url that this blog is running on. A URL is in the form mariadb://root:[rootpassword]@[host]:[ip]/[database]

Then we need secrets for the credentials to the AWS S3 bucket where we will place the backup. In this example they are called “eyevinnawskeyid” and “eyevinnawssecret”.

Step 2: create a test job

To test that everything is setup correctly we will manually create a test job.

Press Create and verify that a backup is taken and the result ends up on the S3 bucket.

Step 3: create a GitHub workflow file

In this example we will be using a GitHub workflow schedule to create a backup job in Open Source Cloud.

This workflow uses the GitHub action Eyevinn OSC action available on the GitHub action marketplace.

Conclusion

This provided an example for how you can launch jobs for an open source project that is made available as a service in Eyevinn Open Source Cloud as a step in a GitHub workflow.

Proudly powered by WordPress