Building An Automated Video Encoding Pipeline In AWS (Graviton 2, Batch and ECS)

Last year on 11th May, Amazon launched their first 6th generation EC2 general-purpose instance, a.k.a “Graviton 2”, their next-generation Arm-based chip, utilizing 64-bit Arm Neoverse N1 cores. These processors boast up to 40% overall improvement on cost-performance ratio, compared to similar-sized M5 instances.

I’ve been meaning to build a “portable” video encoding pipeline for quite some time and this seemed like a good time and opportunity.

This whole CloudFormation template is available on my Github.

Idea

The idea is very simple and straightforward.

We will spawn docker images that will encode the video and then put the final encodes nicely in a S3 bucket.

This is as simple as it can get. If you wish to understand all the nitty-gritty details, move to the “Architecture” section.

Why Not Just Use AWS Elastic Transcoder Instead?

First of all, this pipeline isn’t a replacement for Elastic Transcoder or any other service. This is just another DIY kind of alternate solution.

Now, few reasons why I think that Elastic Transcoder might not be for everyone and maybe a bit more costly if you’re encoding tons of videos daily. ** How? ** you might ask. Well, because the Elastic Transcoder service would charge you “per minute” of the video you’re trying to encode. Let’s say if you’re trying to encode 1 source video into multiple resolutions. It’ll be like paying for every minute of that video, for every resolution per encode type.

Also, what if you’re just trying to encode smaller videos? Like, a 50-second video. You’ll be paying for “60 seconds (1 minute is the least you pay for)”. So, uh, why pay for those extra 10 seconds when you can literally build your own pipeline and just pay for the underlying structure instead. Keep in mind that the pipeline will be encoding multiple videos and there’s no “pay per minute”.

This setup will be costly if you just want to encode few videos every once in a while. The best choice would be to go for Elastic Transcoder instead in that case.

Another added advantage of having your own video encoding pipeline is that you get to customize it however you want. From the input buckets to the EC2 you use and the docker image you use. And the best part, you could customize the encoding script to use FFmpeg supported arguments to further control how you want to encode your video(s).

Pretty slick, eh?.

Spawned Services

Before starting any of this, just be vary that the default services built in this CloudFormation template will cost you money, so if you want to stay in your “free tier“, change the values accordingly. I’ll link some documentation throughout this article that was helpful to me, might be handy for you too.

Some of these services are chargeable and you’ll have to check the cost of these separately and they’ll vary on multiple factors like region/ how much you’re running these services/ if there are any changes you’ve done to this CF template etc.

Good thing is that you can downgrade some of these properties and this architecture just might fall into “free tier”.

The downside of “free tier” would be that it’ll be slow. M6g.medium vs t2.micro is just a stupid comparison and doesn’t have many debatable points.

Architecture/Working

Cute diagram, but let me try to explain what’s happening when and why. Keep in mind that when I say “folder” or “directory” inside S3, I don’t mean it literally. Because S3 doesn’t have a concept of folders/directories, rather, it follows a “flat structure”. However, to help us visualize, they do show it as “nested” navigation. Now, with that in mind, read ahead.

We have a single “S3” bucket that serves as both, Input and Output. We create an event notification whenever we upload/add any new file in the “input” folder & it should trigger our Lambda function with all the details of that object and S3 bucket. This lambda function will filter out the data we need, like “S3 bucket name” and “object’s key name”. This lambda function then adds a new job in our AWS Batch and sets some environment variables for the same. These environment variables are the ones that our encoding script needs (key name, bucket name). Once this batch job is created, it’ll spin our docker image and that docker will call our encoding script. Once that encode script is done encoding the videos, it’ll move the final encoded files in the “Output” folder in the S3 bucket. If that “Output” folder isn’t there, it’ll be created automatically.

Encoding script’s working is pretty much simple. Once it has all the required info, it’ll download that particular object from S3 and gather some info from that file like its resolution and codec information. Then “FFmpeg” will proceed with the encoding in x265 or x264 encode.

Determining the encoding type (x264/x265) is done by the file name itself. You can add “[x264]” or “[x265]” or if you want both types of encodes, then just add“[x264] [x265]” to the file name. If you don’t add these in your file name, then we’ll take “x264” as the default encoding type.

The same thing goes for the resultant resolution. Just provide “[1920]” or “[1920, 1280, 640]” if you want multiple encode in multiple resolutions. The script will generate these for you. Just remember that we’re passing the “width” of the resulting video. So, 1920 would generally go with a “1920×1080” resolution. Scaling the video is done automatically. If you don’t pass anything, the video will take and use whatever source video’s resolution is.

Example file name: My Video [1920,1280] [x264][x265].mp4

This will create 4 files.

• x264 encode with resolution of “1920×1080”.
• x264 encode with resolution of “1280×720”.
• x265 encode with resolution of “1920×1080”.
• x265 encode with resolution of “1280×720”.

Things To Know Beforehand

• Go through “Prerequisites” section before running this template.
• Once you’re done launching this template, you’ll need to create a “Folder” named “input” (small i).
• Once your input video is processed, it’s deleted from the “input” folder. So, the source video will be gone. This is done to regulate storage prices. If you wish to remove this functionality, edit the python script before building the docker image.

Prerequisites

These steps are MOST IMPORTANT and should be done before running the CloudFormation template, otherwise, you will run into problems.

• Add AWSBatchServiceRole policy by following this guide from AWS.
• Add VPC with a public/private subnet by following this guide from AWS.
• You’ll need to EDIT the “Parameters” section to add the values you will get from following these few previous steps (VPC IDs, Subnets etc.)
• Install DOCKER in your system and follow this guide to build multi-arch. supported docker images. Just follow the tutorial for Creating a multi-arch image builder and once you’re done running this architecture via CF template, follow instructions from this same blog but just this part: Creating multi-arch images for x86 and Arm64 and push them to Amazon ECR repository.

Explaining CloudFormation Template

Since this task has tons of things to set up, this might be a little lengthy CloudFormation template. So, I’ll try to explain few things and add some references where I got few things from.

• Base CloudFormation template from : Justin Plute’s Medium Blog.
• Some more info on setting up an encoding pipeline from AWS Blog.
• Some more ideas from another AWS Blog.

Metadata Section:

This is a general section to take inputs while setting up this whole job. You can provide the “Environment” name, that’ll be prepended in the services spawned via this template. You need to specify the VPC ID and Subnet IDs you got from the Prerequisites section.
The best way is to put those IDs in the template itself so that you won’t have to select the values manually every time you set it up.

S3 Bucket and Lambda Invoke Permission:

Whenever an object is added in our S3 bucket, we raise an event to call Lambda, which creates and adds a Job Request in AWS Batch. This has a “DependsOn” property. This tells S3 to wait for the LambdaInvokePermission to be created. Read more about this problem HERE. Yep, this was another place where I was stuck while building this template.

ComputeEnvironment:

This is where we set our “Compute Environment” in AWS Batch. I’m running the docker images on a c6g.medium based machine. It’s an instance powered by Arm-based AWS Graviton 2 processors. It’s the lowest/entry point configuration. If you want to change your Compute Environment type, that’ll be the place to do so.

If you want to attach a Key-Pair with this instance, you could do so by adding a Ec2KeyPair property like this:

Ec2KeyPair: "EncodingJobKeyPairEC2"

Where EncodingJobKeyPairEC2 is the name of an existing Key-pair in that region in your AWS Account. Make sure you have an existing Key-pair with this exact name in the same location where you’re launching this whole template.

Finally, when you’re done with the changes and initial setup, then create a CloudFormation stack with the video_encoding_pipeline_cf.yaml file in the Github repository.

But Why Use Graviton 2?

Like I mentioned at the very beginning of this article, Graviton 2 processors offer better performances for a much lower cost. Arm-based processors typically use less energy, so we’re good on that front as well. Processes like video encoding, image processing, ML/AI, etc. require much more from the CPU than just the RAM. So, if your CPU could handle much more load, you will definitely see some better performance. You could try replacing the Graviton 2 processor from this CF template and build it with some other ones and you’ll definitely see a little bit of performance difference.

However, don’t think in terms of encoding 1 video. Because, if you’re building an encoding pipeline, then you’re expected to be encoding multiple videos regularly. So, the sooner your dockers are done running and the sooner your EC2 compute scales down, you’ll be charged for that amount of time and the resources.

Final Notes

So, this was pretty much what I could think of while building this template and writing this article. Any suggestions are welcome and feel free to build from this template and use it wherever you’d want.

Even though I linked the Github repository for this pipeline in the very first paragraph, but here you go with the direct link to the CloudFormation template.

P.S: I’ve created 2 POC web apps to demonstrate how else you could use this pipeline to build your own video streaming website (like YouTube) and a simple file encoding application that’ll notify you when your encode(s) are completed.