This post is the result of a sort of challenge. The objective is to create an architecture to generate, manage and deliver HTTP streaming videos using free and/or open-source tools and/or applications. It need to answer to many questions : tools must be open source or free; reliability of the platform and the ability to scale up quickly. The architecture can be split in two parts : the content preparation and the delivery. I will expose an overview for each part of the architecture. Then I will list differents tools you can use. To finish, I will give you in details the architecture I will choose to deploy. But to realize this, few or some developments can be required, the language you will use, will be your choice. The operating system will be on Linux.

Content preparation

Here is the part to prepare the content. It will be separate in 3 differents items, I call them services :

• Manager this service will orchestrate all the tasks. In this part, I choose the self-development but another option is possible (see below).
• Bus the service bus will deliver all messages between others services
• Worker each worker will be in charge of one specific task : encode, upload, download, package.

All of these is exposed in this figure :

The generated assets are transfered in the delivery zone storage.

Service : Manager

Here is the intelligence of the system. You have two choice :

• create your own system. In this case, the manager need some developments, you can choose the language you want : Python, PHP, NodeJS, Java, Perl, etc…
• using a system already developed. I think about Kaltura platform for example. This option need some integration and learning time if you want to use it.

I recommend to associate your code to a database. The database will save the metadata of your video assets and some configurations datas.

• relational database : MySQL, PostgreSQL, SQLLite, …
• non-relational database : Redis, …

Service : Bus

The service bus will use the concept of Message queue. That is very interesting because it helps to manage more easily the communication between each application : you send the message and the library or the system make the rest. There are many tools in the open source world :

• ZeroMQ,
• Gearman,
• RabbitMQ

For each message queue solutions, you will find connectors for separates languages (C, C++, PHP, Python, Perl, …). Depends on the message queue solution you choose, you need to make more or less developments.

Service : Worker

Workers services are here to make specific tasks :

• encode : this service is based on the most famous open source tools in the encoding’s world : ffmpeg and its differents libraries (libfaac, libx264, …). But you can use mencoder too.
• package : the objective is not to encode the video asset, but to change the format. Some tools are available :
  • MP4Box : to manage MP4 files or MPEG-DASH
  • f4fpackager : to generate HDS, you will need to deploy the HDS module for Apache in the delivery platform
  • mediasegmenter : to generate HLS
  • FLV2DTSC : to generate DTSC if you want to use MistServer (a solution I told in previous articles).
• transferts : at the end of a process, you will need to transfert the asset to a destination. Transfert workers is able to use some protocols : FTP, SCP, HTTP, etc… So you can push the content to the delivery storage.

All of this services can be in the same physical server. If you have more assets to manage and generate, it is possible very easily to scale up. Workers can be deploy on dedicated server and the bus service is here to manage the network. You can add another service manager to add redundance. Thoses updates will have a impact on the reliability of the platform.

Here is one solution which include the most part of those precepts : Transcodem. You can fork this project on GitHub too.

Delivery

Ok, now your assets are ready to deliver and are in the delivery storage. In our architecture, we want to stream content based on HTTP protocol (HDS, HLS, Smooth Streaming). You can start with one physical server, but all tools and the architecture I expose can scale up easily to many physicals servers. We have many tools to help us :

• HTTP webservers : Apache, NGinx, Lighttpd and differents plugins (module origin from Adobe to stream HDS for example)
• Media servers : Red5, MistServer

And if your platform will support a high load, you can use proxy cache solution like Varnish. I gave in a previous post a solution to deliver content with Varnish and MistServer.

To finish, you can split the two part or merge them in one infrastructure. I am opened to talk about this architecture and which tools we can use to deploy it.