Back

Automatic Scaling with Chef and Kaltura API

Consider the following Kaltura cluster, built with Chef and Amazon’s EC2: A Chef server, 1 load balancer, 2 fronts nodes, 2 batch nodes, 2 Sphinx nodes, and a single MySQL DB.
Usually, this cluster layout will do well in handling the average load of a medium sized user-generated video site or video app. What if all of sudden there are significantly more videos uploaded, how can you avoid downtime due to the increase in traffic?
This post demonstrates how to automatically scale a Kaltura cluster based on system load monitoring using Opscode Chef and Kaltura API.
To simulate the heavy load, we will use Kaltura’s PHP5 Client Library, to call the bulk upload API adding videos to the transcoding queue, and build a Kaltura watchdog script that will run as a cronjob to alert us when the conversion load hits a certain threshold.
As the watchdog alerts on loaded transcoding queue, a Chef knife command (using its EC2 plugin) will launch additional batch instances to handle the load.

Live Demo

What you need

 

Setting up

To connect the Chef server to the Kaltura cluster and run the Kaltura watchdog script, install the kaltura-base package. To install kaltura base, SSH to the Chef machine and, as super user:

rpm -ihv https://installrepo.kaltura.org/releases/kaltura-release.noarch.rpm
yum install kaltura-base
/opt/kaltura/bin/kaltura-base-config.sh

Note: Doing only this configuration step will not start any unneeded Kaltura daemons or expose the Kaltura web interfaces from the Chef server. The kaltura-base package will only allow our watchdog script to connect the rest of the Kaltura cluster and monitor it via the Kaltura API.
Next, also on the Chef machine, edit: /opt/kaltura/app/tests/monitoring/config.ini

[monitor-partner]
id = -4
widgetId = "_-4"
secret = $PARNER_SECRET_HERE
adminSecret = $PARNER_ADMIN_SECRET_HERE
[batch-partner]
id = -1
adminSecret = $PARNER_ADMIN_SECRET_HERE

To retrieve the account API secret keys for partner id -4, run:

. /etc/kaltura.d/system.ini
mysql -h $DB1_HOST -u $DB1_USER -p$DB1_PASS $DB1_NAME
mysql> select secret,admin_secret from kaltura.partner where id=-4\G
Sample output:
 secret: 68b329da9893e34099c7d8ad5cb9c940
 admin_secret: 68b329da9893e34099c7d8ad5cb9c940

To obtain the account API admin secret key for partner id -1:

mysql> select secret,admin_secret from kaltura.partner where id=-1\G

Sample output:
 admin_secret: 03c1db4a2c091c8bd32e375b614f7070

To test the watchdog, run:

php /opt/kaltura/app/plugins/monitor/nagios/exec.php --warning-threshold 10 --error-threshold 11 --script "/opt/kaltura/app/tests/monitoring/api_v3/getBatchQueueSize.php --service-url $SERVICE_URL --job-type CONVERT"
Sample output:
 Scheduler Queue for CONVERT is: 5

 

The watchdog script

See the watchdog code on GitHub. ( Feel free to fork and submit pull requests! )

#!/bin/sh
SYSTEM_INI=/etc/kaltura.d/system.ini
if [ -r "$SYSTEM_INI" ];then
  . /etc/kaltura.d/system.ini
else
  echo "Could not source $SYSTEM_INI:("
  exit 1
fi
if [ $# -lt 3 ];then
  echo "Usage: $0 <AMI id> <warning threshold> <critical threshold>"
  exit 2
fi
AMI_IMG=$1
LOWER_THRESHOLD=$2
UPPER_THRESHOLD=$3
OUT=`php /opt/kaltura/app/plugins/monitor/nagios/exec.php --warning-threshold $LOWER_THRESHOLD --error-threshold $UPPER_THRESHOLD --script "/opt/kaltura/app/tests/monitoring/api_v3/getBatchQueueSize.php --service-url $KALTURA_VIRTUAL_HOST_NAME --job-type CONVERT"`
RC=$?
if [ $RC -eq 1 ];then
  echo "Reached WARNING threshold"
  # here you can do some actions, for example:
  # send yourself an email
  # launch instances
fi
# we are at CRITICAL, lets launch
if [ $RC -eq 2 ];then
  echo "$OUT
Lunching another batch instance ..
"
  knife ec2 server create --availability-zone us-east-1d --flavor m3.medium --image $AMI_IMG --identity-file ~/csi.pem --run-list "recipe[nfs],recipe[kaltura::batch]" --ssh-user ec2-user 2>&1 | tee /tmp/log
fi

Save the code to /usr/local/bin/kaltura_laod_watchdog.sh and give it the executable permission:

chmod +x /usr/local/bin/kaltura_laod_watchdog.sh

Test the watchdog using bulk upload. From Chef server, run the following:

cd /opt/kaltura/web/content/docs/
unzip kaltura_batch_upload_falcon.zip
php /opt/kaltura/bin/upload_bulk.php $SERVICE_URL $PARTNER_ID $USER_SECRET STRESSER /opt/kaltura/web/content/docs/kaltura_batch_upload_falcon.csv bulkUploadCsv.CSV

Run the upload_bulk script a few times to get a conversion queue going.
Normally, you will run the watchdog in crontab, at about 5 min interval. To see it in action, lets run it manually:

/usr/local/bin/kaltura_laod_watchdog.sh <AMI id> <warning threshold> <critical threshold>

Let’s pass very small thresholds to the watchdog to see it working. Pass 1 for warning and 10 for critical. (Naturally, in Production, numbers will be higher.) From command line, run the following command:

while [ true ];do  /usr/local/bin/kaltura_laod_watchdog.sh YOUR_AMI_IMG 1 10; sleep 20;done

This will run the watchdog in an endless loop in the shell we’re at so we can see its output:

Instance ID: i-da9f3a8a
Flavor: m3.medium
Image: ami-0f9e9066
Region: us-east-1
Availability Zone: us-east-1d
Security Groups: default
Tags: Name: i-da9f3a8a
SSH Key: jess
Waiting for instance......
Public DNS Name: ec2-54-197-120-85.compute-1.amazonaws.com
Public IP Address: 54.197.120.85
Private DNS Name: ip-10-169-36-57.ec2.internal
Private IP Address: 10.169.36.57
Waiting for sshd.....................................................done
Connecting to ec2-54-197-120-85.compute-1.amazonaws.com
ec2-54-197-120-85.compute-1.amazonaws.com Installing Chef Client...
ec2-54-197-120-85.compute-1.amazonaws.com --2014-05-05 07:56:29-- https://www.opscode.com/chef/install.sh
ec2-54-197-120-85.compute-1.amazonaws.com Resolving www.opscode.com... 184.106.28.91
ec2-54-197-120-85.compute-1.amazonaws.com Connecting to www.opscode.com|184.106.28.91|:443... connected.
ec2-54-197-120-85.compute-1.amazonaws.com HTTP request sent, awaiting response... 200 OK
ec2-54-197-120-85.compute-1.amazonaws.com Length: 15934 (16K) [application/x-sh]
ec2-54-197-120-85.compute-1.amazonaws.com Saving to: “STDOUT”
ec2-54-197-120-85.compute-1.amazonaws.com
100%[======================================>] 15,934 --.-K/s in 0s
ec2-54-197-120-85.compute-1.amazonaws.com
ec2-54-197-120-85.compute-1.amazonaws.com 2014-05-05 07:56:29 (604 MB/s) - written to stdout [15934/15934]
ec2-54-197-120-85.compute-1.amazonaws.com
ec2-54-197-120-85.compute-1.amazonaws.com Downloading Chef 11.12.2 for el...
ec2-54-197-120-85.compute-1.amazonaws.com downloading https://www.opscode.com/chef/metadata?v=11.12.2&prerelease=false&nightlies=false&p=el&pv=6&m=x86_64
ec2-54-197-120-85.compute-1.amazonaws.com to file /tmp/install.sh.1364/metadata.txt
ec2-54-197-120-85.compute-1.amazonaws.com trying wget...
ec2-54-197-120-85.compute-1.amazonaws.com url https://opscode-omnibus-packages.s3.amazonaws.com/el/6/x86_64/chef-11.12.2-1.el6.x86_64.rpm
ec2-54-197-120-85.compute-1.amazonaws.com md5 b8ef6b908b42f1cf97ac7864a8587d9d
ec2-54-197-120-85.compute-1.amazonaws.com sha256 cc6bafac692a9b6db791310f46917a0c5857bd5b5e69c65daabf0beac3595cfc
ec2-54-197-120-85.compute-1.amazonaws.com downloaded metadata file looks valid...
ec2-54-197-120-85.compute-1.amazonaws.com downloading https://opscode-omnibus-packages.s3.amazonaws.com/el/6/x86_64/chef-11.12.2-1.el6.x86_64.rpm
ec2-54-197-120-85.compute-1.amazonaws.com to file /tmp/install.sh.1364/chef-11.12.2-1.el6.x86_64.rpm
ec2-54-197-120-85.compute-1.amazonaws.com trying wget...
ec2-54-197-120-85.compute-1.amazonaws.com Comparing checksum with sha256sum...
ec2-54-197-120-85.compute-1.amazonaws.com Installing Chef 11.12.2
ec2-54-197-120-85.compute-1.amazonaws.com installing with rpm...
ec2-54-197-120-85.compute-1.amazonaws.com warning: /tmp/install.sh.1364/chef-11.12.2-1.el6.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 83ef826a: NOKEY
ec2-54-197-120-85.compute-1.amazonaws.com Preparing... ########################################### [100%]
ec2-54-197-120-85.compute-1.amazonaws.com 1:chef ########################################### [100%]
ec2-54-197-120-85.compute-1.amazonaws.com Thank you for installing Chef!
ec2-54-197-120-85.compute-1.amazonaws.com Starting first Chef Client run...
ec2-54-197-120-85.compute-1.amazonaws.com [2014-05-05T07:56:49-04:00] WARN:
ec2-54-197-120-85.compute-1.amazonaws.com * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
ec2-54-197-120-85.compute-1.amazonaws.com SSL validation of HTTPS requests is disabled. HTTPS connections are still
ec2-54-197-120-85.compute-1.amazonaws.com encrypted, but chef is not able to detect forged replies or man in the middle
ec2-54-197-120-85.compute-1.amazonaws.com attacks.
ec2-54-197-120-85.compute-1.amazonaws.com
ec2-54-197-120-85.compute-1.amazonaws.com To fix this issue add an entry like this to your configuration file:
ec2-54-197-120-85.compute-1.amazonaws.com
ec2-54-197-120-85.compute-1.amazonaws.com ```
ec2-54-197-120-85.compute-1.amazonaws.com # Verify all HTTPS connections (recommended)
ec2-54-197-120-85.compute-1.amazonaws.com ssl_verify_mode :verify_peer
ec2-54-197-120-85.compute-1.amazonaws.com
ec2-54-197-120-85.compute-1.amazonaws.com # OR, Verify only connections to chef-server
ec2-54-197-120-85.compute-1.amazonaws.com verify_api_cert true
ec2-54-197-120-85.compute-1.amazonaws.com ```
ec2-54-197-120-85.compute-1.amazonaws.com
ec2-54-197-120-85.compute-1.amazonaws.com To check your SSL configuration, or troubleshoot errors, you can use the
ec2-54-197-120-85.compute-1.amazonaws.com `knife ssl check` command like so:
ec2-54-197-120-85.compute-1.amazonaws.com
ec2-54-197-120-85.compute-1.amazonaws.com ```
ec2-54-197-120-85.compute-1.amazonaws.com knife ssl check -c /etc/chef/client.rb
ec2-54-197-120-85.compute-1.amazonaws.com ```
ec2-54-197-120-85.compute-1.amazonaws.com
ec2-54-197-120-85.compute-1.amazonaws.com * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
ec2-54-197-120-85.compute-1.amazonaws.com
ec2-54-197-120-85.compute-1.amazonaws.com Starting Chef Client, version 11.12.2
ec2-54-197-120-85.compute-1.amazonaws.com Creating a new client identity for i-da9f3a8a using the validator key.
ec2-54-197-120-85.compute-1.amazonaws.com resolving cookbooks for run list: ["nfs", "kaltura::batch"]
ec2-54-197-120-85.compute-1.amazonaws.com Synchronizing Cookbooks:
ec2-54-197-120-85.compute-1.amazonaws.com - kaltura
ec2-54-197-120-85.compute-1.amazonaws.com - line
ec2-54-197-120-85.compute-1.amazonaws.com - nfs
ec2-54-197-120-85.compute-1.amazonaws.com Compiling Cookbooks...
ec2-54-197-120-85.compute-1.amazonaws.com Converging 12 resources
ec2-54-197-120-85.compute-1.amazonaws.com Recipe: nfs::default
ec2-54-197-120-85.compute-1.amazonaws.com * package[nfs-utils] action install (up to date)
ec2-54-197-120-85.compute-1.amazonaws.com * package[rpcbind] action install (up to date)
ec2-54-197-120-85.compute-1.amazonaws.com * template[/etc/sysconfig/nfs] action create
ec2-54-197-120-85.compute-1.amazonaws.com - update content in file /etc/sysconfig/nfs from 9264ee to 08cfdf
ec2-54-197-120-85.compute-1.amazonaws.com --- /etc/sysconfig/nfs 2013-01-08 11:08:57.000000000 -0500
ec2-54-197-120-85.compute-1.amazonaws.com +++ /tmp/chef-rendered-template20140505-1433-wnzzn1 2014-05-05 07:57:17.461402979 -0400
ec2-54-197-120-85.compute-1.amazonaws.com @@ -1,70 +1,9 @@
ec2-54-197-120-85.compute-1.amazonaws.com -#
ec2-54-197-120-85.compute-1.amazonaws.com -# Define which protocol versions mountd
ec2-54-197-120-85.compute-1.amazonaws.com -# will advertise. The values are "no" or "yes"
ec2-54-197-120-85.compute-1.amazonaws.com -# with yes being the default
ec2-54-197-120-85.compute-1.amazonaws.com -#MOUNTD_NFS_V2="no"
ec2-54-197-120-85.compute-1.amazonaws.com -#MOUNTD_NFS_V3="no"
ec2-54-197-120-85.compute-1.amazonaws.com -#
ec2-54-197-120-85.compute-1.amazonaws.com -#
ec2-54-197-120-85.compute-1.amazonaws.com -# Path to remote quota server. See rquotad(8)
ec2-54-197-120-85.compute-1.amazonaws.com -#RQUOTAD="/usr/sbin/rpc.rquotad"
ec2-54-197-120-85.compute-1.amazonaws.com -# Port rquotad should listen on.
ec2-54-197-120-85.compute-1.amazonaws.com -#RQUOTAD_PORT=875
ec2-54-197-120-85.compute-1.amazonaws.com -# Optinal options passed to rquotad
ec2-54-197-120-85.compute-1.amazonaws.com -#RPCRQUOTADOPTS=""
ec2-54-197-120-85.compute-1.amazonaws.com -#
ec2-54-197-120-85.compute-1.amazonaws.com -#
ec2-54-197-120-85.compute-1.amazonaws.com -# Optional arguments passed to in-kernel lockd
ec2-54-197-120-85.compute-1.amazonaws.com -#LOCKDARG=
ec2-54-197-120-85.compute-1.amazonaws.com -# TCP port rpc.lockd should listen on.
ec2-54-197-120-85.compute-1.amazonaws.com -#LOCKD_TCPPORT=32803
ec2-54-197-120-85.compute-1.amazonaws.com -# UDP port rpc.lockd should listen on.
ec2-54-197-120-85.compute-1.amazonaws.com -#LOCKD_UDPPORT=32769
ec2-54-197-120-85.compute-1.amazonaws.com -#
ec2-54-197-120-85.compute-1.amazonaws.com -#
ec2-54-197-120-85.compute-1.amazonaws.com -# Optional arguments passed to rpc.nfsd. See rpc.nfsd(8)
ec2-54-197-120-85.compute-1.amazonaws.com -# Turn off v2 and v3 protocol support
ec2-54-197-120-85.compute-1.amazonaws.com -#RPCNFSDARGS="-N 2 -N 3"
ec2-54-197-120-85.compute-1.amazonaws.com -# Turn off v4 protocol support
ec2-54-197-120-85.compute-1.amazonaws.com -#RPCNFSDARGS="-N 4"
ec2-54-197-120-85.compute-1.amazonaws.com -# Number of nfs server processes to be started.
ec2-54-197-120-85.compute-1.amazonaws.com -# The default is 8.
ec2-54-197-120-85.compute-1.amazonaws.com -#RPCNFSDCOUNT=8
ec2-54-197-120-85.compute-1.amazonaws.com -# Stop the nfsd module from being pre-loaded
ec2-54-197-120-85.compute-1.amazonaws.com -#NFSD_MODULE="noload"
ec2-54-197-120-85.compute-1.amazonaws.com -# Set V4 grace period in seconds
ec2-54-197-120-85.compute-1.amazonaws.com -#NFSD_V4_GRACE=90
ec2-54-197-120-85.compute-1.amazonaws.com -#
ec2-54-197-120-85.compute-1.amazonaws.com -#
ec2-54-197-120-85.compute-1.amazonaws.com -#
ec2-54-197-120-85.compute-1.amazonaws.com -# Optional arguments passed to rpc.mountd. See rpc.mountd(8)
ec2-54-197-120-85.compute-1.amazonaws.com -#RPCMOUNTDOPTS=""
ec2-54-197-120-85.compute-1.amazonaws.com -# Port rpc.mountd should listen on.
ec2-54-197-120-85.compute-1.amazonaws.com -#MOUNTD_PORT=892
ec2-54-197-120-85.compute-1.amazonaws.com -#
ec2-54-197-120-85.compute-1.amazonaws.com -#
ec2-54-197-120-85.compute-1.amazonaws.com -# Optional arguments passed to rpc.statd. See rpc.statd(8)
ec2-54-197-120-85.compute-1.amazonaws.com -#STATDARG=""
ec2-54-197-120-85.compute-1.amazonaws.com -# Port rpc.statd should listen on.
ec2-54-197-120-85.compute-1.amazonaws.com -#STATD_PORT=662
ec2-54-197-120-85.compute-1.amazonaws.com -# Outgoing port statd should used. The default is port
ec2-54-197-120-85.compute-1.amazonaws.com -# is random
ec2-54-197-120-85.compute-1.amazonaws.com -#STATD_OUTGOING_PORT=2020
ec2-54-197-120-85.compute-1.amazonaws.com -# Specify callout program
ec2-54-197-120-85.compute-1.amazonaws.com -#STATD_HA_CALLOUT="/usr/local/bin/foo"
ec2-54-197-120-85.compute-1.amazonaws.com -#
ec2-54-197-120-85.compute-1.amazonaws.com -#
ec2-54-197-120-85.compute-1.amazonaws.com -# Optional arguments passed to rpc.idmapd. See rpc.idmapd(8)
ec2-54-197-120-85.compute-1.amazonaws.com -#RPCIDMAPDARGS=""
ec2-54-197-120-85.compute-1.amazonaws.com -#
ec2-54-197-120-85.compute-1.amazonaws.com -# Set to turn on Secure NFS mounts.
ec2-54-197-120-85.compute-1.amazonaws.com -#SECURE_NFS="yes"
ec2-54-197-120-85.compute-1.amazonaws.com -# Optional arguments passed to rpc.gssd. See rpc.gssd(8)
ec2-54-197-120-85.compute-1.amazonaws.com -#RPCGSSDARGS=""
ec2-54-197-120-85.compute-1.amazonaws.com -# Optional arguments passed to rpc.svcgssd. See rpc.svcgssd(8)
ec2-54-197-120-85.compute-1.amazonaws.com -#RPCSVCGSSDARGS=""
ec2-54-197-120-85.compute-1.amazonaws.com -#
ec2-54-197-120-85.compute-1.amazonaws.com -# To enable RDMA support on the server by setting this to
ec2-54-197-120-85.compute-1.amazonaws.com -# the port the server should listen on
ec2-54-197-120-85.compute-1.amazonaws.com -#RDMA_PORT=20049
ec2-54-197-120-85.compute-1.amazonaws.com +# Generated by Chef for ip-10-169-36-57.ec2.internal
ec2-54-197-120-85.compute-1.amazonaws.com +# Local modifications will be overwritten.
ec2-54-197-120-85.compute-1.amazonaws.com +STATD_PORT=32765
ec2-54-197-120-85.compute-1.amazonaws.com +STATD_OUTGOING_PORT=32766
ec2-54-197-120-85.compute-1.amazonaws.com +MOUNTD_PORT=32767
ec2-54-197-120-85.compute-1.amazonaws.com +LOCKD_UDPPORT=32768
ec2-54-197-120-85.compute-1.amazonaws.com +LOCKD_TCPPORT=32768
ec2-54-197-120-85.compute-1.amazonaws.com +RQUOTAD="no"
ec2-54-197-120-85.compute-1.amazonaws.com - restore selinux security context
ec2-54-197-120-85.compute-1.amazonaws.com
ec2-54-197-120-85.compute-1.amazonaws.com * service[portmap] action start (up to date)
ec2-54-197-120-85.compute-1.amazonaws.com * service[portmap] action enable (up to date)
ec2-54-197-120-85.compute-1.amazonaws.com * service[nfslock] action start (up to date)
ec2-54-197-120-85.compute-1.amazonaws.com * service[nfslock] action enable (up to date)
ec2-54-197-120-85.compute-1.amazonaws.com * bash[mkdir -p /opt/kaltura/web] action run
ec2-54-197-120-85.compute-1.amazonaws.com - execute "bash" "/tmp/chef-script20140505-1433-1vfhyw0"
ec2-54-197-120-85.compute-1.amazonaws.com
ec2-54-197-120-85.compute-1.amazonaws.com * mount[/opt/kaltura/web] action mount
ec2-54-197-120-85.compute-1.amazonaws.com - mount ip-10-11-147-101:/opt/kaltura/web to /opt/kaltura/web
ec2-54-197-120-85.compute-1.amazonaws.com
ec2-54-197-120-85.compute-1.amazonaws.com Recipe: kaltura::batch
ec2-54-197-120-85.compute-1.amazonaws.com * log[Installing Kaltura batch] action write
ec2-54-197-120-85.compute-1.amazonaws.com
ec2-54-197-120-85.compute-1.amazonaws.com * template[/etc/yum.repos.d/kaltura.repo] action create
ec2-54-197-120-85.compute-1.amazonaws.com - create new file /etc/yum.repos.d/kaltura.repo
ec2-54-197-120-85.compute-1.amazonaws.com - update content in file /etc/yum.repos.d/kaltura.repo from none to 31731e
ec2-54-197-120-85.compute-1.amazonaws.com --- /etc/yum.repos.d/kaltura.repo 2014-05-05 07:57:19.465402978 -0400
ec2-54-197-120-85.compute-1.amazonaws.com +++ /tmp/chef-rendered-template20140505-1433-mu18ls 2014-05-05 07:57:19.466402979 -0400
ec2-54-197-120-85.compute-1.amazonaws.com @@ -1 +1,29 @@
ec2-54-197-120-85.compute-1.amazonaws.com +[Kaltura]
ec2-54-197-120-85.compute-1.amazonaws.com +name = Kaltura Server
ec2-54-197-120-85.compute-1.amazonaws.com +baseurl = https://54.211.235.142/releases/stable/RPMS/$basearch/
ec2-54-197-120-85.compute-1.amazonaws.com +gpgkey = https://54.211.235.142/releases/RPM-GPG-KEY-kaltura
ec2-54-197-120-85.compute-1.amazonaws.com +gpgcheck = 1
ec2-54-197-120-85.compute-1.amazonaws.com +enabled = 1
ec2-54-197-120-85.compute-1.amazonaws.com +
ec2-54-197-120-85.compute-1.amazonaws.com +[Kaltura-noarch]
ec2-54-197-120-85.compute-1.amazonaws.com +name = Kaltura Server arch independent
ec2-54-197-120-85.compute-1.amazonaws.com +baseurl = https://54.211.235.142/releases/stable/RPMS/noarch
ec2-54-197-120-85.compute-1.amazonaws.com +gpgkey = https://54.211.235.142/releases/RPM-GPG-KEY-kaltura
ec2-54-197-120-85.compute-1.amazonaws.com +gpgcheck = 1
ec2-54-197-120-85.compute-1.amazonaws.com +enabled = 1
ec2-54-197-120-85.compute-1.amazonaws.com +
ec2-54-197-120-85.compute-1.amazonaws.com +[Kaltura-testing]
ec2-54-197-120-85.compute-1.amazonaws.com +name = Kaltura Server arch independent
ec2-54-197-120-85.compute-1.amazonaws.com +baseurl = https://54.211.235.142/releases/nightly/RPMS/$basearch/
ec2-54-197-120-85.compute-1.amazonaws.com +gpgkey = https://54.211.235.142/releases/RPM-GPG-KEY-kaltura
ec2-54-197-120-85.compute-1.amazonaws.com +gpgcheck = 1
ec2-54-197-120-85.compute-1.amazonaws.com +enabled = 1
ec2-54-197-120-85.compute-1.amazonaws.com +
ec2-54-197-120-85.compute-1.amazonaws.com +[Kaltura-testing-noarch]
ec2-54-197-120-85.compute-1.amazonaws.com +name = Kaltura Server arch independent
ec2-54-197-120-85.compute-1.amazonaws.com +baseurl = https://54.211.235.142/releases/nightly/RPMS/noarch
ec2-54-197-120-85.compute-1.amazonaws.com +gpgkey = https://54.211.235.142/releases/RPM-GPG-KEY-kaltura
ec2-54-197-120-85.compute-1.amazonaws.com +gpgcheck = 1
ec2-54-197-120-85.compute-1.amazonaws.com +enabled = 1
ec2-54-197-120-85.compute-1.amazonaws.com +
ec2-54-197-120-85.compute-1.amazonaws.com - change mode from '' to '0600'
ec2-54-197-120-85.compute-1.amazonaws.com - change owner from '' to 'root'
ec2-54-197-120-85.compute-1.amazonaws.com - change group from '' to 'root'
ec2-54-197-120-85.compute-1.amazonaws.com - restore selinux security context
ec2-54-197-120-85.compute-1.amazonaws.com
ec2-54-197-120-85.compute-1.amazonaws.com * package[kaltura-batch] action install[2014-05-05T07:58:31-04:00] WARN: package[kaltura-batch] matched multiple Provides for kaltura-batch but we can only use the first match: kaltura-batch. Please use a more specific version.
ec2-54-197-120-85.compute-1.amazonaws.com
ec2-54-197-120-85.compute-1.amazonaws.com - install version 9.15.0-2 of package kaltura-batch
ec2-54-197-120-85.compute-1.amazonaws.com
ec2-54-197-120-85.compute-1.amazonaws.com * template[/root/kaltura.ans] action create
ec2-54-197-120-85.compute-1.amazonaws.com - create new file /root/kaltura.ans
ec2-54-197-120-85.compute-1.amazonaws.com - update content in file /root/kaltura.ans from none to 48b242
ec2-54-197-120-85.compute-1.amazonaws.com --- /root/kaltura.ans 2014-05-05 08:03:15.880402979 -0400
ec2-54-197-120-85.compute-1.amazonaws.com +++ /tmp/chef-rendered-template20140505-1433-cc5yz7 2014-05-05 08:03:15.881402979 -0400
ec2-54-197-120-85.compute-1.amazonaws.com @@ -1 +1,32 @@
ec2-54-197-120-85.compute-1.amazonaws.com +TIME_ZONE="America/New_York"
ec2-54-197-120-85.compute-1.amazonaws.com +KALTURA_FULL_VIRTUAL_HOST_NAME="ip-10-169-36-57.ec2.internal:80"
ec2-54-197-120-85.compute-1.amazonaws.com +KALTURA_VIRTUAL_HOST_NAME="ip-10-169-36-57.ec2.internal"
ec2-54-197-120-85.compute-1.amazonaws.com +DB1_HOST="ip-10-152-141-132.ec2.internal"
ec2-54-197-120-85.compute-1.amazonaws.com +DB1_PORT="3306"
ec2-54-197-120-85.compute-1.amazonaws.com +DB1_PASS="somepasswd1"
ec2-54-197-120-85.compute-1.amazonaws.com +DB1_NAME="kaltura"
ec2-54-197-120-85.compute-1.amazonaws.com +DB1_USER="3306"
ec2-54-197-120-85.compute-1.amazonaws.com +SERVICE_URL="ip-10-167-11-119.ec2.internal"
ec2-54-197-120-85.compute-1.amazonaws.com +SPHINX_SERVER1="ip-10-169-36-57.ec2.internal"
ec2-54-197-120-85.compute-1.amazonaws.com +SPHINX_SERVER2=" "
ec2-54-197-120-85.compute-1.amazonaws.com +DWH_HOST="ip-10-152-141-132.ec2.internal"
ec2-54-197-120-85.compute-1.amazonaws.com +DWH_PORT="3306"
ec2-54-197-120-85.compute-1.amazonaws.com +DWH_PASS="somepasswd1"
ec2-54-197-120-85.compute-1.amazonaws.com +SPHINX_DB_HOST="ip-10-152-141-132.ec2.internal"
ec2-54-197-120-85.compute-1.amazonaws.com +SPHINX_DB_PORT="3306"
ec2-54-197-120-85.compute-1.amazonaws.com +ADMIN_CONSOLE_ADMIN_MAIL="jess.portnoy@kaltura.com"
ec2-54-197-120-85.compute-1.amazonaws.com +ADMIN_CONSOLE_PASSWORD="somepasswd2"
ec2-54-197-120-85.compute-1.amazonaws.com +CDN_HOST="ip-10-167-11-119.ec2.internal"
ec2-54-197-120-85.compute-1.amazonaws.com +KALTURA_VIRTUAL_HOST_PORT="80"
ec2-54-197-120-85.compute-1.amazonaws.com +SUPER_USER="root"
ec2-54-197-120-85.compute-1.amazonaws.com +SUPER_USER_PASSWD="p6fsicBFbzqCUKsCJtdM"
ec2-54-197-120-85.compute-1.amazonaws.com +ENVIRONMENT_NAME="Kaltura Video Platform"
ec2-54-197-120-85.compute-1.amazonaws.com +PROTOCOL="http"
ec2-54-197-120-85.compute-1.amazonaws.com +CONFIG_CHOICE="0"
ec2-54-197-120-85.compute-1.amazonaws.com +IS_SSL="n"
ec2-54-197-120-85.compute-1.amazonaws.com +RED5_HOST="ip-10-169-36-57.ec2.internal"
ec2-54-197-120-85.compute-1.amazonaws.com +USER_CONSENT="0"
ec2-54-197-120-85.compute-1.amazonaws.com +CRT_FILE="/etc/ssl/certs/localhost.crt"
ec2-54-197-120-85.compute-1.amazonaws.com +KEY_FILE="/etc/pki/tls/private/localhost.key"
ec2-54-197-120-85.compute-1.amazonaws.com +CHAIN_FILE="NOCHAIN"
ec2-54-197-120-85.compute-1.amazonaws.com - change mode from '' to '0600'
ec2-54-197-120-85.compute-1.amazonaws.com - change owner from '' to 'root'
ec2-54-197-120-85.compute-1.amazonaws.com - change group from '' to 'root'
ec2-54-197-120-85.compute-1.amazonaws.com - restore selinux security context
ec2-54-197-120-85.compute-1.amazonaws.com
ec2-54-197-120-85.compute-1.amazonaws.com * bash[setup batchMgr daemon] action run
ec2-54-197-120-85.compute-1.amazonaws.com - execute "bash" "/tmp/chef-script20140505-1433-1i3fdgy"
ec2-54-197-120-85.compute-1.amazonaws.com
ec2-54-197-120-85.compute-1.amazonaws.com Recipe: nfs::default
ec2-54-197-120-85.compute-1.amazonaws.com * service[portmap] action restart
ec2-54-197-120-85.compute-1.amazonaws.com - restart service service[portmap]
ec2-54-197-120-85.compute-1.amazonaws.com
ec2-54-197-120-85.compute-1.amazonaws.com * service[nfslock] action restart
ec2-54-197-120-85.compute-1.amazonaws.com - restart service service[nfslock]
ec2-54-197-120-85.compute-1.amazonaws.com
ec2-54-197-120-85.compute-1.amazonaws.com
ec2-54-197-120-85.compute-1.amazonaws.com Running handlers:
ec2-54-197-120-85.compute-1.amazonaws.com Running handlers complete
ec2-54-197-120-85.compute-1.amazonaws.com
ec2-54-197-120-85.compute-1.amazonaws.com Chef Client finished, 10/16 resources updated in 627.954966882 seconds
Instance ID: i-da9f3a8a
Flavor: m3.medium
Image: ami-0f9e9066
Region: us-east-1
Availability Zone: us-east-1d
Security Groups: default
Security Group Ids: default
Tags: Name: i-da9f3a8a
SSH Key: jess
Root Device Type: ebs
Root Volume ID: vol-53d02d1a
Root Device Name: /dev/sda1
Root Device Delete on Terminate: true
Public DNS Name: ec2-54-197-120-85.compute-1.amazonaws.com
Public IP Address: 54.197.120.85
Private DNS Name: ip-10-169-36-57.ec2.internal
Private IP Address: 10.169.36.57
Environment: _default
Run List: recipe[nfs], recipe[kaltura::batch]

As you can see, we successfully launched a new EC2 instance, and applied the nfs and kaltura::batch Chef recipes using chef-client.

What’s next?

To extend this functionality into production mode, run a manager that will:

  • Keep monitoring the transcoding queue using the watchdog
  • Keep a list of new batch servers launched when the load gets high
  • When the load calms down, stops the batch daemon on the new transcoding node, waits 20 minutes to makes sure the load remains low, and terminate the instance

Note: that the same practice can be applied to other cloud infrastructures or VM clusters (such as VMWare) using their respective APIs.
If you build on it, please submit a pull request on the GitHub project.

Let's Get Going