Automated HTML5 video encoding revisited

A little while ago I approached the subject of scripting automated encoding for HTML5 video formats. I started using the process regularly as I built the Blogsmith Bundle video site. Pretty soon I got some help from friend and TUAW editor Mike Rose, and working together we refined the script and improved the speed tremendously. It was built for use on Mac OS X, but with just a few lines modified, it should work on any *NIX system.

What it does

The script is specifically designed to take an MPEG-4, H.264 file, move it to a new folder based on the filename, create WEBM and OGV versions of it in the new folder, and upload the whole folder to a server using rsync. SSH information and target directories are specified in the config at the top of the script. At the end, it takes what it knows about the files and filenames and creates a TextMate Markdown blog template, inserts the video shortcode for the VideoJS WordPress plugin (which is common syntax for other plugins as well) and opens it for editing in TextMate. The rsync command really needs keyless SSH login for this to be considered fully-automated.

If you run this script with Hazel, you can just render an H.264 video (.mov files that pass the H.264 test¹ will automatically be renamed to .mp4) to that folder from your screen-recording application of choice, sit back for a few minutes (or start writing the post) and wait for the pre-populated TextMate blog post to pop open. Then type or paste your text in, hit Control-Command-P (assuming you’re set up for TextMate->WordPress blogging) and your video is posted.

Improvements

The first thing that happened in this rewrite was a major simplification of the WEBM encoding process, which now only takes a third of the time it did before and the files are not significantly larger in most cases. They may even look better. The old command involved two passes, one of them turning out to be unnecessary; ffmpeg is smarter than I thought. The new command (using the variables from the script) is just /usr/local/bin/ffmpeg -i "$FILENAME" -b 614400 -s $MAXSIZE -aspect 16:9 "$BASENAME".webm.

I also added better logging, optional progress reports with Growl, batch file handling and a few other refinements. The logging, if enabled in the config at the top, will output information including processing time for each file and total processing time to STDOUT (command line) and to the system log where you can watch it from Console. If you enable growl notifications, the messages at the beginning and end of each conversion will be “growled” on your screen.

Using the script

You’ll need a couple of additional utilities to do the conversions.

Requirements

To run the script and do the encoding, you’ll need ffmpeg and ffmpeg2theora. You can install ffmpeg from Homebrew (brew install ffmpeg) quite easily, if you’re set up for that. You can also pull the binary out of ffmpegX with a few simple steps. You can, of course, build your own if you have the developer tools installed.

ffmpeg2theora (needed for OGV conversion) no longer appears to be available from brew, but there are OS X binaries (and source) available for download.

The script is below. You’ll want to modify the information in the configuration section at the top, and modify the template output at the bottom to suit your needs. Anything between <<-POSTTEMPLATE and POSTTEMPLATE is freeform text that you can edit in any way you like, including the use of $variables. If you’d rather do something such as copy a shortcode to your clipboard and show a Growl message, just use pbcopy and growlnotify. You could also just comment out the template part and let Hazel tell you when it’s done, if you want to handle things more manually. I’ll let you work out the details on that.

Command line usage

To use the script from the command line, save it as “html5encode.sh” somewhere in your path, change to its parent folder in Terminal and run chmod a+x html5encode.sh. The you can run it on any h.264 .mov or .mp4 file and it will handle the rest. It also handles batches, so you can specify multiple targets or use *.mp4 to run it on all .mp4 files in a folder. It will process them one by one, creating new folders for each.

Hazel

If you want to run it with Hazel, you can follow the same procedure and set up a rule that runs a shell script when a .mov or .mp4 is detected in a watch folder. The script can then be run as a command or be pasted in its entirety into the Hazel script editor within the rule.

You should be able to pull off using this with OS X Folder Actions, as well, but I haven’t played with that yet.

Bonus tip: If you name your original .mov or .mp4 file with a CamelCased name, it will break apart the filename and create the title of your post based on it. Not an essential feature, but kind of nifty. Well, I think so anyway.

The script

#!/bin/bash
#################################################################
### html5encode.sh by Brett Terpstra and Mike Rose
### Published 05/01/2011
### Freely distributed, modifications welcomed (with attribution)
#################################################################
### Configuration ###############################################
#################################################################
MAXSIZE="960x540"
DISPLAYWIDTH="600"
DISPLAYHEIGHT="338"
SSHURL="username@ssh.host.name"
SSHDIR="/server/folder/target/for/rsync/"
WEBDIR="/front-end/url/path/video/" # used for blog template
LOGGING=true # send status messages and times to STDOUT and syslog
GROWLLOG=false # duplicate messages to growl, if installed
##################################################################
### END Configuration ############################################
##################################################################

# function to handle logging (if enabled) to STDOUT and STDERR 
# as well as Growl (if enabled)
function logit() {
  if $LOGGING ; then
    logger -st "HTML5 Encoder" "$1"
    if $GROWLLOG ; then
      /usr/local/bin/growlnotify -t "HTML5 Encoder" -a "Terminal" -m "$1"
    fi
  fi
}

# Count the inputs for log message
if [[ $# -gt 1 ]]; then
  logit "Starting batch conversion."
  maintimer1=`date '+%s'`
else
  logit "Starting HTML5 Encoder"
fi

# Loop through each passed file
for file in "$@"; do
  timer1=`date '+%s'`
  INPUT=$file
  # Check that input file is H.264
  isH264=`mdls -raw -name kMDItemCodecs $INPUT|grep H.264`

  if [ !$isH264 ] ; then
    logit "$INPUT is not h.264"
    continue
  fi

  DIRNAME=`dirname "$INPUT"`
  FILENAME=`basename "$INPUT"`
  BASENAME=${FILENAME%%.*}
  logit "Conversion of $FILENAME started on `date '+%D'` at `date '+%r'`"
  bytesize=`stat -f '%z' $FILENAME`
  filesize=`echo "scale = 2 ; $bytesize/1048576"|bc -lq`
  cd "$DIRNAME"
  if [ -d "$BASENAME" ]; then
    logit "Found $FILENAME, but directory $BASENAME already exists. Aborting"
    continue
  fi
  mkdir "$BASENAME"
  mv "$FILENAME" "$BASENAME/"
  cd "$BASENAME"
  if [[ ${FILENAME#*.} -eq "mov" ]]; then mv $FILENAME ${BASENAME}.mp4; fi  
  FILENAME=${BASENAME}.mp4
  /usr/local/bin/ffmpeg -i "$FILENAME" -b 614400 -s $MAXSIZE -aspect 16:9 "$BASENAME".webm
  logit "Completed webm conversion"
  /usr/local/bin/ffmpeg2theora --videoquality 5 --audioquality 1 --max_size $MAXSIZE "$FILENAME" -o "$BASENAME.ogv"
  logit "Completed ogv conversion"
  /usr/local/bin/ffmpeg -i "$FILENAME" -ss 0 -vframes 1 -vcodec mjpeg -f image2 "${BASENAME}Poster.jpg"
  logit "Created poster image"

  # Create a title from camelcased filename
  TITLE=`echo "$BASENAME"|sed 's/\([A-Z][^A-Z]*\)/& /g'|sed 's/ $//'`
  SERVER=`echo "$SSHURL"|sed 's/^.*\@//'`
  logit "Uploading to $SERVER..."
  cd ..
  rsync -v -r -e ssh "`pwd`/$BASENAME" $SSHURL:$SSHDIR
  logit "Finished Uploading"
  # remove trailing slash from $WEBDIR
  WEBDIR=`echo "$WEBDIR"|sed 's/\/$//'`
  cat > "$BASENAME/$BASENAME.blog.markdown" <<-POSTTEMPLATE
Type: Blog Post (Markdown)
Blog: BlogsmithVideo
Title: $TITLE
Keywords: 
Status: draft
Pings: On
Comments: On
Category: Tutorial

Synopsis

[video mp4="$WEBDIR/$BASENAME/$BASENAME.mp4" ogg="$WEBDIR/$BASENAME/$BASENAME.ogv" webm="$WEBDIR/$BASENAME/$BASENAME.webm" poster="$WEBDIR/$BASENAME/${BASENAME}Poster.jpg" preload="true" width="$DISPLAYWIDTH" height="$DISPLAYHEIGHT"]

<!--more-->
Transcript

POSTTEMPLATE

  open "$BASENAME/$BASENAME.blog.markdown" -a "TextMate.app"
  timer2=`date '+%s'`
  time=`echo "scale=2 ; ($timer2-$timer1)/60"|bc -lq`
  logit "Conversion of $FILENAME complete"
  logit "It took $time minutes to process a ${filesize}M MP4 to webm and ogv and upload to `echo "$SSHURL"|sed 's/^.*\@//'`."
done
if [[ $# -gt 1 ]]; then
  maintimer2=`date '+%s'`
  total=`echo "scale=2 ; ($maintimer2-$maintimer1)/60"|bc -lq`
  logit "Batch conversion complete, total time $total minutes."
fi

Hopefully this will make a few people’s lives easier. Mike and I have battle tested it and it seems to be a pretty great fit for our needs. If you modify it for your own needs, please share (with credit) so that it can become as well-rounded as possible!