automating repetitive tasks by scripting Cinelerra EDL, part I

I produce a bi-monthly video of my band's jam sessions:
http://feeds.feedburner.com/StormpigsPodcast

The format of the video edit is roughly the same for every video:

• intro titles
  
• staff titles
• songs
• end credits
  

Since the Cinelerra EDL (edit decision list) is an XML text file, it occurred to me that I should be able to add the titles by editing this text file.

Cinelerra's Edit Decision List (EDL)
Let's look at Cinelerra's EDL. This is an XML text file that gets written when you save a Cinelerra project. The easiest way to review the EDL is to open it in a browser, as the browser will let you expand and collapse the XML elements. Here's a screen cap of the EDL in Firefox:


When I create my monthly video podcast, I place title effects on Cinelerra's timeline for each song of about a dozen songs. The title effect looks like this:


Placing title effects is a real time sink, especially when I have to place one for each song. I place title effects about 20 seconds into each song. I separate each song by placing a label between the clips. So the labels indicate where in the video track I will need to place title effects. You can see that the title effect is placed after the appearance of a label in Cinelerra's timeline below:


The Task
My goal is to have a shell script find the position of the label in the EDL and plop the title effects about 20s after the appearance of the label. After I add the labels programmatically, I can do fancier edits to the project later on.

In today's post, I will describe a bit more about my workflow and the EDL, and also show you a basic shell script command to insert a title effect into the EDL.

The Procedure
First, I put my basic edit together assembling the clips on the timeline, doing my audio and video fades and placing labels between the clips. This is my first round of edits. At this point, my timeline is very simple. I only have a video track and two audio tracks:


I will use a script to insert a title effect into the EDL file. In the Cinelerra EDL, a title effect (designated by the PLUGINSET XML element) looks like this:


Let's inspect the EDL in order to find the block of labels. If you grep for 'LABEL TIME' in the EDL, you'll find an XML code block that looks like this:


I will need to edit the video TRACK that appears directly beneath the LABELS in the EDL. Since there is quite a bit of confusing repetition in the EDL file, finding the list of LABELs helps identify which TRACK we need to edit. I've capitalized the words in the above sentence to highlight the XML elements you'll need to look for when editing EDL.

Using a bash shell script, I'll add a title effect after the last MASKAUTOS XML tag, but before the closing TRACK tag. Again, this is the closing TRACK tag of the video TRACK:


I've created a second file containing just one title effect, the PLUGINSET example from above:


I will then use a bit of sed (stream editor) magic to insert that file into the right spot in the main Cinelerra EDL for the project. Here is the script command that will place the PLUGINSET into the correct position in the XML:
sed -e "/LABEL TIME/,/^<\/MASKAUTOS/{ /<\/MASKAUTOS>/r titleEffectPluginset.txt
}" cinelerraEdl.xml > cinelerraEdlNewEdit.xml

Let's break this apart:
sed -e "/LABEL TIME/ # This finds the first occurrence of "LABEL TIME" (a label) in the EDL. Remember that the appearance of labels in the EDL tells us that the video TRACK in which we need to insert our title effect comes next in the file.

,/^<\/MASKAUTOS/{ # This finds the first occurrence of a closing MASKAUTOS tag after the string "LABEL TIME". Our title effect, the PLUGINSET, will be inserted after this closing tag.

/<\/MASKAUTOS>/r titleEffectPluginset.txt # Insert the title effect boilerplate after the end MASKAUTOS XML tag

}" cinelerraEdl.xml > cinelerraEdlNewEdit.xml # Have sed perform the edit on "cinelerraEdl.xml", but save the output to "cinelerraEdlNewEdit.xml

The Result - Before


The Result - After


You can see that sed has inserted the title effect (the PLUGINSET XML boilerplate) in between the closing MASKAUTOS tag and the closing tag of the video TRACK. Pretty cool!

This is a simple example of editing Cinelerra EDL, but is the first step to helping me automate otherwise manual tasks in my monthly video podcast creation.

I will try to expand upon this subject in future posts.

-enjoy-
The Mule

iTunes/iPod video workflow, scripted

I am going to revisit my conversation about workflow. Here's what we briefly discussed:
- the transformation of an idea (video) into reality and distributing it (in this case, using iTunes)

With today's internet, the steps involved are broad-based:
- idea creation, storyboard, distribution, production, archiving, marketing

Update 2008/11/22
I've added a few more details to my workflow in a new post here.
end update

I shall give you an example and how I reduced the amount of time spent creating and distributing my content.

When not working, my world is playing music with a band of itinerant musicians called the StormPigs. We gather together once every couple of months to play freeform music. No prior thought involved, just play. The joy of this is being together and having a good time. Otherwise, we are all busy professionals with full-time jobs and families. In remembrance of that good time, I produce videos of the event, distributed via iTunes and YouTube.

As time seems compressed these days, I want to spend as little time as possible behind the keyboard (though I am a technologist by trade). And as readers of this blog know, I am avid proponent of Linux. The beauty of Linux is that the system is completely configurable and flexible. But with this power comes a price. You have to invest the time to learn the shell, some bits of scripting and other Linux arcana. Consequently, it is daunting to the newcomer. But the benefits return to you many times over, as time you once spent on minutiae can now be spent thinking of new ideas for shows and creating new content, rather than simply focusing on the details of getting a file up to the server or copy and pasting content from one application to the other.

I am not a programmer of fanciful GUI front ends. I am a guy who just needs to get work done. So I try to solve my problems in the simplest way possible using the Linux programs and bash shell scripts to tie multiple programs together.

Here is the latest problem I needed to solve:
- how to I get my videos onto the web and in an iTunes ready form as quickly as possible?

In that light, I have come up with some scripts to help me on my way:
- the first encodes my editing video project into various formats (HDV, DVD, podcast)
- the second creates the list of songs from the video that will go into the podcast
- the third merges that songlist information into the podcast
- the fourth creates a new iTunes RSS feed (XML file) from the songlist information
- the fifth uploads the new podcast to my webserver
- the last is a wrapper that starts the other five

The encoding script is most useful and is the basis for the others. This script takes 720P video and 48Khz MPEG, Layer II stereo audio output rendered from a Cinelerra project and converts it to various formats: MPEG2 Program Stream, HDV, DVD and iTunes/iPod compatible formats. You will need the following programs installed for this script to work:
-mplex
-vlc
-ffmpeg

I've found the quality of each output file to be very good to excellent.

You can wrap some validation and input around this script, but the guts of the script look like this, where the arguments enclosed in curly braces will be replaced by the names of your input and output files:
#!/bin/bash
echo "Input: M2A audio and MPEG2 720P video streams"
echo "Output: program stream"
mplex -f 3 -b 2000 ${AUDIO} ${VIDEO} -o ${PS}

echo "Input: program stream"
echo "Output: MPEG2-TS, HDV"
vlc ${PS} --sout '#duplicate{dst=std{access=file,mux=ts,dst="'${HDV}'"}}' vlc:quit

echo "Input: MPEG2-TS, HDV"
echo "Output: MPEG2 DVD"
ffmpeg -i ${HDV} -target dvd -threads 8 ${DVD}

echo "Input: MPEG2 DVD"
echo "Output: iTunes/iPod compatible MP4"
ffmpeg -y -i ${DVD} -an -v 1 -threads 8 -vcodec h264 -b 250k -bt 175k -refs 1 -loop 1 -deblockalpha 0 -deblockbeta 0 -parti4x4 1 -partp8x8 1 -me full -subq 1 -me_range 21 -chroma 1 -slice 2 -bf 0 -level 30 -g 300 -keyint_min 30 -sc_threshold 40 -rc_eq 'blurCplx^(1-qComp)' -qcomp 0.7 -qmax 51 -qdiff 4 -i_qfactor 0.71428572 -maxrate 450k -bufsize 2M -cmp 1 -s 720x480 -f mp4 -pass 1 /dev/null

ffmpeg -y -i ${DVD} -v 1 -threads 8 -vcodec h264 -b 250k -bt 175k -refs 1 -loop 1 -deblockalpha 0 -deblockbeta 0 -parti4x4 1 -partp8x8 1 -me full -subq 6 -me_range 21 -chroma 1 -slice 2 -bf 0 -level 30 -g 300 -keyint_min 30 -sc_threshold 40 -rc_eq 'blurCplx^(1-qComp)' -qcomp 0.7 -qmax 51 -qdiff 4 -i_qfactor 0.71428572 -maxrate 450k -bufsize 2M -cmp 1 -s 720x480 -acodec aac -ab 160k -ar 48000 -ac 2 -f mp4 -pass 2 -threads 8 ${MP4}

Stay tuned for discussions of the remaining scripts to make your life easier,
CM