automating repetitive tasks by scripting Cinelerra, part II

This is a follow up to my earlier article:
/2010/01/automating-repetitive-tasks-by.html

Saving Time
In this entry, I discuss writing a shell script that automatically adds a second video track to your Cinelerra project. This comes in handy for projects that you do repeatedly. For example, a weekly podcast that uses a standard watermark.

For instance, I produce a monthly podcast for my band. You can see the pig watermark in the screen cap below:


In Cinelerra, the pig watermark is created by adding a second video track to my base Cinelerra project. This second video track includes a jpg stretched to the length of the first video track. Also, there is a chroma key effect applied to the pig track in order for it to be transparent against the background of the jam video.

The two items, a graphic of a pig and a video track with the image of the pig and applied effects are represented by XML in the Cinelerra EDL. Remember, the Cinelerra EDL (edit decision list) is the XML file that represents all the edits that you've performed in creating your video masterpiece. It looks like this:


In this blog post, I will show you a shell script that inserts these two chunks of XML into a basic Cinelerra project file. For my script to work properly, note that the base project needs at least one video track and at least one label. I'll get into the reason for that later. The basic project file can also have audio tracks as well.

The Asset, the Asset EDL and the Video Track EDL
One asset and two sections of XML represent the video track of the pig:
1) Here is the graphic of the pig, as a Google Doc.

2) Here is the EDL of the asset (pig2.jpg):

The EDL is also available in a Google Doc.

3) Here is the EDL of the video track of the pig:

The full EDL is also available as a Google Doc.

Since I use the pig watermark on each podcast, I reuse the two chunks of XML above for each installment of my monthly podcast. So I've saved both the asset and the video track XML above from an older project to separate files (pigAsset.txt and pigVideoTrack.txt). Using the shell script, I will insert both of these files at the appropriate places in my base Cinelerra project (my latest podcast). To make life simple, my base project has only one track, the video track of the band rehearsal.

Disclaimer
You're probably saying that this seems like a lot of work and you maybe getting confused. Well..it kind of is a pain in the ass, but it will be worth it when you can whip out a shell script and automatically add your watermark to your project! But I digress..

Shell Script Fun
Here is the meat of the script:
PWD=/mnt/scripts/scriptsTimeline/addPig
ASSET=$PWD/pigAsset.txt
TRACK=$PWD/pigVideoTrack.txt
OLDLENGTH=85965
NEWLENGTH=$(videoLength.sh $INPUT)

echo "OLDLENGTH is $OLDLENGTH"
echo "NEWLENGTH is $NEWLENGTH"
echo "INPUT is $INPUT"
echo "OUTPUT is $OUTPUT"

# adds both via a pipe WORKING
sed -e "s/^<\/ASSETS>/<\!-- $REM -->/g" $INPUT | sed -e "/<\!-- $REM -->/r $ASSET" | sed -e "/LABEL TIME/,/^<\/LABELS/{ /<\/LABELS>/r $TRACK
}" | sed -e "s/$OLDLENGTH/$NEWLENGTH/g" > $OUTPUT
The full script is here, again as a Google Doc.

The core of the script revolves around a neat bit of sed (stream editor) trickery that I mentioned in my first scripting article. In this case, the core function of the script does something similar, but still hinges upon the appearance of labels in the EDL ("LABEL TIME") as a positional reference within the EDL to perform the insertion:

sed -e "s/^<\/ASSETS>/<\!-- $REM -->/g" $INPUT | sed -e "/<\!-- $REM -->/r $ASSET" | sed -e "/LABEL TIME/,/^<\/LABELS/{ /<\/LABELS>/r $TRACK
}" | sed -e "s/$OLDLENGTH/$NEWLENGTH/g"

If we break this command apart, we see that it does a few things:
1) it adds the pig as an asset (the $ASSET constant in the second sed command) to a project (the $INPUT constant listed in the first sed command)

sed -e "s/^<\/ASSETS>/<\!-- $REM -->/g" $INPUT | sed -e "/<\!-- $REM -->/r $ASSET" 

2) it adds a video track of said pig (the $TRACK constant) with this part of the command

| sed -e "/LABEL TIME/,/^<\/LABELS/{ /<\/LABELS>/r $TRACK

3) to make the new track the same length as the new track, it calculates the length of the base track and then substitutes that value for the initial length of the pig track.

| sed -e "s/$OLDLENGTH/$NEWLENGTH/g"

Now, I threw a curveball in there with that NEWLENGTH constant. This number is the current length of the video track in your input file and it is calculated via a second, slightly shorter script called "videoLength.sh".

Here is the Google Doc of that script.

Putting It All Together
To test how the script adds a track, download the following components to the same directory:
1) the jpg of the pig, save as pig2.jpg
2) the xml representing the pig, save as pigAsset.txt
3) the xml representing a new video track with effects, save as pigVideoTrack.txt
4) a test Cinelerra project, save as testProject.xml
5) the script, save as addPig.sh
6) the videoLength shell script

Don't forget to make both scripts executable:
chmod a+x addPig.sh
chmod a+x videoLength.sh, save as videoLength.sh

Script usage:
./addPig.sh [inputEDLfile] [outputEDLfile]

ex.
./addPig.sh testProject.xml trackAdded.xml

You may want to load testProject.xml into Cinelerra first to see that it only has one track. The script will create a new EDL file once it is run. Once you run the script, load your output EDL into Cinelerra. You should see the new track with the associated effects. The track should be the same length as testProject's video track.

Most likely, though, you won't see the pig, just because the pig is not in the same asset directory as the EDL specifies. Simply change pigAsset.txt to reference the pig into the proper directory, rerun the script and all should be good!

Summary
These scripts definitely can be improved upon, especially by someone who has better tools than sed to manipulate XML. But it was fun to automate a mundane task in Cinelerra. Now I don't have to manually add that watermark to my project anymore and neither do you!

the mule

mpeg toc maker

I got tired of having Cinelerra hang while create table of contents files for a list MPEG videos. So I wrote my own script to create the table of contents files for a list of MPEGs in a directory. Once you specify the LIST, the script plops the table of contents files in your users' .bcast directory in ~/.bcast.

I output the file name, start time, end time and elapsed time for the table of contents creation. This output looks like this:

SCRIPTSTART:                  20100130_121201
       FILE:                     mvi_0703.m2t      START: 20100130_121201        END: 20100130_121204    TIME(s):       3
       FILE:                     mvi_0705.m2t      START: 20100130_121204        END: 20100130_121222    TIME(s):      18
       FILE:                     mvi_0706.m2t      START: 20100130_121222        END: 20100130_121255    TIME(s):      33
       FILE:                     mvi_0708.m2t      START: 20100130_121255        END: 20100130_121325    TIME(s):      30
       FILE:                     mvi_0709.m2t      START: 20100130_121325        END: 20100130_121341    TIME(s):      16
       FILE:                     mvi_0710.m2t      START: 20100130_121341        END: 20100130_121347    TIME(s):       6
       FILE:                     mvi_0711.m2t      START: 20100130_121347        END: 20100130_121401    TIME(s):      14
       FILE:                     mvi_0713.m2t      START: 20100130_121401        END: 20100130_121403    TIME(s):       2
       FILE:                     mvi_0714.m2t      START: 20100130_121403        END: 20100130_121415    TIME(s):      12
       FILE:                     mvi_0715.m2t      START: 20100130_121415        END: 20100130_121415    TIME(s):       0
       FILE:                     mvi_0716.m2t      START: 20100130_121416        END: 20100130_121416    TIME(s):       0
       FILE:                     mvi_0717.m2t      START: 20100130_121416        END: 20100130_121421    TIME(s):       5
       FILE:                     mvi_0718.m2t      START: 20100130_121421        END: 20100130_121442    TIME(s):      21
       FILE:                     mvi_0719.m2t      START: 20100130_121442        END: 20100130_121442    TIME(s):       0
       FILE:                     mvi_0720.m2t      START: 20100130_121442        END: 20100130_121446    TIME(s):       4
       FILE:                     mvi_0721.m2t      START: 20100130_121446        END: 20100130_121449    TIME(s):       3
       FILE:                     mvi_0722.m2t      START: 20100130_121449        END: 20100130_121454    TIME(s):       5
       FILE:                     mvi_0724.m2t      START: 20100130_121454        END: 20100130_121455    TIME(s):       1
       FILE:                     mvi_0725.m2t      START: 20100130_121455        END: 20100130_121507    TIME(s):      12
       FILE:                     mvi_0727.m2t      START: 20100130_121507        END: 20100130_121515    TIME(s):       8
       FILE:                     mvi_0728.m2t      START: 20100130_121515        END: 20100130_121531    TIME(s):      16
       FILE:                     mvi_0730.m2t      START: 20100130_121531        END: 20100130_121534    TIME(s):       3
       FILE:                     mvi_0731.m2t      START: 20100130_121534        END: 20100130_121535    TIME(s):       1
  SCRIPTEND:                  20100130_121535 RUNTIME(s):             213

*** Update 2010/05/28 ***
I added a little logic to the beginning of the script that checks for the presence of a single argument to the script. That argument is the name of a single file to convert. So now, you have two choices:
1) convert a list of mpeg files as specified in the LIST variable or
2) convert a single file via the command line like so:
./tocMaker.sh videoToConvert.m2t
*** end update ***

Here is the script:

#!/bin/bash
#
# tocMaker.sh
# Author: SCF
# Website: http://crazedmuleproductions.blogspot.com
#
# Purpose: This script builds the table of contents files for a list of mpeg files
#
# Input: List of mpeg files as specified in the LIST variable or a single file via command line like so:
#  ./tocMaker.sh videoToConvert.m2t
#
# Output: Runs mpeg3toc and puts toc's in your users' .bcast directory
#
DEBUG=
if [ $1 ]
then
 LIST=$1
else
 LIST="*.m2t"
fi
echo "converting $LIST"
TOTALTIME=0
function dt {
       DATE=$(date +"%Y%m%d_%H%M%S")
       echo $DATE
}
function st {
       SECONDS=$(date +%s)
       echo $SECONDS
}
PWD=$(pwd)
DIR=$(echo $PWD | sed 's/\//_/g' | sed 's/^_//g')

SCRIPTSTART=$(dt)
printf "%13s %32s\n" "SCRIPTSTART:" $SCRIPTSTART

for FILE in $(ls -1 $LIST)
do
       START=$(dt)
       COUNTBEGIN=$(st)
       NEWFILE=$(echo $FILE | sed 's/\./_/g')
       FULLPATH="${DIR}_${NEWFILE}.toc"
       ORIG_PATH="$PWD/$FILE"
       if [ $DEBUG ]
       then
               echo "$FILE was: $FILE, is: $NEWFILE"
               echo "FULLPATH is $FULLPATH"
               echo "command is mpeg3toc $ORIG_PATH ~/.bcast/$FULLPATH"
       fi
       mpeg3toc $ORIG_PATH ~/.bcast/$FULLPATH
       END=$(dt)
       COUNTFINISH=$(st)
       ELAPSED=$(expr $COUNTFINISH - $COUNTBEGIN)
       printf "%13s %32s %11s %15s %11s %15s %11s %7d\n" "FILE:" $FILE "START:" $START "END:" $END "TIME(s):" $ELAPSED
       TOTALTIME=$(expr $ELAPSED + $TOTALTIME)
done
SCRIPTEND=$(dt)
printf "%13s %32s %11s %15d\n" "SCRIPTEND:" $SCRIPTEND "RUNTIME(s):" $TOTALTIME

cheers,
the mule

automating repetitive tasks by scripting Cinelerra EDL, part I

I produce a bi-monthly video of my band's jam sessions:
http://feeds.feedburner.com/StormpigsPodcast

The format of the video edit is roughly the same for every video:

• intro titles
  
• staff titles
• songs
• end credits
  

Since the Cinelerra EDL (edit decision list) is an XML text file, it occurred to me that I should be able to add the titles by editing this text file.

Cinelerra's Edit Decision List (EDL)
Let's look at Cinelerra's EDL. This is an XML text file that gets written when you save a Cinelerra project. The easiest way to review the EDL is to open it in a browser, as the browser will let you expand and collapse the XML elements. Here's a screen cap of the EDL in Firefox:


When I create my monthly video podcast, I place title effects on Cinelerra's timeline for each song of about a dozen songs. The title effect looks like this:


Placing title effects is a real time sink, especially when I have to place one for each song. I place title effects about 20 seconds into each song. I separate each song by placing a label between the clips. So the labels indicate where in the video track I will need to place title effects. You can see that the title effect is placed after the appearance of a label in Cinelerra's timeline below:


The Task
My goal is to have a shell script find the position of the label in the EDL and plop the title effects about 20s after the appearance of the label. After I add the labels programmatically, I can do fancier edits to the project later on.

In today's post, I will describe a bit more about my workflow and the EDL, and also show you a basic shell script command to insert a title effect into the EDL.

The Procedure
First, I put my basic edit together assembling the clips on the timeline, doing my audio and video fades and placing labels between the clips. This is my first round of edits. At this point, my timeline is very simple. I only have a video track and two audio tracks:


I will use a script to insert a title effect into the EDL file. In the Cinelerra EDL, a title effect (designated by the PLUGINSET XML element) looks like this:


Let's inspect the EDL in order to find the block of labels. If you grep for 'LABEL TIME' in the EDL, you'll find an XML code block that looks like this:


I will need to edit the video TRACK that appears directly beneath the LABELS in the EDL. Since there is quite a bit of confusing repetition in the EDL file, finding the list of LABELs helps identify which TRACK we need to edit. I've capitalized the words in the above sentence to highlight the XML elements you'll need to look for when editing EDL.

Using a bash shell script, I'll add a title effect after the last MASKAUTOS XML tag, but before the closing TRACK tag. Again, this is the closing TRACK tag of the video TRACK:


I've created a second file containing just one title effect, the PLUGINSET example from above:


I will then use a bit of sed (stream editor) magic to insert that file into the right spot in the main Cinelerra EDL for the project. Here is the script command that will place the PLUGINSET into the correct position in the XML:
sed -e "/LABEL TIME/,/^<\/MASKAUTOS/{ /<\/MASKAUTOS>/r titleEffectPluginset.txt
}" cinelerraEdl.xml > cinelerraEdlNewEdit.xml

Let's break this apart:
sed -e "/LABEL TIME/ # This finds the first occurrence of "LABEL TIME" (a label) in the EDL. Remember that the appearance of labels in the EDL tells us that the video TRACK in which we need to insert our title effect comes next in the file.

,/^<\/MASKAUTOS/{ # This finds the first occurrence of a closing MASKAUTOS tag after the string "LABEL TIME". Our title effect, the PLUGINSET, will be inserted after this closing tag.

/<\/MASKAUTOS>/r titleEffectPluginset.txt # Insert the title effect boilerplate after the end MASKAUTOS XML tag

}" cinelerraEdl.xml > cinelerraEdlNewEdit.xml # Have sed perform the edit on "cinelerraEdl.xml", but save the output to "cinelerraEdlNewEdit.xml

The Result - Before


The Result - After


You can see that sed has inserted the title effect (the PLUGINSET XML boilerplate) in between the closing MASKAUTOS tag and the closing tag of the video TRACK. Pretty cool!

This is a simple example of editing Cinelerra EDL, but is the first step to helping me automate otherwise manual tasks in my monthly video podcast creation.

I will try to expand upon this subject in future posts.

-enjoy-
The Mule

iTunes/iPod video workflow, scripted

I am going to revisit my conversation about workflow. Here's what we briefly discussed:
- the transformation of an idea (video) into reality and distributing it (in this case, using iTunes)

With today's internet, the steps involved are broad-based:
- idea creation, storyboard, distribution, production, archiving, marketing

Update 2008/11/22
I've added a few more details to my workflow in a new post here.
end update

I shall give you an example and how I reduced the amount of time spent creating and distributing my content.

When not working, my world is playing music with a band of itinerant musicians called the StormPigs. We gather together once every couple of months to play freeform music. No prior thought involved, just play. The joy of this is being together and having a good time. Otherwise, we are all busy professionals with full-time jobs and families. In remembrance of that good time, I produce videos of the event, distributed via iTunes and YouTube.

As time seems compressed these days, I want to spend as little time as possible behind the keyboard (though I am a technologist by trade). And as readers of this blog know, I am avid proponent of Linux. The beauty of Linux is that the system is completely configurable and flexible. But with this power comes a price. You have to invest the time to learn the shell, some bits of scripting and other Linux arcana. Consequently, it is daunting to the newcomer. But the benefits return to you many times over, as time you once spent on minutiae can now be spent thinking of new ideas for shows and creating new content, rather than simply focusing on the details of getting a file up to the server or copy and pasting content from one application to the other.

I am not a programmer of fanciful GUI front ends. I am a guy who just needs to get work done. So I try to solve my problems in the simplest way possible using the Linux programs and bash shell scripts to tie multiple programs together.

Here is the latest problem I needed to solve:
- how to I get my videos onto the web and in an iTunes ready form as quickly as possible?

In that light, I have come up with some scripts to help me on my way:
- the first encodes my editing video project into various formats (HDV, DVD, podcast)
- the second creates the list of songs from the video that will go into the podcast
- the third merges that songlist information into the podcast
- the fourth creates a new iTunes RSS feed (XML file) from the songlist information
- the fifth uploads the new podcast to my webserver
- the last is a wrapper that starts the other five

The encoding script is most useful and is the basis for the others. This script takes 720P video and 48Khz MPEG, Layer II stereo audio output rendered from a Cinelerra project and converts it to various formats: MPEG2 Program Stream, HDV, DVD and iTunes/iPod compatible formats. You will need the following programs installed for this script to work:
-mplex
-vlc
-ffmpeg

I've found the quality of each output file to be very good to excellent.

You can wrap some validation and input around this script, but the guts of the script look like this, where the arguments enclosed in curly braces will be replaced by the names of your input and output files:
#!/bin/bash
echo "Input: M2A audio and MPEG2 720P video streams"
echo "Output: program stream"
mplex -f 3 -b 2000 ${AUDIO} ${VIDEO} -o ${PS}

echo "Input: program stream"
echo "Output: MPEG2-TS, HDV"
vlc ${PS} --sout '#duplicate{dst=std{access=file,mux=ts,dst="'${HDV}'"}}' vlc:quit

echo "Input: MPEG2-TS, HDV"
echo "Output: MPEG2 DVD"
ffmpeg -i ${HDV} -target dvd -threads 8 ${DVD}

echo "Input: MPEG2 DVD"
echo "Output: iTunes/iPod compatible MP4"
ffmpeg -y -i ${DVD} -an -v 1 -threads 8 -vcodec h264 -b 250k -bt 175k -refs 1 -loop 1 -deblockalpha 0 -deblockbeta 0 -parti4x4 1 -partp8x8 1 -me full -subq 1 -me_range 21 -chroma 1 -slice 2 -bf 0 -level 30 -g 300 -keyint_min 30 -sc_threshold 40 -rc_eq 'blurCplx^(1-qComp)' -qcomp 0.7 -qmax 51 -qdiff 4 -i_qfactor 0.71428572 -maxrate 450k -bufsize 2M -cmp 1 -s 720x480 -f mp4 -pass 1 /dev/null

ffmpeg -y -i ${DVD} -v 1 -threads 8 -vcodec h264 -b 250k -bt 175k -refs 1 -loop 1 -deblockalpha 0 -deblockbeta 0 -parti4x4 1 -partp8x8 1 -me full -subq 6 -me_range 21 -chroma 1 -slice 2 -bf 0 -level 30 -g 300 -keyint_min 30 -sc_threshold 40 -rc_eq 'blurCplx^(1-qComp)' -qcomp 0.7 -qmax 51 -qdiff 4 -i_qfactor 0.71428572 -maxrate 450k -bufsize 2M -cmp 1 -s 720x480 -acodec aac -ab 160k -ar 48000 -ac 2 -f mp4 -pass 2 -threads 8 ${MP4}

Stay tuned for discussions of the remaining scripts to make your life easier,
CM