converting 1080p video, part II

A couple nights back, I spent a few hours finding a usable intermediate format for 1080P video editing. Of course, my goal was to prove that I can still use my preferred NLE Cinelerra in order to edit the gorgeous, high definition output from the Canon EOS 5D Mark II. An alternative way of putting it is that this is my justification for purchasing what will amount to a $4000 camera!

Import: which format works well?
Since Cinelerra is rather picky about the types of video it accepts (see my original Quicktime compatability document), it was a challenge to find a Cinelerra-editable intermediate format for that test 1080p video I told you about in the first part of this two-part blog entry. The video is a five-second city scene of a bicycler launching from a stop. That video is 23MB.

Keep in mind that various versions of media encoders (like ffmpeg and mplayer) do not always implement codecs and container formats consistently. Also, program dependencies for the encoders across different Linux distros complicate matters. So what I state as working below may not work for you. (Your mileage may vary). Note that I am currently using Fedora 7, 64-bit.

Caveats aside, transcoding the biker video to the following formats did NOT work:
- I420 from ffmpeg
- rawvideo (RAW YV12) from ffmpeg

Here are a couple formats that did work:
1) MJPEG output from ffmpeg
2) RAW YV12 output from mplayer

MJPEG from FFmpeg
Playback Performance
This format seems to be a likely candidate, as I got about 15fps on my dual, quad core Dell SC1430 in Cinelerra.

File Size
The resulting file size is small..about 35MB for five seconds of 1080P.

Image Quality Compared to Original
The image quality is very good. However, the colors seem bit oversaturated and dark in contrast to the original. With a bit of tweaking to the conversion command, I might be able to get more realistic colors out of the conversion. Here's a comparison shot:


Editing Notes
No problems surfaced with a couple initial edits and renderings. As this is a compressed format, there will be some loss of quality over successive encodings.

Here is the command I used to convert the video:
ffmpeg -i Video1.MOV -b 3000k -vcodec mjpeg -ab 256k -ar 44100 -acodec mpeg4aac -coder 1 -flags +loop -cmp +chroma -partitions +parti4x4+partp8x8+partb8x8 -me hex -subq 5 -me_range 16 -g 250 -keyint_min 25 -sc_threshold 40 -i_qfactor 0.71 testmjpeg.mov

This is the same conversion command that I ran in part I of this blog entry.

Information about the file looks like this in mplayer:
Playing testmjpeg.mov. ISO: File Type Major Brand: Original QuickTime Quicktime/MOV file format detected. [mov] Video stream found, -vid 0 VIDEO: [jpeg] 1920x1080 24bpp 30.000 fps 0.0 kbps ( 0.0 kbyte/s) ========================================================================== Opening video decoder: [ffmpeg] FFmpeg's libavcodec codec family Selected video codec: [ffmjpeg] vfm: ffmpeg (FFmpeg MJPEG decoder) ==========================================================================


RAWYV12 from mplayer
Playback Performance
Playback is slow, 7fps, roughly 1/2 the speed of MJPEG. Not a show stopper.

File Size
Huge. About 450MB for about five seconds of video. Woah!

Image Quality Compared to Original
Image quality is very good as well. Less saturated then the mjpeg version. That's what almost 100MB/sec pays for.


Editing Notes
This is an uncompressed format, corresponding to the YUV 4:2:0 spec:
http://en.wikipedia.org/wiki/YUV#Y.27UV420p_.28and_Y.27V12.29

Also, here is an excellent article on understanding color sampling:
http://www.dvxuser.com/articles/colorspace/

Cinelerra loads the raw YV12 file, but the timeline does not show thumbnails. Not a show stopper, but a bit of a bummer. The one benefit of yuv4mpeg stream from mplayer is that you can feed that stream directly into mjpegtools for further processing.

Update 2008/11/22
It seems that 1080p in raw yuv format is actually unusable in my Fedora 9 Cinelerra install. When I load a project with a large raw yuv file, Cinelerra hangs for ten minutes as it loads the file. Once loaded, the GUI tends to freeze for another ten minutes. Thus, it seems that MJPEG may be my only choice for rendering 1080p.

I will continue to experiment.
end update

Here is the command I used to convert the video:
mplayer Video1.MOV -vo yuv4mpeg

I received this message while trying to convert the video
************************************************
**** Your system is too SLOW to play this! ****
************************************************
Possible reasons, problems, workarounds:
- Most common: broken/buggy _audio_ driver
- Try -ao sdl or use the OSS emulation of ALSA.
- Experiment with different values for -autosync, 30 is a good start.

So, I amended my command to add the autosync feature, which gradually adjusts the audio/video synchronization based on audio delay:
mplayer Video1.MOV -autosync 30 -vo yuv4mpeg

That seemed to alleviate the messages. Without redirecting to a specified filename, Mplayer outputs a file called "stream.yuv".

Information about the file looks like this in mplayer:
Playing stream.yuv. YUV4MPEG2 file format detected. YUV4MPEG2 Video stream 0 size: display: 1920x1080, codec: 1920x1080 VIDEO: [YV12] 1920x1080 12bpp 30.000 fps 0.0 kbps ( 0.0 kbyte/s) ========================================================================== Opening video decoder: [raw] RAW Uncompressed Video VDec: vo config request - 1920 x 1080 (preferred colorspace: Planar YV12) VDec: using Planar YV12 as output csp (no 0) Movie-Aspect is 1.78:1 - prescaling to correct movie aspect. VO: [xv] 1920x1080 => 1920x1080 Planar YV12 [ASPECT] Warning: No suitable new res found! [ASPECT] Warning: No suitable new res found! Selected video codec: [rawyv12] vfm: raw (RAW YV12) ==========================================================================

Quality
There seem to be quite a difference in the colors rendered by each format. The MJPEG-formatted file seems to much closer to the original than the raw YV12.

Applying Effects

For fun, I applied an oil painting effect to this five seconds of video. I then rendered the output to MJPEG format. I was disappointed to find out that five seconds of video with the oil paint effect applied takes a full three minutes and twenty seconds to render at 1080p in my dual, quad core Dell SC1430. Holy cow! I guess I won't be doing Star Wars green screens in 1080p!


In the interest of full disclosure, the oil paint effect is a notorious CPU hog and is not indicative of the type of work the average editor will perform. Another note: while rendering the output, all eight CPUs were maxed out at 95% utilization. Kowabunga!

To give people hope, applying a simple title to the image doesn't do too much damage to load on the CPU. The video with a title rendered to MJPEG format took twenty seconds.

Thoughts
My goal is to get some really fantastic looking video rendered from Cinelerra. In combination with the new Canon and some wise encoding choices, I believe I should be able to do this. This article shows that I'm getting closer, but I'm not there yet. I'm hoping to get closer with another round of experimentation.

One final thing..be ready to spend money on RAM. I noticed my paltry 2GB was immediately sucked up by Cinelerra when I was doing these various test videos. I'm going to get another 4GB and will let you know if that helps.

more to come.
The Mule

converting 1080p video, part I

I was interested in buying the new Canon EOS 5D Mark II, a really sweet bit of kit:
Amazon: Canon EOS 5D Mark II
Canon Product Page
Canon Learning Center
Rob Galbraith Review
Manual

I was excited at the possibility of stellar high quality video at a somewhat reasonable price. See Vincent Laforet's great demo video here:
http://blog.vincentlaforet.com/2008/10/10/without-further-ado-reverie/

So I downloaded test video output from a review of the preproduction cam here:
http://www.dpreview.com/previews/canoneos5dmarkII/page15.asp

Disclaimer
DPReview stated that these videos are direct from the cam, though according to Canon, the production model uses MPEG4 video compression. This does not jibe with my results, as I found by reviewing the output of mplayer and Quicktime on my MacBook that these test videos actually use H264 video compression. Strange. Of course, we'll see what the real deal is once the camera is available for purchase. Regardless, my testing procedures below maybe helpful for people using Linux and/or Cinelerra that need to work around similar problems.

Update 10/25/2008
I downloaded a couple longer clips from the review here:
http://www.dpreview.com/news/0809/08091705canon_5dmarkII.asp

My results were consistent with the original post. Interestingly, it took six minutes to convert one minute of 1080P video on my dual, quad core Dell SC1430. Oh Momma, 1080P is a resource hog!
End Update

Linux Media Player Results
Not so good. I was able to replay the video in mplayer fine, but vlc and avidemux2 hung half way, and xine played, though very choppily. Most of the above showed output errors of various kinds.

Next, I thought I'd see if its 1080P video output was usable at all in Cinelerra.

Cinelerra Results
Putting the original 1080P video into Cinelerra gave me a bunch of ugly errors, like so:
[h264 @ 0x2aaaaaf7e1b0]concealing 8160 DC, 8160 AC, 8160 MV errors [h264 @ 0x2aaaaaf7e1b0]AVC: Consumed only 161417 bytes instead of 161420 [h264 @ 0x2aaaaaf7e1b0]Unknown NAL code: 22

And the Cinelerra timeline and compositor windows weren't very promising either:


Convert!
So, I did what all good Cinelerra users do:
Convert the video to a usable format!

This command seems to produce very good output from a 1080P, H264 encoded video:
ffmpeg -i Video1.MOV -b 3000k -vcodec h264 -ab 256k -ar 44100 -acodec mpeg4aac -coder 1 -flags +loop -cmp +chroma -partitions +parti4x4+partp8x8+partb8x8 -me hex -subq 5 -me_range 16 -g 250 -keyint_min 25 -sc_threshold 40 -i_qfactor 0.71 test.mp4

Let's take the command options apart:
-i Video1.MOV: my input file
-b 3000k: bitrate of 3000K
-vcodec h264: use the h264 output video compression algorithm
-ab 256k: use an audio bitrate of 256kbps
-ar 44100: use an audio sampling rate of 44.1Khz
-acodec mpeg4aac: use the Apple Audio Compression codec

Next set of options related specifically to x264
-coder 1: enables CABAC entropy encoder used by x264 (better compression)
-flags +loop: enables loop filter
-cmp +chroma
-partitions +parti4x4+partp8x8+partb8x8: improves quality by looking at inter/intra partitions (i b p)
-me hex: set motion estimation method to hex (range-2 search of six surrounding points)
-me_range 16: max range of motion search (hex is limited to 16), higher useful on HD
-subq 5: good medium for higher speed encoding
-g 250: group of pictures size
-keyint_min 25: set minimum length between IDR frames (keyframes)
-sc_threshold 40: adjusts sensitivity of x264's scenecut detection
-i_qfactor 0.71: qscale (quality) difference between I-frames and P-frames

Finally, the output:
test.mp4: my output file

I don't know much about those switches. Here are two good pages explaining at least the words, if not their meaning, of these cryptic options:
FFmpeg x264 mapping
x264 (H264) options

Update 10/25/2008
However, I did see a number of these errors again while doing the conversion:
[h264 @ 0x3c2d71ab30]AVC: Consumed only 197265 bytes instead of 197268ts/s
[h264 @ 0x3c2d71ab30]AVC: nal size 0

Searching the web, it seems that they may have to do with an old version of ffmpeg:
http://lists.mplayerhq.hu/pipermail/ffmpeg-user/2007-October/012075.html

I am currently running the following: =============================================================================
Package Arch Version Repository Size
=============================================================================
Removing:
ffmpeg x86_64 0.4.9-0.37.20070503.lvn7 installed 527 k
ffmpeg-libs x86_64 0.4.9-0.37.20070503.lvn7 installed 3.8 M

And it's going to be a pain to compile with external ffmpeg libraries. I think I will hold off on that for now.
End Update

The original (Video1.MOV) and converted (test.mp4) files were marketedly different sizes:
-rw-r--r-- 1 root root 4214234 2008-10-23 23:15 test.mp4 -rw-r--r-- 1 root root 23690277 2008-10-23 22:55 Video1.MOV

As read by mplayer with -identify option, diffing the original and converted file's properties showed no differences. Here is what mplayer thinks both files :
[mov] Video stream found, -vid 0
VIDEO: [avc1] 1920x1080 24bpp 30.000 fps 0.0 kbps ( 0.0 kbyte/s)
ID_DEMUXER=mov
ID_VIDEO_FORMAT=avc1
ID_VIDEO_BITRATE=0
ID_VIDEO_WIDTH=1920
ID_VIDEO_HEIGHT=1080
ID_VIDEO_FPS=30.000
ID_VIDEO_ASPECT=0.0000
==========================================================================
Opening video decoder: [ffmpeg] FFmpeg's libavcodec codec family
Selected video codec: [ffh264] vfm: ffmpeg (FFmpeg H.264)
==========================================================================
ID_VIDEO_CODEC=ffh264
Audio: no sound
Starting playback...
VDec: vo config request - 1920 x 1080 (preferred colorspace: Planar YV12)
VDec: using Planar YV12 as output csp (no 0)
Movie-Aspect is undefined - no prescaling applied.
VO: [xv] 1920x1080 => 1920x1080 Planar YV12

This was odd, as mplayer spat out errors like this while playing the original:
[h264 @ 0xcfa780]AVC: Consumed only 197265 bytes instead of 197268 [h264 @ 0xcfa780]AVC: nal size 0

Once the file was converted, mplayer showed none of these errors.

Linux Media Player Results
I'm happy to say that the converted file replayed well in all the Linux media players, xine, mplayer and vlc, with the notable exception that playback in avidemux2 was slow.

In mplayer, the converted file did not show any errors, outside of a minor ASPECT warning error:
[ASPECT] Warning: No suitable new res found!

From a still, you can't really tell much of a difference..at least I couldn't. Note that the below images have been cropped for easier comparison, so they are not the full 1080P (1920x1280) resolution.

Original


Converted


The big test was bringing the converted file into Cinelerra. Since the OS level media players didn't have a problem, I was hopeful that Cinelerra wouldn't have any issues. Loading the file, I received no errors. However, playback was horrendous. I'd say that I got about 1fps in the compositor, though Preferences (Shift-P) showed that I was getting 32fps. Hmmm. Something definetly wrong here.

Final shot before I go to sleep
While in Cinelerra, I rendered the already converted 1080P content into one of the formats Cinelerra is happiest using, Quicktime using JPEG Photo compression. Rendering that video took no time at all (there was no audio track). Reloading the video back into Cinelerra, the file play more smoothly than the original conversion, but still had problems. I got 6fps. CPU on my dual, quad core was about 20% across all CPUs during playback (see below image). But memory use went through the roof! Swap was up too:



Woah! What's up with that? Well, looks like I have more testing to do. The possibility of gorgeous 1080P HD video has me slavering. Perhaps my eyes are too big for my stomach? As in everything Linux, this will take time.

to be continued..
the mule

Reference
http://bbs.archlinux.org/viewtopic.php?id=35836
thanks Skottish!

iTunes/iPod video workflow, scripted

I am going to revisit my conversation about workflow. Here's what we briefly discussed:
- the transformation of an idea (video) into reality and distributing it (in this case, using iTunes)

With today's internet, the steps involved are broad-based:
- idea creation, storyboard, distribution, production, archiving, marketing

Update 2008/11/22
I've added a few more details to my workflow in a new post here.
end update

I shall give you an example and how I reduced the amount of time spent creating and distributing my content.

When not working, my world is playing music with a band of itinerant musicians called the StormPigs. We gather together once every couple of months to play freeform music. No prior thought involved, just play. The joy of this is being together and having a good time. Otherwise, we are all busy professionals with full-time jobs and families. In remembrance of that good time, I produce videos of the event, distributed via iTunes and YouTube.

As time seems compressed these days, I want to spend as little time as possible behind the keyboard (though I am a technologist by trade). And as readers of this blog know, I am avid proponent of Linux. The beauty of Linux is that the system is completely configurable and flexible. But with this power comes a price. You have to invest the time to learn the shell, some bits of scripting and other Linux arcana. Consequently, it is daunting to the newcomer. But the benefits return to you many times over, as time you once spent on minutiae can now be spent thinking of new ideas for shows and creating new content, rather than simply focusing on the details of getting a file up to the server or copy and pasting content from one application to the other.

I am not a programmer of fanciful GUI front ends. I am a guy who just needs to get work done. So I try to solve my problems in the simplest way possible using the Linux programs and bash shell scripts to tie multiple programs together.

Here is the latest problem I needed to solve:
- how to I get my videos onto the web and in an iTunes ready form as quickly as possible?

In that light, I have come up with some scripts to help me on my way:
- the first encodes my editing video project into various formats (HDV, DVD, podcast)
- the second creates the list of songs from the video that will go into the podcast
- the third merges that songlist information into the podcast
- the fourth creates a new iTunes RSS feed (XML file) from the songlist information
- the fifth uploads the new podcast to my webserver
- the last is a wrapper that starts the other five

The encoding script is most useful and is the basis for the others. This script takes 720P video and 48Khz MPEG, Layer II stereo audio output rendered from a Cinelerra project and converts it to various formats: MPEG2 Program Stream, HDV, DVD and iTunes/iPod compatible formats. You will need the following programs installed for this script to work:
-mplex
-vlc
-ffmpeg

I've found the quality of each output file to be very good to excellent.

You can wrap some validation and input around this script, but the guts of the script look like this, where the arguments enclosed in curly braces will be replaced by the names of your input and output files:
#!/bin/bash
echo "Input: M2A audio and MPEG2 720P video streams"
echo "Output: program stream"
mplex -f 3 -b 2000 ${AUDIO} ${VIDEO} -o ${PS}

echo "Input: program stream"
echo "Output: MPEG2-TS, HDV"
vlc ${PS} --sout '#duplicate{dst=std{access=file,mux=ts,dst="'${HDV}'"}}' vlc:quit

echo "Input: MPEG2-TS, HDV"
echo "Output: MPEG2 DVD"
ffmpeg -i ${HDV} -target dvd -threads 8 ${DVD}

echo "Input: MPEG2 DVD"
echo "Output: iTunes/iPod compatible MP4"
ffmpeg -y -i ${DVD} -an -v 1 -threads 8 -vcodec h264 -b 250k -bt 175k -refs 1 -loop 1 -deblockalpha 0 -deblockbeta 0 -parti4x4 1 -partp8x8 1 -me full -subq 1 -me_range 21 -chroma 1 -slice 2 -bf 0 -level 30 -g 300 -keyint_min 30 -sc_threshold 40 -rc_eq 'blurCplx^(1-qComp)' -qcomp 0.7 -qmax 51 -qdiff 4 -i_qfactor 0.71428572 -maxrate 450k -bufsize 2M -cmp 1 -s 720x480 -f mp4 -pass 1 /dev/null

ffmpeg -y -i ${DVD} -v 1 -threads 8 -vcodec h264 -b 250k -bt 175k -refs 1 -loop 1 -deblockalpha 0 -deblockbeta 0 -parti4x4 1 -partp8x8 1 -me full -subq 6 -me_range 21 -chroma 1 -slice 2 -bf 0 -level 30 -g 300 -keyint_min 30 -sc_threshold 40 -rc_eq 'blurCplx^(1-qComp)' -qcomp 0.7 -qmax 51 -qdiff 4 -i_qfactor 0.71428572 -maxrate 450k -bufsize 2M -cmp 1 -s 720x480 -acodec aac -ab 160k -ar 48000 -ac 2 -f mp4 -pass 2 -threads 8 ${MP4}

Stay tuned for discussions of the remaining scripts to make your life easier,
CM