Cinelerra render compatibility on Fedora 10, x86-64

I spent the last four days building a new Fedora 10, x86-64 server for video editing based on programs available through the RPM Fusion repository. I'll post something soon about that whole process, which required more work and frustration than expected. However, my primary goal was to create a flexible and solid Cinelerra video editing rig.

Update 2009/02/15
Here's a post on how to build Cinelerra from source on Fedora 10, x86-64
*** end update ***

Testing the New System
In order to verify my new build, I needed to test high quality, rendered output from Cinelerra. More specifically, I wanted to test formats that required no external multiplexing (muxing). This will save me time in the future.

Assumptions
Also, note the test will be conducted against a larger resolution, 720P project. This will influence the playback ability of some of the programs.

Testing Procedure
During the testing phase, I was able to put Cinelerra's batch processing feature to use. This is very nice if you need to render large masses of files. Especially nice if you want to repeat or standardize that process across different Linux distributions.

Thinking more about this, it might be a boon to the cvs.cinelerra.org community if I made the XML of the batch list available to the community. That way, we could standardize the testing of output on different Linux distributions. At the end of the day, we could say things like "mplayer on SUSE 11 doesn't playback a 720P Quicktime rendered from Cinelerra that uses MPEG4 video compression with AAC audio encoding. However, that same file does work in mplayer on Fedora 10." So, it would be helpful to the QA process. I will bring this up as an idea to the community. Here is my latest effort at making the batch render feature part of the Cinelerra testing process:
http://crazedmuleproductions.blogspot.com/2010/01/batch-render-redux.html

Source File
The original video is 21 seconds long and is in 720P format, 1280x720 at 29.97fps.

In order to test the new operating system, I exported various combinations of audio and video formats, choosing fixed bitrates that seemed appropriate given the resolution of the source project:


I tested the output in standard Linux media players (vlc/mplayer/ffplay/totem), as well as reimported them into Cinelerra.

Results
In order to make it easy for people to comprehend the results, I've put together this graphic:

Using a stop light analogy, green means that the file plays with no problem. So, if you want to know the "best" or "good", compatible formats to use in Linux media players and Cinelerra, go for the green. Red means the file doesn't play at all. And yellow means that the file plays, but plays with some problem. For instance, the audio may be out of sync or stutters, or the video plays but maybe at the wrong resolution. Be advised that my system is fairly powerful, so your mileage may vary:
* Dell SC1430, dual quad core, 1.6Ghz, 2GB RAM
* RAID0 sys/working drive, RAID1 storage drive
* NVidia 8800GT video card

Observations
JPEG and MPEG-4 Quicktimes seem to work well and have the side benefit of being re-usable within Cinelerra. MPEG-4 rendering is unbelievably quick. H264 direct out of Cinelerra seems not to work at all in most cases. There was a tie for the file type with the widest compatibility among the chosen media players: Ogg/Vorbis and surprisingly, Microsoft MPEG4 using twos complement audio.

It is interesting to note the varying file sizes of the output from Cinelerra: Besides the uncompressed formats, Motion JPEG A creates the largest file sizes, while either of the MPEG4 formats are the most space efficient. Given the chart above, this is an important data point that video editors will be able to consider when selecting a particular format.

Conclusion
Though it is powerful, Cinelerra has been somewhat crippled by new users' ability to get usable content out of Cinelerra. I think this chart will improve that situation, even if it has only been proven out on Fedora 10, 64-bit. In that light, it would be interesting to see the same chart from an alternate Linux distribution. With my source Cinelerra and batch process EDL files, it should be easy to replicate the procedure on another system. I'd be happy to share them if an interested party wants to drop me a line.

Finally, I realize there are some missing file types like avi container, MPEG2 program and transport stream formats. The MPEG2 formates generally require external muxing and thus, were off the list for this test. However, I will try to test the at some point.

enjoy,
The Mule

MediaGate MG-350HD review

Happy New Year folks! I thought I'd take a break from the usual Cinelerra-HDV talk to discuss a new toy I got for Christmas: the MediaGate MG-350HD.


Since I am in a band and am its archivist, I have a lot of audio and video files that I've created over the years. So I wanted to get a device that would playback the high-definition videos and audio I've recorded on my home theater. The MediaGate MG-350HD seemed to fit the bill as an all-in-one media player: it plays video and audio files of almost any type (no Quicktimes, though) streamed from one or more network shares via wired or wireless ethernet connection. It also plays files from an attached USB drive or hard drive you supply and can act as a network storage device. In practice, the device works well enough with its bare bones setup and media selection/playback interfaces, but needs more setup and careful installation than the shiny brochure would have you believe.

I will tailor the following discussion to technically inclined individuals who will read the MGs' manual or search google for reviews and information that has already been discussed. You can get the manual and firmware upgrades online from http://www.airlinktek.com/ in the Support section. I will highlight the time consuming points for me or information about the device that is lacking in the online resources. It is best if one reads the manual first. Though the supplied manual has enough information to get the device setup and working, it is poorly written by a non-native english speaker.

For those who don't like reading manuals, my suggestion would be to simply hook it up to your TV using any of the provided composite, svideo, component or DVI connections, turn it on and press "setup" on the remote. If you don't have a picture on the TV immediately, select an alternate video mode by pressing the "TV out" button multiple times to cycle through the different options. With the remote, you control all the device configuration and media playback functions of the MediaGate.

After opening the box, I decided to configure the device in logical fashion by trying out the simpler functions and then advance to the more complicated setups:
- attach a USB drive and play files from it
* USB setup was easy and playback worked using my Corsair Voyager 2GB USB drive, though you cannot copy files to the USB drive.
- install a hard drive into the unit and play files from it
* Drive setup was also easy using an 80GB drive I had lying around and playback worked using the provided USB cable.
- connect it to my network via wired LAN connection
* Wired setup worked using DHCP after some fiddling.
- copy files to the internal hard drive using its network attached storage drivers
* works over a proprietary network protocol (NDAS) from Ximeta (http://www.ximeta.com).
* NDAS drivers only work on Windows 98/2000/XP, but there are experimental Linux drivers on their website that I haven't yet tried.
- connect it to my wireless network
* Wireless setup worked using DHCP and access point scan (ad hoc access point attachment is also available)
* laborious WEP key input using the remote
- stream files from my main server to the MediaGate
* tricky Windows share setup was the source of a couple hours of frustration
* streaming high def content only worked using internal disk or wired connection
* dvd and lower resolution content can be streamed using wireless

Speaking of playback, here are the file formats that the MG 350HD played for me:
- DVD format MPEGs (720x480 MPEG2s with 16 bit, 48Khz stereo or MPEG1 audio) and their individual, demuxed audio/video streams
- VOB files
- DIVX
- MPEG2 TS (transport stream) HD files (720p)
- AVIs
- PNGs/JPGs
- MP3s/WAVs/WMAs/OGGs
- MPEG4s created by iTunes

It doesn't play:
- Quicktime videos
- WMVs (I need to test this more)
- Ogg/Theora videos

This is not an exhaustive list, but at least will give you an idea of file compatibility. Playlists of videos and music can be created from any of the attached devices and played back in order or shuffled. The interface to sort through your media and create playlists is basic at best and no where near as slick or as easy as an iPod. Since the interface is where you will be spending quite a bit of time, it is also unfortunate to note that it is not customizable or skinnable. Firmware updates come once per quarter or less frequently, so there will be a long wait for any enhancements to the interface.

SMB shares from Windows 2000 were easily recognized. For XP, you need an account named "guest", "media" or "administrator" with a blank password. This obviously creates a security hole. Also, I needed to assign Read privileges to the XP Guest account for the directories I wanted to share.

The most pain I experienced in the share setup was getting the device to recognize the SMB share I created on my Fedora Linux box. Lots of issues with this one:
- proper user authentication
* you need to use either an account named "guest", "media" or "administrator" with a blank password. Security hole on the systems with the shares
- lowering SELinux lockdowns for SAMBA
* select all options under SAMBA in the SELinux config utility
- appropriate file privileges
* chmod 755 on any files you want to play
- change the default SMB security level
* use a security level of "share" in the /etc/samba/smb.conf file

I had no trouble streaming high definition content from a network share using a wired connection. Using wireless I was only able to stream DVD resolution files (including MPGs and DIVx) as well as any music files without dropouts. Over wireless, strong signal strength is a must for playing back DVD resolution files without dropouts. If you're not playing back any files, the "info" button on the remote will show you the wireless signal strength. While playing back files, "info" will show you the specifications of the file being played back. Many of the buttons on the remote serve double duty depending on if you are in setup versus playback mode.

In sum, the MediaGate MG-350HD met my expectations and can playback an impressive array of video and audio files. The interface needs a -face lift, but the unit does a yeoman's job of serving up content. Definetly, you will need a wired connection or copy any high def media you have to an internal hard drive in order to view those on your HDTV via the MediaGate. Nice product.

cam compatible HDV edit chain, part II (short)

I posted a more detailed version of this two posts ago (http://crazedmuleproductions.blogspot.com/2006_10_01_archive.html), but I thought I'd reiterate it in a shorter format:
1) from Cinelerra, export video using YUV4MPEG using mpeg2enc (.m2v) compression scheme with special params:
mpeg2enc --verbose 0 --aspect 3 --format 3 --frame-rate 4 --video-bitrate 18300 --nonvideo-bitrate 384 --interlace-mode 0 --force-b-b-p --video-buffer 448 --video-norm n --keep-hf --no-constraints --sequence-header-every-gop --min-gop-size 6 --max-gop-size 6 -o %
2) from Cinelerra, export audio using mpeg audio, layer II (384Kbps)
3) multiplex audio and video streams to MPEGPS (Program Stream) format using mplex:
mplex -f 3 -b 2000 video.m2v audio.m2a -o output.ps
4) convert .ps to .ts, MPEGTS (Transport Stream) format using VLC
5) export to cam, consider using 10sec blank leader

Update 5/17/2009
Note that if you bring the video into Cinelerra and the thumbnail is green, this means that the first frame of the video is not a keyframe. This is a problem with VLC. You can avoid this problem by using this ffmpeg command in place of Step 4 above:
ffmpeg -y -i OUTPUT.ps -acodec copy -f mpegts -qscale 1 OUTPUT.m2t
*** end update ***

Update 3/14/2008
Once the video is converted, you may then output to the camera using test-mpeg2 from libiec61883:
/2006/10/success-and-failure-in-land-of-dvhdv.html

Also, take note of a possible 2GB limit in libiec61883 uploading your content to the cam:
/2006/10/libiec61883-now-exports-files-greater.html
*** end update ***

Update 11/17/06
Last night, I found that doing performing other operations on your system like surfing the web or playing audio while running mplex or VLC convert may cause problems with the video as it uploads to the camera. So don't do anything while you convert your videos.

In my case, I was 28 minutes into an hour long video uploading to the cam when the import stopped. I no longer had uploading video displayed on the LCD screen of the cam. I only saw blue, the upload seemed to continue, but I got no final output. The solution was to remux the original streams using mplex and rebuild the MPEGTS using VLC. The important point being that one of these two processes is very sensitive to fluctuations in CPU or hard drive performance. So leave your PC alone while these processes are active.
*** end update ***

That's all she wrote for today. Enjoy!
the mule

rendering cam-compatible HDV MPEGTS files

I have been successful in exporting HDV back out to my cam. I haven't posted detailed output from the individual steps before, so here it is:

General Steps
1) export video from Cinelerra as mpeg2 using mpeg2enc
2) export audio from Cinelerra as mp3
3) mux in mplex
4) open MPEGPS in VLC and convert to TS.

Detail
1) export video from Cinelerra as mpeg2 using mpeg2enc:
mpeg2enc --verbose 0 --aspect 3 --format 3 --frame-rate 4 --video-bitrate 18300 --nonvideo-bitrate 384 --interlace-mode 0 --force-b-b-p --video-buffer 448 --video-norm n --keep-hf --no-constraints --sequence-header-every-gop --min-gop-size 6 --max-gop-size 6 -o /mnt/videos/20050721/20050721.m2v

2) export audio from Cinelerra as mp3:
--------------------------------------------
Input File : 'stdin' 48.0 kHz
Output File: '/mnt/videos/20050721/test.mp3'
384 kbps MPEG-1 Layer II j-stereo Psy model 1
[De-emph:Off Copyright:No Original:No CRC:Off]
[Padding:Normal Byte-swap:Off Chanswap:Off DAB:Off]
ATH adjustment 0.000000
--------------------------------------------
encode_init: using tablenum 0 with sblimit 27
Hit end of audio data
Avg slots/frame = 1152.000; b/smp = 8.00; bitrate = 384.000 kbps

3) mux in mplex
[root@computer 20050721]# mplex -f 3 -b 2000 -o test.ps test.mp3 20050721.m2v
INFO: [mplex] mplex version 1.8.0 (2.2.4 $Date: 2005/08/28 17:50:54 $)
INFO: [mplex] File test.mp3 looks like an MPEG Audio stream.
INFO: [mplex] File 20050721.m2v looks like an MPEG Video stream.
INFO: [mplex] Found 1 audio streams and 1 video streams
INFO: [mplex] Selecting generic MPEG2 output profile
INFO: [mplex] Multiplexing video program stream!
INFO: [mplex] Scanning for header info: Audio stream c0 (test.mp3)
INFO: [mplex] MPEG AUDIO STREAM: c0
INFO: [mplex] Audio version : 1.0
INFO: [mplex] Layer : 2
INFO: [mplex] CRC checksums : no
INFO: [mplex] Bit rate : 49152 bytes/sec (384 kbit/sec)
INFO: [mplex] Frequency : 48000 Hz
INFO: [mplex] Mode : 0 stereo
INFO: [mplex] Mode extension : 0
INFO: [mplex] Copyright bit : 0 no copyright
INFO: [mplex] Original/Copy : 0 copy
INFO: [mplex] Emphasis : 0 none
INFO: [mplex] Scanning for header info: Video stream e0 (20050721.m2v)
INFO: [mplex] VIDEO STREAM: e0
INFO: [mplex] Frame width : 1280
INFO: [mplex] Frame height : 720
INFO: [mplex] Aspect ratio : 16:9 display
INFO: [mplex] Picture rate : 29.970 frames/sec
INFO: [mplex] Bit rate : 18300000 bits/sec
INFO: [mplex] Vbv buffer size : 229376 bytes
INFO: [mplex] CSPF : 0
INFO: [mplex] SYSTEMS/PROGRAM stream:
INFO: [mplex] rough-guess multiplexed stream data rate : 19077648
INFO: [mplex] Setting best-guess data rate.
INFO: [mplex] Run-in Sectors = 752 Video delay = 58123 Audio delay = 61126
INFO: [mplex] New sequence commences...
INFO: [mplex] Audio c0: buf= 0 frame=000000 sector=00000000
INFO: [mplex] Video e0: buf= 0 frame=000000 sector=00000000
INFO: [mplex] Scanned to end AU 39469
INFO: [mplex] STREAM e0 completed @ frame 39469.
INFO: [mplex] STREAM c0 completed @ frame 54874.
INFO: [mplex] Multiplex completion at SCR=118584775.
INFO: [mplex] Audio c0: buf= 2304 frame=054874 sector=00031342
INFO: [mplex] Video e0: buf= 0 frame=054843 sector=00509538
INFO: [mplex] AUDIO_STATISTICS: c0
INFO: [mplex] Audio stream length 63216000 bytes.
INFO: [mplex] Syncwords : 54875
INFO: [mplex] Frames : 54875 padded
INFO: [mplex] Frames : 0 unpadded
INFO: [mplex] BUFFERING min 18 Buf max 1169
INFO: [mplex] VIDEO_STATISTICS: e0
INFO: [mplex] Video Stream length: 1031417371 bytes
INFO: [mplex] Sequence headers: 6579
INFO: [mplex] Sequence ends : 1
INFO: [mplex] No. Pictures : 39470
INFO: [mplex] No. Groups : 6579
INFO: [mplex] No. I Frames : 6579 avg. size 58215 bytes
INFO: [mplex] No. P Frames : 32891 avg. size 19714 bytes
INFO: [mplex] No. B Frames : 0 avg. size 0 bytes
INFO: [mplex] Average bit-rate : 6264800 bits/sec
INFO: [mplex] Peak bit-rate : 9203600 bits/sec
INFO: [mplex] BUFFERING min 943996 Buf max 1944874
INFO: [mplex] MUX STATUS: no under-runs detected.
[root@computer 20050721]#


4) open MPEGPS in VLC and convert to MPEGTS.

Update 3/14/2008
Once the video is converted, you may then output to the camera using test-mpeg2 from libiec61883:
/2006/10/success-and-failure-in-land-of-dvhdv.html

Also, take note of a possible 2GB limit in libiec61883 uploading your content to the cam:
/2006/10/libiec61883-now-exports-files-greater.html

HDV MPEG2 transport stream file sizes/render rates

The data rate of the 720P MPEG2-TS files output from my cam is about 108.95MB/min or 1.82MB/s. Here is a table of video length-to-size conversions.

duration size

12m 1.32GB
15m 1.65GB
18m35s 2.07GB
19m12s 2.12GB
20m01s 2.18GB
34m 3.70GB
Exporting 720P HDV from Cinelerra takes two processes:
1) render the video
2) render the audio

Here are some rendering times using mpeg2enc and mpeg layer 2 audio compression:

duration mpeg2enc render rate
63m 310m 4.92min per min of video

duration mp2 render rate
63m 6m 0.09min per min of audio

Mplex takes about 7 minutes to mux about an hour of audio and video.

hard work is paying off / HDV workflow

OK! I've got the latest video up on iTunes and now I just need to edit the XML for it. Editing the damn XML always takes too long. And of course, I want to get the track list right with the correct track times and witty comments. Ufff. OK! So the track list is done and now to send the boys the email for the latest video! Hurrah!

More for my benefit than anyone elses, here is my HDV workflow using Cinelerra:
1) import video
2) edit video
3) for multiple segments, make sure you set "align to frames" and audio and video segments end at the exact same time.
4) for HD output, render using mpeg2enc settings
5) for DVD output, render using ffmpeg -target DVD from Cinelerra, something like this:
ffmpeg -f yuv4mpegpipe -i - -y -i /mnt/videos/20050721/20050721.wav -target dvd %
6) for iPod output, render using ffmpeg -target DVD, then output to mpeg4 video/aac audio mov container:
ffmpeg -i inputdvd.mpg -f mov -vcodec mpeg4 -qscale 7 -s 320x180 -r 29.97 -aspect 16:9 -acodec aac -ac 2 -ab 128 output.mov
7) import .mov to XP iTunes
8) upload .mov to website
9) edit RSS XML
10) final test in iTunes

Quicktime for Linux compatibility chart

I spent Sunday afternoon and evening creating this compatibility chart for QT for Linux formatted files exported from Cinelerra. There are many compression schemes that can be used within this file format and I did not have a list to tell me what scheme works and what doesn't in Cinelerra and some default media players like Mplayer, Quicktime for Win and Windows Media Player 10. Hence the guide.

Enjoy!
html format
open office calc format