joining, concatenating video files

I was trying to combine four MP4's and had a bit of a learning experience, so I thought I'd share.

Here are the four files I was trying to combine, a total of roughly 1.5GB:
[sodo@computer tmp]$ ll part?.mp4
-rw-------. 1 sodo sodo 282308395 Feb 29 19:19 part1.mp4
-rw-------. 1 sodo sodo 547860894 Feb 29 19:39 part2.mp4
-rw-------. 1 sodo sodo 429237647 Mar  3 13:54 part3.mp4
-rw-------. 1 sodo sodo 428161483 Feb 29 20:12 part4.mp4

The Few, The Proud
After reading some FFmpeg doc (linked in the references), I found out that MP4's can't be concatenated in their native form.  There are only a few video formats that CAN be concatenated in their native form: MPEG-1, MPEG-2 PS, DV.  After some experimentation and viewing the results, I decided to use mpeg2video as my intermediate format with this command:
ffmpeg -threads 8 -i part1.mp4 -sameq -vcodec mpeg2video -acodec mp2 -ac 2 -ar 44100 -ab 256k 1.mpg

Converting en masse
Since I had four of these files to convert, it made sense for me to use some quick shell control flow to get the job done in one shot (input/output streams in bold below):
for i in 1 2 3 4;do ffmpeg -threads 8 -i part$i.mp4 -sameq -vcodec mpeg2video -acodec mp2 -ac 2 -ar 44100 -ab 256k $i.mpg;done

When running, the output looks like this:
[sodo@computer tmp]$ for i in 1 2 3 4;do ffmpeg -threads 8 -i part$i.mp4 -sameq -vcodec mpeg2video -acodec mp2 -ac 2 -ar 44100 -ab 256k $i.mpg;done
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'part1.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isommp42
    creation_time   : 2012-02-18 19:44:00
  Duration: 00:12:08.29, start: 0.000000, bitrate: 3101 kb/s
    Stream #0.0(und): Video: h264 (High), yuv420p, 1280x720, 2946 kb/s, 29.97 fps, 29.97 tbr, 60k tbn, 59.94 tbc
    Metadata:
      creation_time   : 1970-01-01 00:00:00
    Stream #0.1(und): Audio: aac, 44100 Hz, stereo, s16, 151 kb/s
    Metadata:
      creation_time   : 2012-02-18 19:44:26
[buffer @ 0x2329fe0] w:1280 h:720 pixfmt:yuv420p tb:1/1000000 sar:0/1 sws_param:
[mpeg @ 0x2322280] VBV buffer size not set, muxing may fail
Output #0, mpeg, to '1.mpg':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isommp42
    creation_time   : 2012-02-18 19:44:00
    encoder         : Lavf52.111.0
    Stream #0.0(und): Video: mpeg2video, yuv420p, 1280x720, q=2-31, 200 kb/s, 90k tbn, 29.97 tbc
    Metadata:
      creation_time   : 1970-01-01 00:00:00
    Stream #0.1(und): Audio: mp2, 44100 Hz, stereo, s16, 256 kb/s
    Metadata:
      creation_time   : 2012-02-18 19:44:26
Stream mapping:
  Stream #0.0 -> #0.0
  Stream #0.1 -> #0.1

frame=21827 fps=117 q=0.0 Lsize= 1385478kB time=00:12:08.26 bitrate=15584.8kbits/s  
video:1357365kB audio:22759kB global headers:0kB muxing overhead 0.387953%


The intermediate blues
The file size of the intermediates ballooned, expectedly:
[sodo@computer tmp]$ ll ?.mpg
-rw-rw-r--. 1 sodo sodo 1418729472 Mar  3 12:59 1.mpg
-rw-rw-r--. 1 sodo sodo 2290866176 Mar  3 13:05 2.mpg
-rw-rw-r--. 1 sodo sodo 2143287296 Mar  3 13:06 3.mpg
-rw-rw-r--. 1 sodo sodo 1918547968 Mar  3 13:11 4.mpg

Simple filesystem concatenation
After I converted the base files into files of a type that could be concatenated, I used this command to concatenate the files:
cat 1.mpg 2.mpg 3.mpg 4.mpg > all.mpg

The resulting file is pretty large ~6GB:
[sodo@computer tmp]$ ll all.mpg 

-rw-rw-r--. 1 sodo sodo 6165041152 Mar  3 13:16 all.mpg

This would be a quite serviceable intermediate file, but upon reflection, I thought that I could probably do the concatenation and conversion to a final format in one step, rather than two.  FFmpeg to the rescue!

FFmpeg's "concat" feature

Because I like my iDevices, I want the file format to be an MP4 container using H264/AAC as my video and audio formats.  So instead of doing a two-step conversion:

1) concatenate the files in the filesystem

2) transcode the video to a file format

I decided to combine both of those steps into one by using the "concat" feature of ffmpeg.  The command looks like this:

ffmpeg -i concat:"/tmp/1.mpg|/tmp/2.mpg|/tmp/3.mpg|/tmp/4.mpg" -acodec libfaac -aq 100 -ac 2 -vcodec libx264 -vpre slow -crf 24 output.mp4

You can change the output specifiers to taste.  Also note that you'll need a presets file defined if you are going to use the "-vpre slow" specifier.

There was no way to tell from FFmpeg's output that the concatenation command was working except for this small line:
Input #0, mpeg, from 'concat:/tmp/1.mpg|/tmp/2.mpg|/tmp/3.mpg|/tmp/4.mpg'

as well as the fact that the "frame=" counter at the bottom of the FFmpeg output incremented beyond the length of the first video.  In this case, 21827.

Verify the total number of frames
Redirecting standard error to standard output
As a sanity check, I like to verify the total number of frames in the output.  I can do this by capturing the text information that FFmpeg outputs when FFmpeg runs.  That text information is not "standard output" in the Unix sense.  The text from an FFmpeg command actually outputs "standard error".  So instead of trying to grep with a command like this:
ffmpeg -i 1.mpg -an -vcodec copy -f mpeg2video -y NUL  | grep 'frame'

You actually need a command like this:
ffmpeg -i 1.mpg -an -vcodec copy -f mpeg2video -y NUL 2>&1 | grep 'frame'

Or save the output to a file and then grep:
ffmpeg -i 1.mpg -an -vcodec copy -f mpeg2video -y NUL 2>&1 | tee test.txt ; grep 'frame' test.txt

The 2>&1 redirects (the ">") standard error (the "2") to standard output (the "1").  For some reason, we need an ampersand in there for the command to work correctly.

Unfortunately, I encountered a nasty little problem when I tried to read the output from that file, explained here:
http://www.techanswerguy.com/2012/03/redirecting-ffmpeg-output-performing.html

If you don't want to worry about the details, here is the solution I used to grab the number of frames in the output:
ffmpeg -i 1.mpg -an -vcodec copy -f mpeg2video -y NUL 2>&1 | awk -F"\015" 'NF > 1 {lf=NF-1;print $lf}' | awk '{print $1}' | awk -F= '{print $2}'

Until then..there you have it!  Appending multiple video files together using FFmpeg's concat feature. Sweet!
*da mule*

References
http://www.ffmpeg.org/faq.html#toc-How-can-I-join-video-files_003f
http://ffmpeg.org/faq.html