extundelete to the rescue!

So..in usual fashion, I had a minor foul up today. I was overzealous in my desire to reclaim disk space and zapped some primary source vids off of my ext4 partition. Ooops. Thankfully, I found extundelete, a program that scans ext3 and ext4 filesystem journals and recovers files from those journals:
http://extundelete.sourceforge.net/

Process
Here's what I did to recover the files.
1) In order to prevent the deleted files being overwritten, I immediately stopped whatever work I was doing and unmounted the drive that the files were on. In my case:
umount /dev/mapper/vg_ogre-lv_root

2) Well, that's my root drive. So before rebooting to my Fedora 12 Live CD, I checked that Fedora had the extundelete program in its repository. I was in luck!
[root@localhost ~]# yum install extundelete
Loaded plugins: presto, refresh-packagekit
fedora/metalink

20 kB 00:00
fedora 4.2 kB 00:00
fedora/primary_db 9.7 MB 00:10
updates/metalink 16 kB 00:00
updates 4.5 kB 00:00
updates/primary_db 4.3 MB 00:01
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package extundelete.i686 0:0.1.8-2.fc12 set to be updated
--> Finished Dependency Resolution

You can also compile from source at the link above.

3) Since extundelete requires you to access an unmounted filesystem, I rebooted to Fedora 12 Live CD

4) I installed extundelete from the Fedora Updates repository
Transaction Summary

Install 1 Package(s)
Upgrade 0 Package(s)

Total download size: 52 k
Is this ok [y/N]: y
Downloading Packages:
Setting up and reading Presto delta metadata
updates/prestodelta 4.4 kB 00:00
Processing delta metadata
Package(s) data still to download: 52 k
extundelete-0.1.8-2.fc12.i686.rpm 52 kB 00:00
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : extundelete-0.1.8-2.fc12.i686 1/1

Installed:
extundelete.i686 0:0.1.8-2.fc12
Complete!

5) In order to restore deleted files, you must have a partition mounted that has enough space for the recovered files. I have a second partition (/mnt/backups) that I use to backup my main root partition. So while running under the Live CD, I created a destination directory and mounted /mnt/backups:
[root@localhost /]# mkdir /mnt/backup[root@localhost /]# mount -t ext3 /dev/mapper/vg_ogre-lv_backup /mnt/backup

6) Though you can specify that extundelete undeletes individual files or all files on a filesystem, I ran extundelete with the proper command switches to undelete an entire directory of files.  Note that you should be in the directory that has plenty of space for the restore, as extundelete defaults to restoring what it finds to the present working directory:
[root@localhost backup]# extundelete /dev/mapper/vg_ogre-lv_root --restore-directory "mnt/videos/stormpigs/20100408"

WARNING: Extended attributes are not restored.

Loading group metadata ... 27025 groups loaded.

Loading journal descriptors ... 30695 descriptors loaded.

Writing output to directory RECOVERED_FILES/

Searching for recoverable inodes in directory mnt/videos/stormpigs/20100408 ...
309 recoverable inodes found.

Looking through the directory structure for deleted files ...Restored inode 2495842 to file RECOVERED_FILES/mnt/videos/stormpigs/20100408/20100408_2.m2t

Restored inode 2495844 to file RECOVERED_FILES/mnt/videos/stormpigs/20100408/20100408_3.m2t

Restored inode 57147521 to file RECOVERED_FILES/mnt/videos/stormpigs/20100408/vadd.txt

Restored inode 57147522 to file RECOVERED_FILES/mnt/videos/stormpigs/20100408/aadd.txt

Restored inode 57147525 to file RECOVERED_FILES/mnt/videos/stormpigs/20100408/temp.txt

Restored inode 57147528 to file RECOVERED_FILES/mnt/videos/stormpigs/20100408/test.xml

Restored inode 57147532 to file RECOVERED_FILES/mnt/videos/stormpigs/20100408/testLabels.sh

Success..or close enough to it
This is good..I've undeleted all but one of the files I had deleted. So, it looks like before I unmounted my root partition, the inode for "20100408_1.m2t" was overwritten. Oh well, two out of three ain't bad.

Update 2011/10/05
This works nicely with ext4 partitions as well.
*** end update ***

Anyway, this short post should give you some comfort that extundelete actually does what it is supposed to. Thanks number9652!!

Reference
http://extundelete.sourceforge.net/
http://sourceforge.net/mailarchive/forum.php?forum_name=extundelete-users

Cinelerra 4 just not working

I was interested in trying out Cinelerra 4, Heroine Warrior's latest version of Cinelerra. To reiterate, there are two versions of Cinelerra available:
Heroine Warrior's, the original coded by Adam Williams (bow down to the man!) and
Cinelerra Community Version, Cinelerra CV

On my Fedora 10, x86-64 setup, I gave the latest HV Cinelerra 4 version a try. I encountered a bunch of hurdles, mainly the GCC 4.3 changes that caused missing header information in mjpegtools-1.9.0_rc3 listed here:
http://bugs.gentoo.org/show_bug.cgi?id=200767

I got the software to compile. At the final "make install" step, the installation starts, but does not complete with only the first 40 or so lines of the install finishing:
[mule@ogre bin]# make install
make -f build/Makefile.cinelerra install
make[1]: Entering directory `/usr/src/cinelerra-4'
make -C plugins install
make[2]: Entering directory `/usr/src/cinelerra-4/plugins'
mkdir -p ../bin/fonts
cp fonts/* ../bin/fonts
mkdir -p ../bin/shapes
cp shapes/* ../bin/shapes
cp ../thirdparty/mjpegtools*/mpeg2enc/mpeg2enc ../bin/mpeg2enc.plugin
make[2]: Leaving directory `/usr/src/cinelerra-4/plugins'
DST=../bin make -C libmpeg3 install
make[2]: Entering directory `/usr/src/cinelerra-4/libmpeg3'
cp x86_64/mpeg3dump x86_64/mpeg3peek x86_64/mpeg3toc
x86_64/mpeg3cat ../bin
make[2]: Leaving directory `/usr/src/cinelerra-4/libmpeg3'
make -C po install
make[2]: Entering directory `/usr/src/cinelerra-4/po'
mkdir -p ../bin/locale/de/LC_MESSAGES
cp sl.mo ../bin/locale/sl/LC_MESSAGES/cinelerra.mo
make[2]: Leaving directory `/usr/src/cinelerra-4/po'
make -C doc install
make[2]: Entering directory `/usr/src/cinelerra-4/doc'
mkdir -p ../bin/doc
cp arrow.png autokeyframe.png camera.png channel.png crop.png cut.png
expandpatch_checked.png eyedrop.png fitautos.png ibeam.png
left_justify.png magnify.png mask.png mutepatch_up.png paste.png
projector.png protect.png record.png recordpatch_up.png rewind.png
singleframe.png show_meters.png titlesafe.png toolwindow.png
top_justify.png wrench.png magnify.png ../bin/doc
cp: warning: source file `magnify.png' specified more than once
cp cinelerra.html ../bin/doc
make[2]: Leaving directory `/usr/src/cinelerra-4/doc'
cp COPYING README bin
make[1]: Leaving directory `/usr/src/cinelerra-4'
[mule@ogre bin]#

Therefore, the installation does not copy the cinelerra binary into /usr/local/bin. If I try to run the binary from the source code directory, I get this:
PluginServer::open_plugin: /usr/src/cinelerra-4/bin/brightness.plugin:
undefined symbol: glUseProgram
PluginServer::open_plugin: /usr/src/cinelerra-4/bin/deinterlace.plugin:
undefined symbol: glUseProgram
undefined symbol: glNormal3f
PluginServer::open_plugin: /usr/src/cinelerra-4/bin/swapchannels.plugin:
undefined symbol: glUseProgram
PluginServer::open_plugin: /usr/src/cinelerra-4/bin/threshold.plugin:
undefined symbol: glUseProgram
PluginServer::open_plugin: /usr/src/cinelerra-4/bin/zoomblur.plugin:
undefined symbol: glEnd
signal_entry: got SIGSEGV my pid=3965 execution table size=16:
awindowgui.C: create_objects: 433
awindowgui.C: create_objects: 440
awindowgui.C: create_objects: 444
awindowgui.C: create_objects: 447
awindowgui.C: create_objects: 453
suv.C: get_cwindow_sizes: 744
suv.C: get_cwindow_sizes: 774
suv.C: get_cwindow_sizes: 800
suv.C: get_cwindow_sizes: 821
editpanel.C: create_buttons: 177
editpanel.C: create_buttons: 303
editpanel.C: create_buttons: 177
editpanel.C: create_buttons: 303
mwindowgui.C: create_objects: 192
mwindowgui.C: create_objects: 195
mwindowgui.C: create_objects: 199
signal_entry: lock table size=6
0x357dc80 RemoveThread::input_lock RemoveThread::run
0x64d0370 CWindowTool::input_lock CWindowTool::run
0x64f3640 TransportQue::output_lock PlaybackEngine::run
0x3441940 TransportQue::output_lock PlaybackEngine::run
0x3442420 MainIndexes::input_lock MainIndexes::run 1
0x3442f80 Cinelerra: Program MWindow::init_gui *
BC_Signals::dump_buffers: buffer table size=0
BC_Signals::delete_temps: deleting 0 temp files
SigHandler::signal_handler total files=0


Even though I have an NVidia graphics card, the error lines above were related to OpenGL. Thus, I thought I might have better luck compiling without OpenGL enabled. After I removed those lines from from hvirtual_config.h, I did a make clean;make. This time around, I was able to get Cinelerra 4 to start properly. Though, it soon locks up when viewing my 720P MPEG-TS files:
[mule@ogre cinelerra-4]$ ./bin/cinelerra
Cinelerra 4 (C)2008 Adam Williams

Cinelerra is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. There is absolutely no warranty for Cinelerra.
[mpeg2video @ 0xeafd00]slice mismatch
[mpeg2video @ 0xeafd00]mb incr damaged
[mpeg2video @ 0xeafd00]mb incr damaged
[mpeg2video @ 0xeafd00]invalid cbp at 14 37
[mpeg2video @ 0xeafd00]slice mismatch
[mpeg2video @ 0xeafd00]ac-tex damaged at 25 40
[mpeg2video @ 0xeafd00]ac-tex damaged at 6 41
[mpeg2video @ 0xeafd00]ac-tex damaged at 7 42

So much for that experiment! I'm going back to the CV version for now.
the mule

outstanding bugs/foul ups/frustration

There were a number of outstanding bugs that some of the cvs.cinelerra.org programmers addressed. One of them was that Cinelerra keeps hanging when I open up projects. It seems to happen on projects I've built before 2.1 that I've saved as 2.1. So I've told the programmers I will test their work once they apply the patches. I drank too much coffee in the afternoon and deleted a number of files that I wish I hadn't off of my FAT partition. Stupid ass. Luckily, I have those on a second drive, so I should be OK. Another thing to do: MAKE DVD BACKUPs OF PARTITION IMAGES, PRONTO!! Ugh. What a hassle. This new rig is NOT making my life any easier!!

Tonight, I finally got to edit some video..the last band video. Amazing! It has been about a month and a half since I started the whole OpenGL/export HDV to tape excursion and while I've learned a lot, it has just been frustrating as hell. The state of HDV on Linux is pretty rough. I'm lucky I've made it this far and I wouldn't have without the help of the community, both cvs.cinelerra.org and linux1394.org. I'd like to finish this edit job so that I can spend some time with my girlfriend before she leaves me. As I'm editing this video, I'm a little upset by the quality: the crappy automatic gain control on the cam makes dark shots look grainy and grey and the focus fades in and out. How good of a video production will I make if the camera work sucks? Yarg. At least I've finished the edits. Off to render!

rebuilding the workstation: lost my damn RAID set!

I *almost* had a minor tragedy last night while rebuilding my Fedora Core 4 box. The *almost* tragedy occurred while reinstalling Core 4. While going through the process, I told the installer to use my existing RAID set, but do not reformat it. I figured what harm could come to the drive if it doesn't get formatted? Well, a lot, apparently. Since I was rushing through the install, I neglected to take proper care of the 40GB of MPEG2 files on the RAID set. Here's what happened.

After the FC installer finished, it asks to reboot the box, so I rebooted. On bootup, the system gave me errors regarding an unrecognized filesystem and dropped me to a filesystem shell to fix what was wrong. I didn't know what was wrong, so I rebooted into a linux rescue disk and simply took the RAID filesystem out of /etc/fstab and rebooted.

On the second bootup, I started fdisk to look at the drives. To my dismay, fdisk did not recognize the RAID set partitions. Agh! I did some preliminary research online. I came to the determination that I figured I had lost my data, though I did have it backed up. But it would be a pain to retrieve the 40GB or so of videos from my backup system. Damn it. So I bit the bullet and created new partitions of type "fd", a Linux RAID autodetect partition as I had done when I setup the RAID set initially. When I wrote the partition table, fdisk gave me some error saying that the drives will resync after the box reboots. I rebooted and looked at the output of "fdisk -l". Things seemed alright, as the drives were recognized as Linux RAID autodetect. So I then reenabled /dev/md0 in /etc/fstab and made sure that /etc/mdadm.conf was correct. Dejectedly I rebooted.

When the system started, it dropped me into a prompt complaining of filesystem problems again. Now what? This time, I looked at the man pages for fsck and figured out the commands I needed to repair the disk. They ran something like this:
fsck -t ext2 /dev/md0 -V -r
-t filesystem type
-V verbose
-r prompt for each repair

A couple of inodes were missing or corrupt. OK. Fsck seemed to continue on with the different stages of the five stage check procedure. About five or ten minutes of this, I was getting worried. Happily, it finished the checks and dropped me to a prompt in order to exit and reboot. OK..that's progress! Also, the box came up clean with no errors. Sweet. I now went to view the filesystem, and to my shock and surprise, my original files were there! Awesome!! But now the real test is reading from and writing to files. I first viewed one of the videos in mplayer. This worked! I then performed an extensive write test. Cinelerra needs table of contents files for each video. These are index file, essentially. So at a prompt, I generated a bunch of toc files for the 30GB or so of video I had by using this one command:
for i in `ls -1 *.m2t` ; do echo $i ; mpeg3toc $i $i.toc ; done

The file creation took about twenty minutes but worked! I then loaded the files into Cinelerra, started editing and wouldn't you know they are good to go! Hooray! But I'm still an idiot.

Moral of the story is make sure you do your research on RAID before you decide to implement it.

system reconfig, #2

Woops with Ghost
OK. A minor foul up in the steps from the last post. Seems that Ghosting the utility partition is not working as expected. Norton Ghost created a total new partition at the end of the XP partition on the 80 gigger. So, I went ahead and deleted it, as this 2GB partition eats into my Linux root partition. I never used the utility partition anyway, but I thought that Ghost would have correctly selected the 100MB partition I had chosen. Oh well. I'm not that concerned.

FDISK is for F*@K!
Another foul up was trying to use a Windows 98 boot disk and the classic fdisk program to both create the XP/Linux partitions and mark one as active. Word to the wise: DON'T USE FDISK! It has an annoying habit of "Verifying Drive Integrity" that takes FOREVER. So, I just decided to reboot and use my old 40GB XP system to carve up the drive and mark the 30GB partition as active. Saved a good deal of time there with that brainstorm. Phffflt!

Going SLEOOWLY.
Applying the XP image to the partition designated for it went super smooth, but the Dell has this ugly habit of taking forever to start up when I have the power to some disks plugged in and some not, but still connected to my Sil680 IDE stripe set RAID card. Frustrating, as it adds two minutes at the front end of every boot cycle. Argh. But now I have my XP system back up and running on my new 80GB drive! Hooray!

Core 4 from Scratch..with a bad CD!
I've decided that installing Fedora Core 4 from disks will make life cleaner. I could apply a partimage of my FC3 build or take the time to reimage the FC4 system, but I just want this to be over, so I am now in the middle of reinstalling FC4. The install has been crapping out on the Open Office RPMs. Seems there is a bad spot on the CD for them, so I deselected those packages for install. Only took me three FC4 installations and an hour to realize that I should do this. Oh, I'm just not very smart. :(

OK! What's left? Finishing the FC4 install and the stripe set.

audio/video sync problems..first complete editing job completed

After struggling with video/audio synchronization issues, I've finally got 56 minutes of passable video. Due to my own stupidity (the fact that my Audio-Technica AT822 stereo mic was attached to, but not plugged in to the video cam) and that the audio coming from the cam is pretty lousy, I needed to use my backup recording device. So I had to sync the audio from my Creative Labs Nomad. Cinelerra is good with audio, but I had to resample to 48000hz to match the audio sampling rate of the original video.

I first tried to match the Nomad audio with the video, but that became an exercise in frustration. I still had the camera audio, so matching the Nomad audio to the cam's audio was MUCH easier than matching up to the video. As well, this method allowed me to play all four audio tracks (2 stereo from Nomad/2 stereo from cam) back at the same time. Monitoring this, if I heard bouncback echo, I knew my synchronization was off. So rule to remember, if you have a couple sources of audio, match to the audio, not what's on the screen, cause you'll be pulling out all your hair trying to match to the vid.

Glad this one is in the bag. It is truly my first complete video created in Cinelerra. Nothing special within, just fades/titles and a little time clock at the end. It was reduced from the original HDV to 320x180, 160kbps sound (247MB). Also, the video quality pretty much sucks because I didn't whitebalance or even focus for that matter. It was inevitable, as I was rushing to get setup for the three measly hours we had in the rehearsal space. Ah well.
http://content.serveftp.net/video/20060406.mpg

Update 2/25/08
Here is a helpful article for anyone needing to troubleshoot audio/video sync problems in Cinelerra:
/2007/05/nudge-avidemux2-and-reminder-about.html

a foul up

Last night, I encoded my first full length video podcast as a Quicktime from Cinelerra. I brought it into iTunes and used the 'Convert Selection for iPod' feature and it spent all night (from 11pm to 11am) converting. The .MOV file specs were pretty heavy:
- 50 minutes long
- 720x404 resolution
- 524MB

I used a PIII, 1.5Ghz laptop w/512MB of memory to do the conversion, hence the slowness. Once the file was converted, both audio and video tracks played back! Hooray! This was a lot better than my previous result wherein the audio was gone. So, I found the new .M4V file on the filesystem and looked at the metadata in the properties. I saw that there was no metadata, so not thinking that it could have a deleterious effect on the file, I added the missing metadata to the file as it existed on XP. Bad, nary I say REAL bad idea!

After adding the metadata to the file, the .M4V would no longer play in iTunes (6.0.4) or Quicktime. To put it politely..RATS!! I tried removing the metadata, but no go..the file still would not play. So, the twelve hours I spent converting the video was all for naught, cause now I have to do it all again. Wunnerful.

Let me be clear though..
1) you CAN edit the metadata for an iTunes video podcast within iTunes
2) you CANNOT edit the metadata for an iTunes video podcast in the filesystem. This will break the audio for the podcast.

This being the case, I decided to move my website over to my audio workstation in order to allow me to use my main video editing workstation to do the iTunes conversion. Screw the portable..conversion took way too long there. Copying the website from the video editing box to the audio workstation would allow me to use either of my primary machines to serve web content depending on which one I was using at the time. Both are dual boot setups, XP or Win2K and Fedora Core 4, so at least now I have flexibility. Of course, I wish it was under different circumstances that I make that improvement to my workstation setup. Argh.

So now the file is converting on my P4, 3.2Ghz, 1GB box. I'm hoping to get at least a 3x improvement in conversion time. So, I'm looking at 5pm today for the file to be done. 1-5pm..4hrs. Yeesh. I need a faster box..ain't it always the case!

UPDATE: Good news! It only took two hours to convert the file..sweet! I think the remarkable improvement may be bolstered by the MPEG compression available in my All-in-Wonder 9800 Pro video card. I should switch video cards and try this to verify..

Other things of note I'd like to get done:
- setup RSS feed for newly created video
- update Beginner's Guide for bitrate calculation/QT M4V metadata warning
- change rendering partition to ext2 (non-journaling file system)
- eventually get RAID0 back up and running for rendering partition

At least my cam should be coming back from the shop this week!

don't be in the drum section of a music store when your website is down

I had gone into the city to Manny's music yesterday to look at a Roland PD80R drum pad based on the fact that the guy on the phone at Manny's told me they had the pad in stock. Well, when I got there, the idjut who was at the desk couldn't find the pad in the system. Argh. Trip wasted. But just at this moment, I received a system down call. Double yarg!!

Word to the wise: when you're on a SEV 1 call, the worst place in the world is to be in the DRUM section of a music store.."What? The system card failed??! Site down??!! What??!" Truly frustrating and nerve racking. In the end, I bugged out of the city thinking they didn't have it. Also, I was more interested in getting back to Jersey to be available to help.

So today I call back to speak to the original guy to ask him "what the F happened?" This guys' name is Brian, and Brian indeed told me they had it; however, it is shared inventory with Sam Ash across the street. Apparently, the guy who helped me in the store when I got there didn't check. YARG!! So I went home, double-pissed, cause I didn't get to lighten my wallet with a new toy AND the fcking web site was down!! 16 hours later and I'm finally catch some zzz's. Harumph.

Well, today Brian was good enough to walk across the street to Sam's and verify they had the pads in stock. So I may or may not go.

how's that for a friggin weekend?