Sunday, February 21, 2010

fsarchiver, good backup for ext4 partitions

Since I had lost my root partition the other day (!), I needed a decent method to backup my new ext4 partitions. Since partimage does not currently support ext4, I found fsarchiver:
http://www.fsarchiver.org/

Overview
I've partitioned my new 4.5TB drives this way:
/boot
/root 3.7TB
/backup 800GB


I formatted the backup filesystem as ext3. This way, I can simply boot with a Fedora Live CD, mount the backup partition and backup my boot and root partitions to the backup. Of course, I'll need to roll that backup off to another storage media. But this strategy helps when I make major updates to my system because I can easily rollback to an earlier version that is stored locally.

Most importantly, restore works!

Detail
Here's what I did the other day to get 'er going.

First, I booted to my Fedora Live CD. It didn't have fsarchiver installed by default, so I did so. You need to become superuser to do this:
[liveuser@localhost ~]$ su
[root@localhost liveuser]# yum install fsarchiver
Loaded plugins: presto, refresh-packagekit
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package fsarchiver.i686 0:0.6.7-1.fc12 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================================================================================================
Package Arch Version Repository Size
=============================================================================================================================================================
Installing:
fsarchiver i686 0.6.7-1.fc12 updates 93 k

Transaction Summary
=============================================================================================================================================================
Install 1 Package(s)
Upgrade 0 Package(s)

Total download size: 93 k
Is this ok [y/N]: y
Downloading Packages:
Transaction Test Succeeded
Installing : fsarchiver-0.6.7-1.fc12.i686 1/1
Installed:
fsarchiver.i686 0:0.6.7-1.fc12

Complete!


I created my backup directory and mounted it:
[root@localhost liveuser]# mkdir /mnt/backup
[root@localhost liveuser]# mount -t ext3 /dev/mapper/vg_ogre-lv_backup /mnt/backup
[root@localhost liveuser]# ls /mnt/backup
lost+found test.txt


Finally, I ran fsarchiver to do the backup and took advantage of its multithreaded capability:
[root@localhost liveuser]# fsarchiver -j7 -o savefs /mnt/backup/lv_root_backup.fsa /dev/mapper/vg_ogre-lv_root
Statistics for filesystem 0
* files successfully processed:....regfiles=306990, directories=31024, symlinks=16561, hardlinks=4157, specials=28
* files with errors:...............regfiles=0, directories=0, symlinks=0, hardlinks=0, specials=0


You could concatenate these steps into a script:
[mule@ogre ~]$ cat systemBackup.sh
su -
yum install fsarchiver
mkdir /mnt/backup
mount -t ext3 /dev/mapper/vg-ogre/lv_backup /mnt/backup
fsarchiver -j7 -o savefs /mnt/backup/lv_root_backup.fsa /dev/mapper/vg_ogre-lv_root


Voila! In seven hours, I backed up approximately 450GB of data:
[root@localhost liveuser]$ ll /mnt/backup
total 459289180
drwx------. 2 root root 16384 2010-02-10 20:09 lost+found
-rw-r--r--. 1 root root 480252970629 2010-07-21 07:34 lv_root_backup.fsa


Update:
When my RAID set was not being checked, an fsarchive of about 760GB took 3.5 hours. Not bad!
***end update***

Restore
Restore works in a similar way. Since fsarchiver allows you to backup multiple filesystems within one archive, you need to specify which filesystem is getting restored.

In the example below, the "id=0" specifies the index (starting at 0) of the filesystem that is in the archive.  The filesystem to be restored cannot be mounted:
fsarchiver restfs /mnt/backup/lv_root_backup.fsa id=0,dest=/dev/mapper/vg_ogre-lv_root


If you had multiple file  systems stored in the archive "lv_root_backup.fsa", then the id number would increment; eg, "id=1" for the second filesystem stored in the archive.

"dest" is the destination filesystem which is getting restored, in this case "/dev/mapper/vg_ogre-lv_root"


Password Protection for Archives
You can also specify a password to password protect your archive.  On backup and restore, the password switch looks the same:
sudo fsarchiver restfs backup_vg_ogre-lv_root.fsa id=0,dest=/dev/sdb1 -c [password]

Worked for me on many occasions.
the mule

References
http://www.fsarchiver.org/QuickStart

Thursday, February 11, 2010

compiling Cinelerra on Fedora 12, x86-64 from source

Happily, I'm over the hurdle with my server rebuild from Fedora 10 to Fedora 12 and have successfully built Cinelerra from source on my Fedora 12, x86-64 box! Yes!

The install was generally smooth, with the exception of a minor glitch: as aac encoding is proprietary, ffmpeg does not include faac support. Doh! So, you'll need to compile and install ffmpeg with faac support from source as per these excellent instructions from Doran:
http://fozzolog.fozzilinymoo.org/tech/2009/11/recompiling-ffmpeg-for-fedora-12-to-add-faac-support.html

RPM Fusion Yo!
Note that you'll need both the free and non-free repositories from RPM Fusion enabled in yum. Don't use the ATrpms repos if you're going to compile from source. I've had strange system problems when I've used them in combination with the Fusion repos.

So here's the deal..

Summary Steps to Installing Cinelerra from Source
  1. download source code
  2. install dependencies for ffmpeg compile
  3. compile ffmpeg with faac support for fedora 12
  4. install rest of Cinelerra dependencies
  5. compile cinelerra

Detailed Steps to Installing Cinelerra from Source
1. Get the Cinelerra source code
git clone git://git.cinelerra.org/j6t/cinelerra.git my_cinelerra

2. Install dependencies for ffmpeg compile
Feel free to use the following script (don't forget to chmod a+x it!):
yum install dirac-devel faac-devel faad2-libs gsm-* imlib2-devel lame-libs* libdc1394-devel libvdpau-devel openjpeg-devel schroedinger-devel speex-devel texi2html x264-* xvidcore-* yasm

3. Compile ffmpeg with faac support

Highlights
[sodo@computer ~]$ yumdownloader --source ffmpeg

[sodo@computer ~]$ rpm -ivh ffmpeg-0.7.6-1.fc15.src.rpm

[sodo@computer ~]$ vi rpmbuild/SPECS/ffmpeg.spec
[sodo@computer ~]$ rpmbuild -ba ~/rpmbuild/SPECS/ffmpeg.spec --with faac

[sodo@computer ~]$ cd rpmbuild/RPMS/
[sodo@computer RPMS]$ cd x86_64/
[sodo@computer x86_64]$ ls -ltr
total 39164
-rw-rw-r--. 1 sodo sodo  259049 Dec 18 14:56 ffmpeg-0.7.6-1.fc15_fozz.x86_64.rpm
-rw-rw-r--. 1 sodo sodo 3138877 Dec 18 14:56 ffmpeg-libs-0.7.6-1.fc15_fozz.x86_64.rpm
-rw-rw-r--. 1 sodo sodo  166417 Dec 18 14:56 ffmpeg-devel-0.7.6-1.fc15_fozz.x86_64.rpm
-rw-rw-r--. 1 sodo sodo 9860213 Dec 18 14:56 ffmpeg-debuginfo-0.7.6-1.fc15_fozz.x86_64.rpm
[sodo@computer x86_64]$ sudo rpm -Uvh ffmpeg-libs-0.7.6-1.fc15_fozz.x86_64.rpm ffmpeg-0.7.6-1.fc15_fozz.x86_64.rpm ffmpeg-devel-0.7.6-1.fc15_fozz.x86_64.rpm ffmpeg-debuginfo-0.7.6-1.fc15_fozz.x86_64.rpm
[sudo] password for sodo: 
Preparing...                ########################################### [100%]
   1:ffmpeg-libs            ########################################### [ 25%]
   2:ffmpeg                 ########################################### [ 50%]
   3:ffmpeg-devel           ########################################### [ 75%]
   4:ffmpeg-debuginfo       ########################################### [100%]


4. Install rest of Cinelerra dependencies
Feel free to use the following script (don't forget to chmod a+x it!):
yum install gsm-devel libvorbis* libogg* libtool* libtheora* libpng* libjpeg* libtiff* esound* audiofile* libraw1394* libavc1394* freetype* fontconfig* nasm e2fsprogs* OpenEXR* fftw fftw-devel libsndfile* libiec61883* libdv* libquicktime ffmpeg xvidcore* lame lame-devel a52* faad2* x264* mjpegtools* faac* vlc*

5. Compile Cinelerra
If you used the same default directory when you downloaded the source code, cd into that directory and type:
./autogen.sh

This will build your configuration files. Next, to configure Cinelerra with default options, type:
./configure

If you want to customize your configuration, just type "./configure --help". At which point, configure will spill its guts to you about all the Secrets of Cinelerra.

Then, to compile the code, run:
make

If you have multiple processors, you can even run "make -j N" where "N" is the number of processors you have minus one.

Finally, if make was successful or you are just feeling good about yourself, go ahead and install Cinelerra:
make install

Hopefully, this post has gotten you started on your way to using Cinelerra, Linux's "50,000 watt flamethrower of multimedia of editing"

Much thanks to Jack Crossfire, our patron saint of compositing code.

the mule

Reference
Don't forget to read the manual!

Saturday, February 06, 2010

lost root partition..oops.

I was running some disk performance statistics on the new Fedora 12 64-bit yesterday according to the very good benchmarking article on 3ware's site:

Benchmarking
I was benchmarking the write performance of my RAID set when it seemed to stall out. The process I was running was writing a bunch of zeros to a 20 gigabyte file. I believe the stall was due to the fact that my RAID controller card's battery was disconnected; hence, write-cacheing was disabled.

I let the process try to finish for four hours. I figured it should have finished writing that 20GB file by that time. However, the fact that the system was still slow to non-responsive indicated that activity was still taking place. But, being an impetuous fool, I was anxious to get working on some video and also thought it might be an interesting test of the resilience of the ext4 filesystem if I just shut the system down. So I as a soft reboot did not do the trick, I hard powered the box off.

Sleeping the Sleep of the Dead
In retrospect, I should have let the box finish whatever it was doing, because as you may have guessed it, the box didn't come back up. Here was the first indication from the kernel messages:
Boot has failed..sleeping forever

And in the dmesg output:
can't mount root filesystem
can't access tty job control turned off

Woops. Dracut did find my volume group:
dracut: 2 logical voumes in "vg_ogre" now active

Something was wrong with the root filesystem mount:
mount: you must specify the filesystem type

Just in case, I rebooted with the following kernel parameters in grub to see more debugging and to drop me to an emergency shell to see if I could debug the problem:
kernel .. debug rdshell

What Up, ext4?
Oh boy. So, ext4 is not as resilient as I believed. I thought the best course of action would be to load up Fedora Live, and look at the disk stats. Since fdisk does not work with GPT partitions, I used parted and thought that I'd use e2fsck to fix any bad blocks. After booting the Live CD, here's what I found:

The swap drive seemed in tact (oh, great):
[liveuser@localhost ~]$ dmesg | grep vg
vgaarb: device added: PCI:0000:07:00.0,decodes=io+mem,owns=io+mem,locks=none
vgaarb: loaded
Adding 12369912k swap on /dev/mapper/vg_ogre-lv_swap. Priority:-1 extents:1 across:12369912k


I thought I'd try to manually mount my / partition. I had to become superuser in order to do this:
[liveuser@localhost ~]$ su
[root@localhost liveuser]# mkdir /mnt/root
[root@localhost liveuser]# mount -t ext4 /dev/mapper/vg_ogre-lv_root /mnt/root
mount: wrong fs type, bad option, bad superblock on /dev/mapper/vg_ogre-lv_root,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so


Dmesg tells me what I already know:
[root@localhost liveuser]# dmesg | tail
[drm] nouveau 0000:07:00.0: 0x00409910: 0x3fbf3fdb
[drm] nouveau 0000:07:00.0: 0x00409e08: 0x0002dea8
[drm] nouveau 0000:07:00.0: 0x00409e0c: 0x00000000
[drm] nouveau 0000:07:00.0: 0x00409e24: 0x0a21026f
EXT4-fs (dm-2): VFS: Can't find ext4 filesystem


I just want to see what fdisk reads about my hardware RAID5 array (3ware 9650SE):
[root@localhost liveuser]# fdisk -l /dev/sda

WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk doesn't support GPT. Use GNU Parted.

WARNING: The size of this disk is 4.5 TB (4499967049728 bytes).
DOS partition table format can not be used on drives for volumes
larger than (2199023255040 bytes) for 512-byte sectors. Use parted(1) and GUID
partition table format (GPT).


Disk /dev/sda: 4500.0 GB, 4499967049728 bytes
255 heads, 63 sectors/track, 547089 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000f0844

Device Boot Start End Blocks Id System
/dev/sda1 1 267350 2147483647+ ee GPT


What does parted see about /dev/sda?
[root@localhost liveuser]# parted /dev/sda print
Model: AMCC 9650SE-4LP DISK (scsi)
Disk /dev/sda: 4500GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number Start End Size File system Name Flags
1 17.9kB 210MB 210MB ext4 boot
2 210MB 4500GB 4500GB lvm


At least the partition is there. But it looks like parted does not have support for checking ext4 filesystems yet:
[root@localhost liveuser]# parted /dev/sda
GNU Parted 1.9.0
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) check 1
No Implementation: Support for opening ext4 file systems is not implemented yet.
(parted) check 2
Error: Could not detect file system.
(parted) quit


e2fsck bound!
Let me run e2fsck (which does have support for ext4 filesystems) and see if I can fix the problem:
[root@localhost liveuser]# e2fsck
Usage: e2fsck [-panyrcdfvtDFV] [-b superblock] [-B blocksize]
[-I inode_buffer_blocks] [-P process_inode_size]
[-l|-L bad_blocks_file] [-C fd] [-j external_journal]
[-E extended-options] device

Emergency help:
-p Automatic repair (no questions)
-n Make no changes to the filesystem
-y Assume "yes" to all questions
-c Check for bad blocks and add them to the badblock list
-f Force checking even if filesystem is marked clean
-v Be verbose
-b superblock Use alternative superblock
-B blocksize Force blocksize when looking for superblock
-j external_journal Set location of the external journal
-l bad_blocks_file Add to badblocks list
-L bad_blocks_file Set badblocks list


My skills at e2fsck are pretty basic. I use the -n option to make no changes while I review what e2fsck finds out about the array:
[root@localhost liveuser]# e2fsck -n /dev/mapper/vg_ogre-lv_root
e2fsck 1.41.9 (22-Aug-2009)
e2fsck: Superblock invalid, trying backup blocks...
Superblock has an invalid journal (inode 8).
Clear? no

e2fsck: Illegal inode number while checking ext3 journal for /dev/mapper/vg_ogre-lv_root


Invalid journal..oops.
[root@localhost liveuser]# e2fsck -v /dev/mapper/vg_ogre-lv_root
e2fsck 1.41.9 (22-Aug-2009)
e2fsck: Superblock invalid, trying backup blocks...
Superblock has an invalid journal (inode 8).
Clear
?y

I had thought that ext4 gave us the safety of a journalled filesystem (like ext3) with increased performance. You would have thought it could have recovered from being shutdown while writing a bunch of zeros to a 20 gigabyte file.

And then of course, hundreds to thousands of these various errors:
Group descriptor 32923 checksum is invalid. FIXED.

Entry 'e61abf8156cc476151baa07d67337cae-le64.cache-3' in ??? (57347) has deleted/unused inode 212. Clear? yes

Unconnected directory inode 98305 (...)
Connect to /lost+found? yes

Free blocks count wrong for group #138 (32768, counted=557).
Fix? yes

Free inodes count wrong for group #308 (8192, counted=8186).
Fix? yes

Directories count wrong for group #308 (0, counted=6).
Fix? yes


Finally..at the bottom of the list of errors:
Recreate journal? yes

Creating journal (32768 blocks): yyyyyyy Done.

*** journal has been re-created - filesystem is now ext3 again ***

/dev/mapper/vg_ogre-lv_root: ***** FILE SYSTEM WAS MODIFIED *****

327475 inodes used (0.12%)
585 non-contiguous files (0.2%)
130 non-contiguous directories (0.0%)
# of inodes with ind/dind/tind blocks: 0/0/0
Extent depth histogram: 310327/414/1
239381919 blocks used (21.85%)
0 bad blocks
42 large files

283167 regular files
27385 directories
0 character device files
0 block device files
0 fifos
3953 links
16849 symbolic links (16659 fast symbolic links)
63 sockets
--------
331417 files
[root@localhost liveuser


Result?
So let's see if I have files in tact after that 18 hour experience..
[root@localhost liveuser]# mount -t ext4 /dev/mapper/vg_ogre-lv_root /mnt/root/
[root@localhost liveuser]# ls /mnt/root
lost+found
[root@localhost liveuser]# ls /mnt/root
lost+found
[root@localhost liveuser]# ls /mnt/root/lost+found/

*348489 *723483 324843 238390

Ah..that would be a "no." Time to reinstall F12. Ugh. Lesson learned. But I need to know why I couldn't recover a journal. Maybe I did not look in the right place. I need to understand journalling better.

Things I Learned Along the Way
Some boot info from the Live CD
[root@localhost liveuser]# grep EFI_ /boot/config-2.6.31.5-127.fc12.i686
CONFIG_EFI_VARS=y
CONFIG_EFI_PARTITION=y


You can force a filesystem check upon the next reboot with this command:
shutdown -rF now

You can run verbose debug messages and drop to an emergency shell by placing these commands on the kernel line in grub:
kernel /vmlinuz-2.6.31.12-174.2.3.fc12.x86_64 ro root=/dev/mapper/vg_ogre-lv_root debug rdshell

References

Thursday, February 04, 2010

time average video effect

Time average is a neat effect. It takes a number of frames of video to average or accumulate.


:averaging
In the below video, I show averaging. Averaging lots of frames gives a very ghost-like effect. Average very few frames shows trails of whatever movement is happening. Medium settings are like you've had too much to drink.

Cinelerra time average effect from crazed mule on Vimeo

:accumulate
A second option called Accumulation is just that..stacking one frame on top of the other. If the scene in the frame is well lit, the frame will blowout..make it so bright that all detail will be lost. The detail gets lost quickly too, when just a few frames are being accumulated. Accumulate is more useful for extremely dark scenes, like making very distant stars become more visible, as in the example below. In the example, I show the original and the time averaged result:

Cinelerra time average effect: accumulate from crazed mule on Vimeo

There is another time averaging parameter called "Inclusion Or" that I have not used. It is explained in the link to the manual below as are the two options "Reprocess Frame Again" and "Disable Subtraction."

cheers,
the mule

Reference
Cinelerra User Manual