Sunday, January 24, 2010

compile time performance improved!

I was compiling Cinelerra today and noticed that CPU usage was very low during the compile..around 10-15% utilized. I have a dual CPU, quad core box. This makes for a total of eight processors. So with all those CPUs, I figured there must be a way to make compiling faster.

Actually, this low CPU use during compiles was something I had noticed the first time I installed Cinelerra. Ashamedly, I've forgotten to investigate this issue in the two years that I've had the box. So today I googled for "make compiler see multiple CPUs" and found the -j switch to "make" the program that does the compiling:
http://blogs.koolwal.net/2009/04/20/tip-compile-your-programs-fasters-with-multiple-processor-machines/

This article also mentions the CONCURRENCY_LEVEL environment variable, but that variable did not work for my box, a Dell SC1430. So I used the -j switch to make instead:
[mule@ogre my_cinelerra]$ make -j7
(CDPATH="${ZSH_VERSION+.}:" && cd . && /bin/sh /home/sfrase/my_cinelerra/missing --run autoheader)
rm -f stamp-h1


Here are my results.

the time it took to compile Cinelerra normally, without -j:
3min 45s
the time it took to compile Cinelerra with the -j8 (for eight cores):
1min 16s

Holy crap! That's a 300% speed improvement!

Glad I finally researched this.
scott

ps - one other note: 7z is a multithreaded version of tar. On Fedora, use 7za
Installing : p7zip-4.65-2.fc12.x86_64
7za - A file archiver with highest compression ratio

SYNOPSIS
7za [adeltux] [-] [SWITCH]


Related Posts
http://crazedmuleproductions.blogspot.com/2007/10/multithreading-in-ffmpeg-and-mpstat.html

2 comments:

Paul said...

As a (former) gentoo user the -j flag is well known to me. I (and other gentoo'ers) have done some testing and found that -j(N+1) where N equals the number of cores gives the best results. On my quad core I could compile the kernel in 1 minute when using -j5. Going higher than N+1 did not improve anything, it could even get worse.

Cacasodo said...

Paul,
Interesting. I will have to give lower values a try. I was just excited that the compile times decreased.
;)

thanks for the post..