dtrace - malloc profiling

Dtrace has been around for long, but due to some restrictions on our solaris zones, I was able to try it out only recently. And what's the verdict? I have been using a lot of debuggers, memory leak analyzers and a few profilers too - and have considerable experience with them. And I have to admit, dtrace turns out to be more impressive than I would have imaginged, and this is just the tip of an iceberg.

For a start - this is what I use to look at all the different paths calling malloc. When you have optimized your program for everything else, you may find malloc to be a big bottleneck, because it uses global locks and can slow down multi-threaded programs considerably. And you would never know of all the paths within your code or in the libraries that you use which may be making malloc calls and holding you back from unleashing the true potential. While mtmalloc can be a solution (its a multithreaded malloc library that offers decent speedup at the cost of extra memory - in one of my tests it decreased startup time from 30 seconds to 20 with a 4GB increase in used memory (from 12G to 16G), which is not bad.

However, premature optimization is the root of all evil. So before you make that call to switch to mtmalloc, just look up the paths that are frequently invoking malloc. The following dtrace one liner will print a trace listing stack dumps and their frequencies for all the calls to malloc.

dtrace -n 'pid$target:libc:malloc:return { @[ustack()] = count(); }' -p `pgrep -n program`

Make sure to replace program above with whatever program you are debugging.

Not just this, with a simple dtrace script, you could also look at all the malloc and free calls and then do some kind of memory leak analysis on the logs. I used this script to identify some leaks, and some inefficient memory management in my program.

#!/usr/sbin/dtrace -s
pid$target:libc:malloc:entry
{
ustack();
}
pid$target:libc:malloc:return
{
printf("%s: %x\n", probefunc, arg1);
}
pid$target:libc:free:entry
{
printf("%s: %x\n", probefunc, arg0);
}

Run it as
dtrace -s malloc.d -p `pgrep program` and make sure you redirect stdout.