FreeBSD kernel profiling with kgmon

Sometimes, it's not possible to find a system bottleneck by profiling and debugging user space processess. You have to profile the kernel to pinpoint the perfomance bottleneck. Here, I will show how to configure a freebsd system for kernel profiling and using kgmon utility to gather kernel profiling data. kgmon produces gprof compatible output. The default output file is called gmon.out .

First of all, we need a profiling enabled kernel configured and build. First step is to use config utility. Now, we need to run config utility from our kernel source configuration directory. I'm using a i386 system. The conf file directory depends on your architecture.
The default directory is /usr/src/sys/ARCH/conf. In my case it's /usr/src/sys/i386/conf/.
We run config utility from where our kernel conf file resides with parameter -p. This tell's the compiler to compile our kernel for low resolution profiling. If we need a high resolution profiling we need to run config with -pp parameter instead of -p. config utility needs a second parameter called SYSTEM_NAME. This is our default system name. FreeBSD's default system name is GENERIC. This can be different on your system, if you build a new kernel with different config file. Older versions of freebsd default GENERIC kernels was not using smp. So it's possible to you have build a new kernel for smp.Therefore you can have a custom build kernel on your system or for another reason. You can see your default system name by using "uname -a" command.
Here is mine is called TESTKERNEL. You'll probably see GENERIC in your case.





After learning you SYSTEM NAME, it's now time to configure our kernel source for profiling. The following picture shows you how to run config with -p option on i386 arch with GENERIC system name.




After running "make cleandepend && depend" command, we run the "make" command on the same directory and it will built a profiling enabled kernel for us.



After the make command, we have build our kernel, now run "make install.debug" command and it will install our debug kernel.

Now, it's time to reboot our machine. After rebooting check kernel messages for following message to see if build is successful for profiling enabled kernel Addresses should be different but seeing message "Profiling kernel" means it's ok.

Profiling kernel, textsize=6845824 [c045cbe0..c0ae4160]

It's time to run kgmon to collect profiling data on our kernel.

# kgmon -b

Now kernel profiling is running for low profiling. For high profiling -B option must be used and don't forget you have to use config -pp for high profiling.
You can now, run the application which was causing the problems on your system to profile it, or create the same situation on server where you were having problems.
Now, after creating and seeing the problem, let's kgmon to dump the profiled kernel data.

# kgmon -p

As I said before, kgmon will create a file called gmon.out for profiled data in gprof format.

Now, we gathered our data, we can now stop profiling.

# kgmon -h

We collected the data we need. Let's see what the gmon.out file contains.

# gprof /boot/kernel/kernel gmon.out

Here is a snippet from gprof output.




I'll explain the meaning of the gprof output in another post.

NOTE: you can apply this technique to other freebsd deriatives like dragonfly bsd and netbsd.

simple tcp server with djb tcpserver

DJB ucspi-tcp package has a program called tcpserver. Simply it accepts incoming tcp connections for a given port and run specified program when a connection established on that port. The good part is that you can write a program in any language which reads stdin and writes data to stdout file descriptors. In this way, you can read data coming from tcp socket simply by stdin file descriptor and send data to tcp client side by writing desired data on stdout file descriptor.

Here is a simple perl code snippet which reads stdin and writes data to stdout.

#!/usr/bin/perl


$| = 1;
print "Hello !\r\n";

while ($line=<STDIN>) {

print "Your input is:".$line."\r\n";

}

Let's call this file as simple.pl
Here is the tcpserver command to run this program on port 9090. tcpserver has other parameters such as given uid and gid to running process. Please see tcpserver man page for details.

# tcpserver -vRH -l test 0 9090 simple.pl

now, you can telnet your server's 9090 port from another machine and test it.

lsof alternatives on FreeBSD

lsof is a utility which gives information about open sockets/files/pipes on many unix systems. You can easily install lsof and try it on your freebsd installation by using pkg_add command.

# pkg_add -r lsof

But FreeBSD has two utilities coming bundled with default installation. They are called fstat and sockstat.

fstat tells you which user,command and pid opened the file, which mount point the file is and information about the open file descriptor like read/write, inode number and mount point of the opened file.Please see man page of fstat for other options.

sockstat gives you information about the opened sockets like which process/command is using it, user of the process,pid,protocol like tcp/udp,stream,dgram,etc.. and connected ports of local and remote servers.