Analyzing linux resources with Systemtap

Systemp lets you gather information about your running linux system. So you can diagnose performance and problems on your system(I/O activity, network, sockets,etc.). It provides a command line interface and a scripting language. There is also a gui interface called stapgui on sourceforge.net. Installing it on Ubuntu is straight forward

sudo apt-get install systemtap
sudo apt-get install linux-image-debug-generic
sudo ln -s /boot/vmlinux-debug-$(uname -r) /lib/modules/$(uname -r)/vmlinux

Systemtap uses kprobes to trace events. Kprobes are markers which are placed on predefined points in kernel. For example if you like to see which programs are run by your system you call syscall exec probe.
Here is a simple stap script which shows which programs are run by your system.


#!/usr/bin/env stap
probe syscall.exec* {
printf("exec %s %s\n", execname(), argstr)
}

As you can see, stap is similiar to C language.
The key point is to choose which probe point you would like to use.
It is also possible to probe only which process you want by pid or it's name.For example:
probe process("/bin/bash").syscall

Here is another example, suppose you have a disk I/O activity and want to know which processes are causing this.

#!/usr/bin/env stap

global reads, writes, total_io
probe kernel.function("vfs_read").return {
reads[execname()] = total_io[execname()] += $return
}
probe kernel.function("vfs_write").return {
writes[execname()] = total_io[execname()] += $return
}
probe timer.s(1) {
foreach(p in total_io- limit 10)
printf("%15s r: %8d KiB w: %8d KiB\n",p, reads[p]/1024,writes[p]/1024)
printf("\n")
}


You can read further at systemtap site
http://sourceware.org/systemtap/langref/

Linux Block Tracing

blktrace let's you see detailed information about I/O traffic on a disk devices.

You need to have a 2.6.17-rc1 or newer kernel with "Block io tracing" and debugfs enabled in kernel to use blktrace command.


You can download and install blkrace from http://brick.kernel.dk/snaps/ site or if you use ubuntu you can install it with apt-get.
on ubuntu:

$ sudo apt-get install blktrace


blktrace requires debugfs to run. run mount command to see if it's mounted.
If you can't see debugfs in mount command output for example create a directory called /debugfs and mount it by following command as root

# mount -t debugfs none /debugfs

now you can run blkrace command.


blktrace -r /debugfs/ -d /dev/sda -o - | blkparse -i -

default format of blkparse output is as follows (you can change output format if you like with -f option):





by this way you can get detailed information about ongoing disk activity on your system.

Description of Events:

A IO was remapped to a different device

B IO bounced

C IO completion

D IO issued to driver

F IO front merged with request on queue

G Get request

I IO inserted onto request queue

M IO back merged with request on queue

P Plug request

Q IO handled by request queue code

S Sleep request

T Unplug due to timeout

U Unplug request

X Split

simple load balancing with iptables

iptables has an extension called clusterip. Clusterip extension uses multicast arp feature to achieve load balancing. Let's say we have two web servers called web1(192.168.0.1) and web2(192.168.0.2) and a virtual ip (192.168.0.10) which will be accepting requests for these machines.

virtual ip:192.168.0.10
web1:192.168.0.1
web2:192.168.0.2

Virtual ip will accept the requests and load balance them between these two web servers.

on web1 server run:

# iptables -I INPUT -d 192.168.0.10 -i eth0 -p tcp --dport 80 -j CLUSTERIP --new --clustermac 01:02:03:04:05:06 --total-nodes 2 --local-node 1 --hashmode sourceip
# ifconfig eth0:1 192.168.0.10 netmask 255.255.255.0 up

on web2 server run:
# iptables -I INPUT -d 192.168.0.10 -i eth0 -p tcp --dport 80 -j CLUSTERIP --new --clustermac 01:02:03:04:05:06 --total-nodes 2 --local-node 2 --hashmode sourceip
# ifconfig eth0:1 192.168.0.10 netmask 255.255.255.0 up

only difference between web1 and web2 commands is local-node option as seen above.
now any web requests coming to 192.168.0.10 will be load balanced between web1 and web2.

clusterip supports three hashmodes (sourceip,sourceip-sourceport and sourceip-sourceport-destport) to determine how to route requests to each servers.

This configuration has one drawback. If one of the nodes fall, the other one does not serves incoming requests for the other one. You need to install linux-ha.

If you want to see which requests served by web1 for example, simply run
# cat /proc/net/ipt_CLUSTERIP/192.168.0.1

Let's say web2 is crashed and we would like to first web server (web1) to take care the requests coming to web2.
on web1 server run:
# echo "+2" >> /proc/net/ipt_CLUSTERIP/192.168.0.1
now on, web1 will take the requests coming to web1.
When you up the web2 server, just run
# echo "-2" >> /proc/net/ipt_CLUSTERIP/192.168.0.1
and web1 will not serve the request for web2.

clamsmtp rbl support patch

clamsmtp is lightweight anti virus smtp proxy. You can put it in front or back of your smtp server. But if you run it in front of your smtp server and relay traffic back to your original smtp server, you cannot use any rbl server.

So, I decided to write a patch which adds this new rbl option to clamsmtpd server. You can download patch from here clamsmtp rbl patch

qmail linefeed problems hotmail , dcc and others

While I was testing DCC Checksum , I discovered that some email messages are truncated and dcc reports that body part is missing with an error message "missing message body". I figured out that not only the body part is missing also the headers also truncated.

A quick investigation with tcpdump showed that spamc is sending truncated messages to spamd server which is passed to dcc server. A second tcpdump run on smtp server showed that some messages is rejected by qmail server with "451 See http://pobox.com/~djb/docs/smtplf.html." message. This was hitting a small amount of mails but the ones especially coming from hotmail.com.

I have quickly fixed the problem with djb's fixcrio application which comes with DJB's ucspi-tcp package. No more "missing message body" error messages anymore with dcc :)

Running fixcrio is easy. Just put the command before qmail-smtpd where you run it.

VI tricks part one

I have finished first part of my VI editor tricks article. Part one explains main commands and tricks like sort data, replacing selected lines and feeding vi document content as input to any program while getting output as new document.
You can read VI tricks part one here.

FreeBSD GJournal (Filesystem Journal)

Freebsd 7.0 has a new geom extenstion called gjournal. You can easily create a journaled filesystem with gjournal on bsd systems.
First of all, gjournal support has to be built in kernel. It's currently built in 7.x series. The kernel option is called UFS_GJOURNAL

You should disable soft updates on filesystems where gjournal is used. Because journalling takes place and there is no need to use soft updates for data recovery on a crash.

While the filesystem is journalled, the mount option async will boost the performance.


For example, let's say we have a disk called ad1 and it has existing filesystem on it.

All we have to do is create a label.

# gjournal label -f /dev/ad1

notice the -f option. It will force convert operation of an existing file system to journal. Journal device will be separate and called in this example as /dev/ad1.journal.

now it's time to load geom gjournal kernel module.

# gjournal load

now add journalling support to our existing filesystem.

# tunefs -J enable /dev/ad1.journal

it's better now to disable soft updates.

# tunefs -n disable /dev/ad1.journal


now we mount our new journalled filesystem with async option.

# mount -o asyn /dev/ad1.journal /mnt

in our example, we don't have any slices on our disk. In slice case new devices will be created (eg. /dev/ad1s1.journal)