Levent Serinol's Blog

Boosting Radware Defense Pro's performance ;)

2010-06-25T08:34:00.001-07:00

Detailed network interface statistics on Linux

2010-02-08T05:36:00.001-08:00

Here is a helpful command for getting interface statistics on your linux box. It's called ip. It's bundled with iproute package. You can see information about dropped, collisions,crc,transmit and other info about your network interface.


root@helga:~# ip -s -s link show eth0
2: eth0:  mtu 1500 qdi
WN qlen 1000
  link/ether 00:24:8c:d5:6a:a7 brd ff:ff:ff:ff:ff:ff
  RX: bytes  packets  errors  dropped overrun mcast
  1633058095 16933162 0       0       0       0    
  RX errors: length  crc     frame   fifo    missed
             0        0       0       0       0    
  TX: bytes  packets  errors  dropped carrier collsns
  1994496524 5588574  0       0       0       0    
  TX errors: aborted fifo    window  heartbeat
             0        0       0       0

I/O usage per process on Linux

2009-09-11T03:58:00.000-07:00

Linux kernel 2.6.20 and later supports per process I/O accounting. You can access every process/thread's I/O read/write values by using /proc filesystem. You can check if your kernel has built with I/O account by just simply checking /proc/self/io file. If it exists then you have I/O accounting built-in.


$ cat /proc/self/io
rchar: 3809
wchar: 0
syscr: 10
syscw: 0
read_bytes: 0
write_bytes: 0
cancelled_write_bytes: 0


Field Descriptions:

rchar  - bytes read
wchar  - byres written
syscr  - number of read syscalls
syscw  - number of write syscalls
read_bytes  - number of bytes caused by this process to  read
            from underlying storage
write_bytes - number of bytes caused by this process to written from
            underlying storage

As you know, ever process is presented by it's pid number under /proc directory. You can access any process's I/O accounting values by just looking /proc/#pid/io file. There is a utility called iotop which collects these values and shows you in like top utility. You see your processes I/O activity with iotop utility.

problem compiling Mysql 5.1 with sphinx engine support

2009-09-04T12:24:00.000-07:00

Mysql's full-text search engine is not very powerful and support comes only with Myisam engine. Sphinx is a full-text search engine replacement for Mysql and Postgresql. Also, It's possible to use Sphinx full-text search engine with both Myisam and InnoDB engines. You can read detailed information about mysql compilation with sphinx on Sphinx site.

I have faced with following error message, while compiling Mysql with innodb and sphinx engine support ( ./configure --prefix=/data/mysql3/ --with-plugins=sphinx,innobase). I used the innodb (base) which bundled with Mysql. You can also compile innodb plugin which has additional features. For example, on the fly compression, speed improvements.


configure: error: unknown plugin: sphinx

The trick was to run autorun.sh command in Mysql's source tree before running configure with sphinx engine support. This script will create a new configure script which knows how to compile sphinx engine.


# ./BUILD/autorun.sh

nethogs network utility

2009-09-03T00:31:00.000-07:00

Nethogs is a small but very useful utility, if you like to see your machine's bandwidth usage per processes. Here is a simple screenshot from my linuxbox.

Fast Copy on Unix systems

2009-07-12T01:41:00.000-07:00

I needed to copy so many small files and directories (1.2 TB) to another machine with netcat (nc). But I faced with a problem when I tried to run nc program on the background with bash. Bash was suspending standard input when I move nc process to background job, this was causing nc processess to die when standard input was blocked. So, I have found another netcat like utility called socat. Socat is not stop execution while it was working as background job on bash shell. I was able to utilize 100mbit ethernet with few socat processes on server and client side. First of all, I run socat processes on destination server as daemons to listen specific ports. Following, command will run socat in daemon mode to listen on tcp port 4123 and pipe incoming data to tar command on the background.

# (socat -u tcp4-listen:4123 - | tar xzp -C .) &

On source server, I run socat to feed incoming tar data to destination socat daemon where it listens on tcp port 4123. My destination server's ip adddess was 192.168.36.199 .

# (tar -czp - /mydir/1/ | socat -u - tcp4:192.168.3.199:4123)&

So, with a simple bash script I was able to run few socat daemons on destination server with different ports and feed them from source server to achieve higher data transer rate.

socket programming with Bash

2009-06-08T09:13:00.000-07:00

I have many memcached servers around. So, I needed a tool to check status information on these servers like top utility. I decided to write it in bash scripting language (100% in bash except printf,date and sleep commands :) ). You can simply enhance the script for example getting server names and ports from a config file or display interval parameter from command line. Memcached supports tcp and udp protocols for communication with clients. I used tcp communication in my bash script. Here is information about Memcached's supported server commands. My example script uses "stats" command. Here is sample output for "stats" command from one memcached server.

I have used the information provided by "stats" command. Script uses pure bash builtin commands to access a tcp server like memcached.

You can download the script from here.

Here is sample output from the script.

If you're using Ubuntu, probably you will not be able to run this script, because bash is not compiled with "--enable-net-redirections" parameter on Ubuntu systems. RedHat based distributions has no problem with bash net-directions like Fedora,Centos and RedHat ES.

Mirroring your OpenSolaris root disk with ZFS

2009-05-31T02:06:00.000-07:00

Solaris/OpenSolaris supports booting with ZFS filesystem. For example, if you look into your opensolaris insallation you'll notice the zfs rpool (root pool). ZFS is default filesystem on opensolaris (optional with Solaris). If you look into detailts of that pool with zpool utility , you'll see that it's a single disk or a partition (depends how you partitioned your disk on installation process of opensolaris). If that single disk crashes, you may lost your data and operating system.That's one point of failure. The simple solution is to convert our zpool into RAID-1 with a secondary disk (with same capacity or bigger one).

As you can see, our zpool contains a single disk called c4t0d0. Let's add a new disk to our rpool to convert it into mirror. zpool will do it automatically for us. One important point here is that Solaris doesn't support boot devices to be in EFI label format. So, if your new disk label is in EFI format, you need to convert it into SMI label. So, I'll use fdisk command to create a Solaris2 partition and remove any EFI labels if present. My new disk name is c4t1d0.

After creating the partition which covers whole disk area, I copied partition map of original rpool member c4t0d0 to our new disk c4t1d0.

Now, it's time to attach our brand new disk into rpool. After attaching the disk, our rpool pool will be converted into mirror (raid-1). Except one thing, the grub boot manager. We need to install it into new disk by hand using installgrub command.

Now, we have created our mirror on rpool. It will take time to sync original disk to new disk. You can watch this process by using zpool's status command.

FreeBSD and procfs

2009-05-23T03:21:00.001-07:00

Many Unix systems have support for proc file system (process file system).Procfs filesystem type is pseudo. FreeBSD is one of that Unix systems. Unlike Linux, which has information other than processes, FreeBSD procfs support is only about the processes on the system. FreeBSD doesn't mount the procfs on boot by default. You need to manually add it to fstab for auto mount on boot or mount it by command for temporarily usage.Common mount point for procfs is /proc on Unix systems.


echo "none            /proc        procfs  rw   0 0" >> /etc/fsab


mount -t procfs none /proc

Every process is presented as directories named by it's pid number on the /proc mount point.
Procfs gives information on running processes on the system like memory mapping,command line arguments of running process, process resource limits and many other.Following is a sample procfs directory structure on a FreeBSD machine.

As you can see, every pid is represented as a directory in procfs. Every directory contains following files. Some of the files are write only or read only where you read information or send information to process.



- status  (read-only) : returns process status
- mem     (read/write): virtual memory image of the process
- file    (depends)   : symbolic link to running process
- regs    (read/write): process registers
- ctl     (write-only): used to send signal to process or 
                        attach/deattach it for debugging
- cmdline (read-only) : command line arguments of running process
- rlimits (read-only) : current resource limits of running process
- map     (read-only) : memory mappings of the running process.
- etype   (read-only) : type of the executable (eg. FreeBSD ELF32)
- fpregs  (read/write): floating point registers

Some of the information provided by these files are in binary format.For example "regs" and "fpregs" files are in binary format. They depend on the architecture of the underlying machine (i386, amd64,sparc64,etc..). Following is the format of the "regs" file on the i386 machine.



struct reg {
        unsigned int    r_fs;
        unsigned int    r_es;
        unsigned int    r_ds;
        unsigned int    r_edi;
        unsigned int    r_esi;
        unsigned int    r_ebp;
        unsigned int    r_isp;
        unsigned int    r_ebx;
        unsigned int    r_edx;
        unsigned int    r_ecx;
        unsigned int    r_eax;
        unsigned int    r_trapno;
        unsigned int    r_err;
        unsigned int    r_eip;
        unsigned int    r_cs;
        unsigned int    r_eflags;
        unsigned int    r_esp;
        unsigned int    r_ss;
        unsigned int    r_gs;
};

You can use "cat" command to read information provided by procfs for text based information unlike the ones I mentioned above in binary format like "regs","fpregs" and "mem".



[root@freebsd ]# cat cmdline 
/usr/sbin/moused-p/dev/ums0-tauto-I/var/run/moused.ums0.pid

You can check a running process's resource limits by looking into rlimit file.



[root@freebsd ]# cat rlimit 
cpu -1 -1
fsize -1 -1
data 536870912 536870912
stack 67108864 67108864
core -1 -1
rss -1 -1
memlock -1 -1
nproc 5547 5547
nofile 11095 11095
sbsize -1 -1
vmem -1 -1

First digits is minimum and last one is maximum values of the given resource name. "-1" means infinite. For examle, nofile (open file descriptor) limit for this process is 11095 as minimum and maximum.

"status" file gives information about process status as follows.


- command name
- pid
- parent pid
- process group id
- session id
- major/minor of the terminal, "-" if no terminal is in action
- process flags
- process start time in seconds and microseconds separated by comma
- user time in seconds and microseconds separated by comma
- system time in seconds and microseconds separated by comma
- wait channel name
- effective userid and group lists separated by comma

Following is "cat status" result of a process.

[root@freebsd ]# cat status 
svscan 28246 1 28245 0 ttyp0 noflags 1242921860,839572 0,318052 3,263576 nanslp 0 0 0,0,0,5 -

Configuring ILOM manager on Sun X4540 Thumper

2009-05-16T04:49:00.000-07:00

Sun X4540 Thumper storage server provides 48TB hard disk capacity on one server. It has 2 AMD Quad x64 Opteron processors and 32GB of memory. Thumper has 48 hard disk slots. You can use 1TB sata hard disks in each slot. This will give you 48TB hard disk area as you can see in following photo.

You can run Solaris,OpenSolaris, Linux and Windows operating systems on Thumper. Thumper includes Sun Microsystem's ILOM (Sun Integrated Lights Out Manager) manager. You can use ILOM to upgrade your server's firmware, remotely accessing your server's console by using your web browser, modify and see sensors, see hardware components and so on. You can access ILOM by using a serial console,ssh client or a web browser.You need to set an ip address for your ILOM manager if you need to use ssh or a web browser.If you don't have a dhcp server around you'll need to configure your ILOM manager with a static ip address. I have used a serial console connected on Thumpers "Ser MGT" port and on the other side a linux desktop and minicom application. You need to change your minicom settings to 9600 baud rate , 8 data bits, no parity and one stop bit (9600-8-N-1) with no hardware and software flow control.

Yes, that's right, It's Linux :) . Sun ILOM uses embeded Linux. When you connect to ILOM by a serial console you will need to login in by using ILOM's factory default login name "root" and password "changeme". ILOM has a simple built-in command interface. You can use "help" command for details when you logged in ILOM manager. Setting the static ip address of ILOM is simple. I captured the screenshot when I gave one of my local ip addresses to Thumper's ILOM.

After, you commit your static ip network settings settings with "set commitpending=true" command, you need to connect Thumper's "net mgt" ethernet interface to your network.Now, you can access ILOM by using any ssh client or a web browser. My plan is to use OpenSolaris and 48TB ZFS Raid-Z on this storage server and see the results on a production environment :)

getting detailed process information on Freebsd

2009-05-03T01:35:00.000-07:00

Freebsd procstat utility gives detailed information about all of the processes on the system or just for a given process id number, such as virtual memory mapping, thread stack, command line arguments and open files.

Running procstat with "-a" argument prints information like pid,pid,login, process name, wchan (which event the process waiting) of all processes on the system.

# procstat -a

Procstat "-c" option shows you the command line arguments of a process and "-f" option shows opened files by given process.Following sample output shows the command line arguments and the files currently opened with their permissions by vi process.

You can access virtual memory mapping information a process with "-v" option. Following sample shows vi process virtual memory mappings.

Finally, "-k" option shows kernel threads stacks details of given process.

watching interrupt usage on Freebsd

2009-05-03T00:18:00.000-07:00

You can easily see how many interrupts taken by each device on Linux by simply looking in /proc/interrupts file. Freebsd has no such information in procfs. But you can access the same information by using "vmstat -i" command in Freebsd.But you can't see which cpu handling which irq as shown by Linux. Following is a sample output of "vmstat -i" command from a Freebsd machine.

You can write a simple c shell script to watch interrupt usage like this.

Vmstat uses sysctl interface to gather interrupt usage information by using hw.intrnames and hw.intrcnt oid names. As names suggest intrnames holds all interrupt names and intrcnt holds their irq counts since system startup for each interrupt.

Linux NFS and "permission denied" mount problem

2009-04-24T03:56:00.001-07:00

Today, I have faced with following strange nfs mount problem.


[root@nfsclient ~]# mount /mymount
mount: XXX.XXX.XXX.XXX:/mymount failed, 
reason given by server: Permission denied

My nfs client is FC4 and nfs server is Centos 4.6.First, I have checked the server from nfs client side by using "showmount -e myserver" command. Exports was okay.After that, I decided to check rpc services on server with "rpcinfo -p" command.


[root@nfsclient ~]# rpcinfo -p myserver

  program vers proto   port
   100000    2   tcp    111  portmapper
   100000    2   udp    111  portmapper
   100024    1   udp    638  status
   100024    1   tcp    641  status
   100021    1   udp  32768  nlockmgr
   100021    3   udp  32768  nlockmgr
   100021    4   udp  32768  nlockmgr
   100021    1   tcp  32772  nlockmgr
   100021    3   tcp  32772  nlockmgr
   100021    4   tcp  32772  nlockmgr
   100011    1   udp    903  rquotad
   100011    2   udp    903  rquotad
   100011    1   tcp    906  rquotad
   100011    2   tcp    906  rquotad
   100003    2   udp   2049  nfs
   100003    3   udp   2049  nfs
   100003    4   udp   2049  nfs
   100003    2   tcp   2049  nfs
   100003    3   tcp   2049  nfs
   100003    4   tcp   2049  nfs
   100005    1   udp    925  mountd
   100005    1   tcp    928  mountd
   100005    2   udp    925  mountd
   100005    2   tcp    928  mountd
   100005    3   udp    925  mountd
   100005    3   tcp    928  mountd

Mandatory NFS services was working correctly as seen above rpc report.I didn't have a firewall between these machines or iptables on any of them.This was not the problem also here.I haven't check nfs client side because it was mounting other nfs server's mount points without a problem at all. So, I have focused on nfs server side. Restarting nfsd service or re-exporting nfs file systems with exportfs didn't work out. I can see nfs clients that mounted server's nfs mount by looking in /var/lib/nfs/rmtab file (showmount -a) on server. But I couldn't access /proc/fs/nfsd/ directory. It was not mounted on nfs server.So,I manually mounted it by using following mount command.


mount -t nfsd nodev /proc/fs/nfsd

After mounting /proc/fs/nfsd manually, I was able to mount the nfs server from client side again.

If you look at into /etc/modprobe.conf.dist file, you can see that when kernel installs module nfsd , it mounts /proc/fs/nfsd and /var/lib/nfs/rpc_pipefs mount points.


install nfsd /sbin/modprobe --first-time --ignore-install nfsd && 
{ /bin/mount -t nfsd nfsd /proc/fs/nfsd > /dev/null 2>&1 || :; }
install sunrpc /sbin/modprobe --first-time --ignore-install sunrpc && 
{ /bin/mount -t rpc_pipefs sunrpc /var/lib/nfs/rpc_pipefs > 
/dev/null 2>&1 || :; }

Process IO Top utility with Solaris DTrace

2009-04-14T15:18:00.000-07:00

DTrace is dynamic tracing framework created by Sun Microsystems for Solaris OS. DTrace let's any system administrator or developer to get internal view of how an operating system works, find any system bottlenecks, examine a live system or process, get information on system calls, network internals and many other things. For this purpose DTrace has many providers for specific tasks. You can get list of the DTrace providers and probe function names by running following command:

# dtrace -l

DTrace also adopted to other operating systems like FreeBSD and Mac OS X. DTrace uses D Language which is subset of C programming language. You can find more information about DTrace on Sun's BigAdmin Portal.

Following example demostrates you, how easy to find the information you want on Solaris with DTrace.I wrote it to see which files are most written or read by which processes on the Solaris.


#!/usr/bin/bash
# process_io_top -  show procesess by top read/write KB I/O per file
#      Written by Levent Serinol (lserinol@gmail.com)
#      http://lserinol.blogspot.com
# Apr/14/2009
#
# USAGE: process_io_top [-s interval] [-p pid]
#   
#  -s interval  # gather and show statistics in given interval (seconds)
#  -p pid  # show read/write KB I/O just for given PID
#  -h   # show usage information
#
# eg:
# process_io_top -s 10 
#
#
#
####################################################################################
interval=5
show_pid=0
pid=0

function usage()
{
echo "
USAGE: io.sh [-s interval] [-p pid]
         -s             # set interval, default is 5 seconds
         -p pid         # pid
  eg, 
         io -p 630                # io activity of pid 630
         io -s 10                 # refresh output in every 10 seconds";
}

while getopts h:p:s:a name
do
        case $name in
        p)      show_pid=1; pid=$OPTARG ;;
        s)      interval=$OPTARG ;;
        h|?)    usage;
               exit 1
        esac
done

/usr/sbin/dtrace -Cws <( cat <<EOF

 inline int PID  = $pid;
 inline int SHOW_PID    = $show_pid;
 inline int INTERVAL    = $interval;


#pragma D option quiet
#pragma D option aggsortrev


dtrace:::BEGIN 
{
  secs = INTERVAL;
  printf("Please wait....\n");
}

io:::start 
/ (SHOW_PID == 1) && ( pid == PID) /
{
 self->rw=args[0]->b_flags & B_READ ? "R" : "W";
 @files[pid,execname,self->rw,args[2]->fi_pathname] = sum (args[0]->b_bcount);
 @total_blks[self->rw]=count();
 @total_bytes[self->rw]=sum (args[0]->b_bcount);
 self->rw=0;
}

io:::start 
/ SHOW_PID == 0 /
/* SHOW_PID == 0 && args[2]->fi_pathname != "<none>" */
{
 self->rw=args[0]->b_flags & B_READ ? "R" : "W";
 @files[pid,execname,self->rw,args[2]->fi_pathname] = sum (args[0]->b_bcount);
 @total_blks[self->rw]=count();
 @total_bytes[self->rw]=sum (args[0]->b_bcount);
 self->rw=0;
}

profile:::tick-1s
{
        secs--;
}


profile:::tick-1s
/secs == 0/
{

 trunc(@files,30);
 normalize(@files,1024);
 system("/usr/bin/clear"); 
 printf("%Y ",walltimestamp);
 printa("%s %@11d blocks, ",@total_blks);
 printa("%s %@11d bytes, ",@total_bytes);
 printf("\n%6s %-12s %3s %8s %3s\n", "PID", "CMD","R/W", "KB", "FILE");
 printa("%6d %-12.12s %1s %@10d %s\n",@files);
 secs = INTERVAL;

}
dtrace:::END
{
        trunc(@files);
}
EOF
)

You can download it here process_io_top. This scripts briefly shows you how to pass arguments from shell to DTrace. Using aggregations and sort them in reverse order by using pragma "aggsortrev". Defining probes (io:::start) more than once with different predicate conditions. Following is a sample output from process_io_top script.

speeding up your nginx server with memcached

2009-03-30T23:51:00.000-07:00

Nginx is a high performance web and proxy (web and mail proxy) server. Generally, nginx is used as a front-end proxy server to Apache webserver. Nginx is known to be slow while serving dynamic pages like php. Normally, nginx is using fast-cgi method which is slow. Therefore, it's a good idea to run Apache as back-end server to Nginx and serve dynamic php pages from Apache. If your website's php pages suitable to cache for a certain time, you can use Nginx proxy module and proxy_store command to cache Apache served php pages output in Nginx automatically as html. Here, I'll give you instructions how to use Nginx's memcache module and Danga Software's memcached deamon to store your content in memory and serve it. Serving content from memory will be faster than serving it from disk. memcached's default listening port is 11211. You can find instructions on Danga Software's website how to compile and run memcached.

Now, we can look our Nginx configuration for memcache implementation. Let's suppose we have two Apache webservers running on two different physical servers. IP addresses of the Apache webservers are 192.168.2.3 and 192.168.2.4. We'll use those Apache webservers as back-end servers. We have a Nginx server as front-end to them on 192.168.2.1 ip adress. First of all, we have to tell Nginx about those back-end servers. We use Nginx upstream module for this purpose. As you can see below, we defined a upstream named "backend". The configuration has our two Apache webservers ip addresses. Upstream module let's you also give weight to each server in configuration. Our first server's hardware configuration is better than the second one, so we gave the first one weight value 2. This configuration should be in http section of Nginx configration file (nginx.conf).


upstream backend {
     server 192.168.2.3 weight=2;
     server 192.168.2.4;
}

We have created our upstream configuration. Now, we have to tell Nginx, which files will be server by memcache module. I have decided to only serve some image types by memcache. The following configuration part should be in server section of Nginx configuration. The "location" directive tell's the nginx to handle every file which ends with given extensions like .jpg,.png and .gif in url. As first step, Nginx will check the url in memcached. Memcached is simple key value memory database. Every row has a unique key.In our case the key is our url. If Nginx, finds the key (url) in memcached, it will get contents of the key from mecached and send it back to client. This operation is running completely from memory. In case that the key (url) not found, it will fallback to 404 and as you can see, we catch 404 error and send request to our back-end Apache servers. Nginx will then send Apache's response to client.


location ~* \.(jpg|png|gif)$ {
access_log   off;
expires      max;
add_header   Last-Modified "Thu, 26 Mar 2000 17:35:45 GMT";
set $memcached_key $uri;
memcached_pass     127.0.0.1:11211;
error_page         404 = /fetch;
}

location /fetch {
internal;
access_log   off;
expires      max;
add_header   Last-Modified "Thu, 26 Mar 2000 17:35:45 GMT";
proxy_pass http://backend;
break;
}

Of course, we have a drawback here. Nginx's memcache module never put anything automatically in memcached. You have to store your information in it manually by using something like a script. Considering our example, if we forget to store information about a file in memcached, it will be always served by back-end Apache servers. Here is a simple php script, which finds given image types and deploy it into memcached for Nginx.


<?php

function rscandir($base='', &$data=array()) {
$array = array_diff(scandir($base), array('.', '..'));

foreach($array as $value) :
  if (is_dir($base.$value)) {
    $data = rscandir($base.$value.'/', $data);

  }
  elseif (is_file($base.$value)) {
   $rest = substr($value, -4);
   if ((!strcmp($rest,'.jpg')) || (!strcmp($rest,'.png'))
                                || (!strcmp($rest,'.gif')) ){
         $data[] = $base.$value;
   }
 }

endforeach;
return $data;
}

$mylist=rscandir("/var/www/mysite");

$srch = array('/var/www/mysite');
$newval = array('');

$memcache_obj = memcache_connect("192.168.2.1", 11211);

while (list($key, $val) = each($mylist)) {
  $url=str_replace($srch,$newval,$val);
  echo "$key => $val -> ".filesize($val)."\n";
  $value = file_get_contents($val);
  memcache_add($memcache_obj, $url, $value, false, 0);
}
?>

You need to run this script one time, it will find all given image types and store them into memcached. I run this on one of the Apache back-end servers. It will store data into memcached. This memcached is located on Nginx server which ip address is 192.168.2.1 .

Hammer Filesystem

2009-03-22T03:31:00.000-07:00

On Feb 17 2009, new release 2.2 of DragonFly BSD released which includes a new filesystem called hammer. Hammer is a new filesystem which intented to replace ffs on DragonFly systems. Hammer filesystem has many advanced futures compared to ffs filesystem like

instant crash recovery
large file systems and multivolume support
data integrity checking
history and snapshots
reblocking (on the fly defragmentation)
mirroring

Let's start by creating a new hammer filesystem on a new disk device by using hammer's newfs_hammer command. As you can see below, we used the '-L' parameter. Every hammer filesystem requires a label name and it's mandatory.

Let's mount in under /mnt directory by using "noatime" option. mount_hammer command must be used for this operation.

Looking to df output you can see our new hammer filesystem labeled as "mydata". Now, let's see where hammer filesystem differs from ffs.

History feature

History metadata of a hammer filesystem is written on every 30 seconds. You can use hammer utility to access your files history data. Usage is simple as, calling hammer utility with history command and given your file name which you want to see whole history.

# hammer history /mnt/test.txt

You may be noticed that I have used "sync" command, because I didn't want to wait the the kernel sync operation occur (30 seconds) and forced it by using sync comand to get history result fast.

Snapshot feature

Now let's see hammer's snapshot feature. I have created a directory called "/snapshots" where I want to keep all of my hammer filesystem snaphosts. We'll use hammer utility for taking snapshots.

# hammer snapshot /mnt /snapshots/mnt_snapshot1

I have created a file on /mnt filesystem and took a snapshot. After a sucsessful snapshot, I deleted the original file from /mnt and accessed it by using the snapshot I took before deleting the file (/snapshots/mnt_snapshot1/test.txt).

Undo Feature

You can see and rollback every change between the taken snapshots and original files by using hammer's undo command. You can use "-d" option to get diff output to see every changes on the given file line by line between every snapshot and original one. Also, "-a" option gives you whole change history of given file. Here, I created a test file called "myfile.txt" under my new hammer filesystem (/mnt) and then took a snapshot. After taking snapshot, I added one more line to my test file and used "undo -d /mnt/myfile.txt" to get diff output and see what changes made the file between snapshots.

Mutivolume Feature

As I mentioned above, you can use multi volumes and create a big disk volume with hammer filesystem. Now, we'll create a single hammer volume by using two disks (ad1 and ad3).
You just call newfs_hammer and mount_hammer commands by only giving two or more disk device names instead of one.Commands are straight forward. First, we need to create new hammer filesystem on disks.

# newfs_hammer -L big /dev/ad1 /dev/ad3

I have used to disks and each size is 8GB. After creating new hammer filesystems on disks , we can mount them as one big volume. (I know 8GB is not big :) ).

# mount_hammer /dev/ad1 /dev/ad3 /mnt

You can see on /mnt mount point we have a disk in size 16GB, total of two 8GB disks.

Mirror Feature

Hammer uses Pseudo File Systems (PFS) to duplicate inode numbers to slaves. Therefore, mount_null command required for this operation. This duplication is triggered by using hammer utlity with mirror-copy parameter. First we need to create two pfs, called for master and slave. I created them as /mymaster and /myslave. Master volume must be created with pfs-master and slave with pfs-slave parameter by using hammer utility. One thing to take in accounting here is that slave pfs must use master's shared-uuid number.

After creating mymaster and myslave pfs we have to use mirror-copy to start initial mirroring operation by using these pfs links , because mount_null can't access them without this copy operation.

After, running first mirror copy operation on our pfs links, we can mount our original hamme volumes where we want and associate them with the pfs links we created and sync (mirror-copy) our mounted hammer volumes.

You can use mount_null mount points to do your filesystems operations. You can see that slave volume (myslave) is read-only and you don't have write access to it. You can only use master volume (mymaster) to make your write operations.If you're looking an alternative filesystem to ffs on BSD systems, I suggest you to evaluate both ZFS on FreeBSD and Hammer on DragonFly BSD.

xfs filesystem fragmentation

2009-03-16T12:52:00.000-07:00

Many filesystems use many techniques to overcome filesystem fragmentation and XFS filesystem is one of them. But in long usage, you may face with fragmentation problems.Here, I will show you how to find fragmentation ratio in your xfs filesystem and how to re-organize your xfs filesystem with xfs_fsr utility to reduce fragmentation on a live xfs filesystem. First, let's check our xfs filesystem's fragmenation ratio using xfs_db (xfs debugger) utility. As shown below, we used two parameters. First one is "-r", this means that we use debugger in read-only mode. We do not want to accidently do anything bad to our filesystem. Also, this option is useful if we are debugging a mounted filesystem. Second option is "-c" command, which simply takes it's parameter as xfs_db command and run it. Without "-c" parameter, xfs_db will run it's interactive shell where you can run xfs_db commands.

We see that our xfs filesystem on /dev/sdb device has 0.14% fragmentation. This is very low fragmentation. We don't need to defrag it. But here how to do it with xfs_fsr. This utility re-organizes a xfs filesystem while it's mounted file by file. As you can see in sample below, it runs with "-t" parameter, this tells xfs_fsr utility to run maximum that given seconds and quit.

There is one more option you can use while using xfs filesystem to avoid future fragmentations. It's called "allocsize", where it preallocates disk space before writing to a file. You have to set this preallocation size while mounting your xfs filesystem. This will prevent fragmentation on your xfs filesystem if you give a reasonable size depending on your average file sizes.

mount -t xfs -o noatime,allocsize=8M /dev/sdb /mydata

you can also set this option in your /etc/fstab.

stopping and resuming a process run in Linux

2009-03-11T15:35:00.000-07:00

You can stop (pause) a running process by sending "-STOP" signal via kill command in Linux. You can re-run same stopped process again by sending "-CONT" signal to it. Assume that we have a process to stop and it's pid is 8674. Following, command will stop the running process id 8674.

# kill -STOP 8674

Now the process is stopped, to resume it's running state, you have to send "-CONT" signal.

# kill -CONT 8674

In Solaris, you can do the same task by using pstop and prun utilities or you can use kill command and use signals "SIGSTOP" and "SIGCONT".

measuring your network throughput on Solaris

2009-03-07T13:25:00.000-08:00

You can use nicstat utility to measure your network device utilization and throughput. The utility itself uses solaris kstat for network device statistics information. Statistics include network read and write in kilobytes, packets and device utilization.You can compile nicstat with gcc as shown below.

# gcc nicstat.c -o nicstat -lkstat -lgen -lsocket

Following image shows a sample output from nicstat utility.

getting network interface statistics on Solaris

2009-03-07T12:13:00.000-08:00

There are many ways to get network interface status and statistics on Solaris. I'll show some of them. First one is well known command netstat. netstat's "-i" parameter will give you statistics of devices used for ip traffic. Statistics include input/output total packets,errors and collisions for each network device.

Next method is by using solaris ndd command. First, we have to find our device name. This can be done by ifconfig command.
My network device name is "e1000g0" as seen in sample output above. Now, it's time to ask ndd which parameters are supported by this device. We do this by calling ndd (ndd /dev/e1000g0 \?) with our device name and "\?" parameter to display every supported parameter.

ndd let's you change some interface settings. They're marked as "read and write". The parameter names clearly shows what they are for. As you can see in sample output below, ndd can give us link speed, link status (cable plugged/unplugged),duplex mode and autoneg status. Following sample output shows us that my network interface link status is 1, which means the network cable is plugged,Network speed is 1000mbit and auto negotiation is on.
The other command is called dladm. dladm can give us above results with one command and in easily readable format. Also, it can give network interface statistics like netstat with "-s" parameter except collisions.

iptables and connection limit

2009-02-28T23:32:00.000-08:00

Linux iptables uses a module called ip_conntrack. The name of the module exactly says what it does. It tracks every connection attempt (every connection state) in a hashed table.Every hash entry contains a linked list. Each linked listed entry is called a bucket presenting a connection. You can see your machine's current connections by using ip_conntrack module's proc entry.

Of course, this table has a default value. It is likely that you will reach this limit on a machine getting high connection rates. First clue will be the following log message on your system logs (/var/log/messages).

ip_conntrack: table full, dropping packet

In this case, your machine (iptables) will drop every incoming connection because there is no free hash entry to store any incoming connection (udp or tcp). In this case, there is a configuration option to bump this connection limit to higher values. It's called ip_conntrack_max. It's in /proc filesystem (/proc/sys/net/ipv4/ip_conntrack_max).

You can easily increase this number by simply echo 'ing new number in to this file.

This conntrack tables is divided in linked list entries by hash number as I mention above. The calculation is CONNTRACK_MAX/HASHSIZE. Every hash contains linked lists for connections.This parameter is locted in /sys/module/ip_conntrack/parameters/hashsize. In our case it's default value is 4096. You remember that before changing our default value of conntrack_max, it was 32768. The default hashsize 4096 is produced by 32768/8 calculation. You can tweak this number on your system to increase performance of iptables. Don't forget to put your settings in /etc/sysctl.conf. Because overriding this /proc/ and /sys/ values is only temporary change. There will be gone when you reboot your machine. The hashsize parameter in /sys/ is called buckets in sysctl conf configuration. Also, check other conntrack parameters via "sysctl net.ipv4.netfilter". By setting some other options like wait time can reduce your entries in conntrack table.

Solaris Interrupts

2009-02-21T01:35:00.000-08:00

In this post, I'll be shown you how to access irq table on a Solaris system and how to get statistics about the number of interrupts produced on Solaris. We'll use two commands intrstat and mdb.
intrstat command shows how many interrupts produced by each device and how many of them handled by each cpu on the system. "%tim" column shows the time where spend by that driver's interrupt handler on processing cpu. You can pass a parameter to intrstat and tell it to gather information in given intervals (in seconds). Following is a sample output of intrstat where it gets statistics in 5 seconds interval.

Now, after getting interrupt statistics we can now examine the irq table on Solaris system. For this job, we'll use solaris mdb (Solaris modular debugger). We can access irq table through the kernel. So, we'll pass the "-k" parameter to mdb. This will tell it to get in kernel debugging mode. mdb will use /dev/ksyms and /dev/kmem to access Solaris kernel. After running "mdb -k", type "::interrupts" command to acess irq table in Solaris system. Here is a sample output:

This table shows us which driver(s) (ISR) using which irq. Share column shows how many devices sharing the same irq. IPL gives irq priority. Type column show irq type used Fixed (legacy), MSI, IPI interprocessor interrupt (xcalls - cross call).

disk error statistics with Solaris

2009-02-21T01:11:00.000-08:00

Solaris iostat command normally shows you disk activity statistics and cpu utilization information. But it has another useful option which shows you disk error statistics. Those parameters are "-e" and "-E"."-e" options show summary disk error statistics for soft, hard and trasnportation errors. The other option "-E" shows all errors on the disks and information about the devices like the vendor,model,size and media erros.

Stopping and Running process on the fly with Solaris/OpenSolaris

2009-02-17T05:19:00.001-08:00

Solaris/OpenSolaris has two utilities called pstop and prun which has ability to stop a running process and run it again. Both of the ulities takes pid or pid/lwp (light weight process - thread) as parameter. As the name says, pstop stops a running process when given an pid of a process. Given the same process id of a stopped process to prun , makes the process runnable again.

Booting OpenSolaris in verbose mode

2009-02-17T00:19:00.000-08:00

When OpenSolaris boots it does not show kernel messages on the console by default. You can see them by passing an argument to kernel when booting. OpenSolaris uses grub boot loader. On the grub menu press "e" to enter edit mode.

Move to the line where kernel parameters are written with cursor keys.

Press "e" again for edit mode for the grub. Than add "-v" parameter to the end of kernel parameters.

After adding "-v" verbose parameter to kernel press "enter" and than "b" to boot OpenSolaris with verbose mode enabled.

If you need this option to be permanent, you need to modify grub config file in OpenSolaris. It's located in /boot/grub/menu.lst . If you use zfs root, than it's located in /rpool/boot/grub/menu.lst.You can do this on a Sun Sparc machine with Solaris when the openboot procedure begins by pressing "STOP-A" combination and type "boot -v"

IRQ affinity in Linux

2009-02-13T05:26:00.000-08:00

Some hardware components like ethernet cards,disk controllers,etc. produces interrupts when needs to get attention from cpu. For example, when ethernet card receives packet from network. You can examine your machine's interrupts usage on cpu's by looking in /proc/interrupts proc entry.

# cat /proc/interrupts

This information includes which devices are working on which irq and how many interrupts processed by each cpu for this device.In normal cases, you will not face with a problem and no need to change irq handling process for any cpu.But in some cases, for example if you are running a linux box as firewall which has high incoming or outgoing traffic, this can cause problems. Suppose that you have 2 ethernet cards on your firewall and both ethernet cards handling many packets. In some cases you can see high cpu usage on one of your cpus.This can be caused by many interrupts produced by your network cards. You can check this by looking in /proc/interrupts and see if that cpu is handling interrupts of both cards. If this is the case, what you can do is looking for most idle cpus on your system and specify those ethernet card irqs to be served by each cpu seperately. But beware that you can only do this in a system with IO-APIC enabled device drivers. You can check if your device supports IO-APIC by looking /proc/interrupts.

In ethernet card case, there is an implementation called NAPI which can reduce interrupt usage on incoming network traffic.

You can see which irq is served by which cpu or cpus by looking in /proc/irq directory. Directory layout is very simple. For every used irq in system it is presented by a directory by it's irq number.Every directory contains a file called smp_affinity where you can set cpu settings. File content shows currently which cpu is serving this irq. The value is in hex format. Calculation is shown below.As you can see in example figure, eth0 is on irq 25 and eth1 is in irq 26. Let's say we want to set irq25 to be served only by cpu3. First of all, we have to calculate the value for cpu3 in hex value. Calculation is shown below.

Calculation

            Binary       Hex
 CPU 0    0001         1
 CPU 1    0010         2
 CPU 2    0100         4
+ CPU 3    1000         8
 -----------------------
 both     1111         f

Calculation is shown for 4 cpu system for simplicity. normally the value for all cpus on a system is represented by 8 digit hex value.As you can see in binary format every bit represents a cpu. We see that binary representation of cpu3 is 8 in hex. Then we write it into smp_affinity file for irq 25 as show below.

# echo 8 > /proc/irq/25/smp_affinity

You can check the setting by looking in to file content.

# cat /proc/irq/25/smp_affinity
00000008

Another example, let's say we want irq25 to handled by cpu0 and cpu1.

   CPU 0    0001         1
+  CPU 1    0010         2
--------------------------
                              0011                        3

Setting bit for cpu0 and cpu1 is giving us the value 3.For example if we need all cpus to handle our device's irq, we set every bit in our calculation and write it into smp_affinity as hex value which is F. There is an implementation in Linux called irqbalance , where a daemon distributes interrupts automatically for every cpu in the system. But in some cases this is giving bad performance where you need to stop the service and do it manually by yourself as I described above for higher performance.Also, irqbalance configuration let's you to configure where it will not balance given irqs or use specified cpus. In this case you can configure it to not touch your manually configured irqs and your preferred cpus and let it run to automatically load balance rest of the irqs and cpus for you.

arrays in bash shell - Part 2

2009-02-10T05:54:00.000-08:00

In my previous post, I explained basic arrays usage in bash shell. Now, I'll show more advanced topics with bash arrays.Let's create a simple array called myarray.

# myarray=(first second third fourth)

Now, assume that you want to learn string length of second element of myarray.

# echo ${#myarray[1]}

this will 6 for string "second".Note here that, array indexes are starting from 0. So, 1 index numbers actually is 2nd element of the array.

Here is another example, this one will skip first element of the array and gives rest of the array.So,following will do the trick.

# echo ${myarray[@]:1}

going further with this example.Let's say we want to skip first element and need no more than 2 elements in the array

# echo ${myarray[@]:1:2}

result will be "second third".

Search and Replace

You can search and replace patterns or characters in given array or any array element.
Here is two examples, first one will search all elements of array for character "r" and replace all characters with "R". Second example, will do the same thing but only for 2nd element of the given array.

# echo ${myarray[@]/r/R}

# echo ${myarray[1]/r/R}

Above examples, will only replaces first occurrences, for replacing whole occurrences use double slashes.For example,

# echo ${myarray[1]//r/R}

Deleting and Adding Array Elements

Deleting any element is very simple. You just have to use "unset" command and give index number of array element.Following, will delete 2nd element of the array.

# unset myarray[1]

now our example array will contain "first third fourth" elements.Deleting the whole array is very simple.Don't pass the index number to unset.

# unset myarray

Now, Let's a new entry to our array.

# myarray=("${myarray[@]}" "fifth_element")

substring operation with bash

2009-02-07T04:15:00.000-08:00

You can get a substring (substr) value of an string variable as following in bash.
Syntax is simple ${VARIABLE_NAME:start:length}

let's assume we have a string variable called mystring and it has the value "test".

mystring="test"

If you want to access first character of the string you can use following statement.
c=${mystring:0:1}

now c value contains "t" character.

c=${mystring:2:2}

Now c contains "st" substring.

Hint: You can calculate the length of a string value as follows.

${#mystring}

Mysql General Query Log

2009-02-06T11:24:00.000-08:00

Before Mysql version 5.1.6, you can only log to files what mysqld was doing. 5.1.6 comes with an option where it let's you keep this log in a table called mysql.general_log. Even further, version 5.1.12 came with a new ability to let you disable general query logging on the fly.

You can enable/disable mysql general query logging by setting global variable "general_log".For example following command, starts general logging on the fly.

SET GLOBAL general_log = 'ON';

You have to options, either you can log the queries on in a file where it was the only option before version 5.1.6 , or you can log in the table I mentioned above called mysql.general_log by using global variable "log_output".
The following statement tells mysql to log queries in table mysql.general_log instead of a file.

set global log_output='table';

In any time, you can truncate the table to delete it's contents.

After logging the queries, you can see them by simply using a select statement.

select * from mysql.general_log;

to make it more readable you can use \G at the end of the query.
select * from mysql.general_log\G;

arrays in bash shell

2009-02-06T02:14:00.000-08:00

BASH shell is very powerful script language. In my previous blog posts, I give some detail about bash regular expression support. In this post, I will give some information about bash array support.
Declaring an array is very simple in bash. Here is some examples.

# declared an array named called "array" and containing elements.
# "This", "is", "an", "array"
array=(This is an array)

# declaring an array called a by giving elements one by one.
a[0]="123"
a[1]="abc"
a[2]="xyz"

# get output of an df -h

myarray=($(df -h))

Reading value element of an array is in following syntax ${array[#index_number]}. Please note the curly bracket notation.

# print element number 2 of array called myarray
echo ${myarray[2]}

# print all elements
echo ${myarray[*]}

# print index numbers of array
echo ${!myarray[*]}

# number of elements in array
echo ${#myarray[*]}

Here is a simple script using arrays. It simply reports the file systems where disk usage is
over 90% percent.

#!/bin/bash

(df -h|grep "^/") | while read d; do
array=( $d )
array[4]=${array[4]%\%}
if [ ${array[4]} -gt 90 ]; then
echo "${array[5]} is over limit. current size is ${array[4]}%"
#do whatever you want here, for example send an email to warn sysadmin about high disk usage.
fi
done

Solaris pargs command

2009-02-03T10:19:00.000-08:00

Solaris pargs gives you environment and arguments information for any given process id. If you pass just a process id (pid) to pargs, it gives you all argument variables of the running process. If you need to have arguments as a single line for using it with for example on shell use -l argument with it. Also pargs can give you environment information of any running process if you run it with -e parameter.
Sometimes this utility can be handy if you cannot see the parameters or environment variables of a process with other methods like ps or any other well known utilities.

pargs

FreeBSD kernel profiling with kgmon

2009-01-30T07:34:00.000-08:00

Sometimes, it's not possible to find a system bottleneck by profiling and debugging user space processess. You have to profile the kernel to pinpoint the perfomance bottleneck. Here, I will show how to configure a freebsd system for kernel profiling and using kgmon utility to gather kernel profiling data. kgmon produces gprof compatible output. The default output file is called gmon.out .

First of all, we need a profiling enabled kernel configured and build. First step is to use config utility. Now, we need to run config utility from our kernel source configuration directory. I'm using a i386 system. The conf file directory depends on your architecture.
The default directory is /usr/src/sys/ARCH/conf. In my case it's /usr/src/sys/i386/conf/.
We run config utility from where our kernel conf file resides with parameter -p. This tell's the compiler to compile our kernel for low resolution profiling. If we need a high resolution profiling we need to run config with -pp parameter instead of -p. config utility needs a second parameter called SYSTEM_NAME. This is our default system name. FreeBSD's default system name is GENERIC. This can be different on your system, if you build a new kernel with different config file. Older versions of freebsd default GENERIC kernels was not using smp. So it's possible to you have build a new kernel for smp.Therefore you can have a custom build kernel on your system or for another reason. You can see your default system name by using "uname -a" command.
Here is mine is called TESTKERNEL. You'll probably see GENERIC in your case.

After learning you SYSTEM NAME, it's now time to configure our kernel source for profiling. The following picture shows you how to run config with -p option on i386 arch with GENERIC system name.

After running "make cleandepend && depend" command, we run the "make" command on the same directory and it will built a profiling enabled kernel for us.

After the make command, we have build our kernel, now run "make install.debug" command and it will install our debug kernel.

Now, it's time to reboot our machine. After rebooting check kernel messages for following message to see if build is successful for profiling enabled kernel Addresses should be different but seeing message "Profiling kernel" means it's ok.

Profiling kernel, textsize=6845824 [c045cbe0..c0ae4160]

It's time to run kgmon to collect profiling data on our kernel.

# kgmon -b

Now kernel profiling is running for low profiling. For high profiling -B option must be used and don't forget you have to use config -pp for high profiling.
You can now, run the application which was causing the problems on your system to profile it, or create the same situation on server where you were having problems.
Now, after creating and seeing the problem, let's kgmon to dump the profiled kernel data.

# kgmon -p

As I said before, kgmon will create a file called gmon.out for profiled data in gprof format.

Now, we gathered our data, we can now stop profiling.

# kgmon -h

We collected the data we need. Let's see what the gmon.out file contains.

# gprof /boot/kernel/kernel gmon.out

Here is a snippet from gprof output.

I'll explain the meaning of the gprof output in another post.

NOTE: you can apply this technique to other freebsd deriatives like dragonfly bsd and netbsd.

simple tcp server with djb tcpserver

2009-01-29T13:00:00.000-08:00

DJB ucspi-tcp package has a program called tcpserver. Simply it accepts incoming tcp connections for a given port and run specified program when a connection established on that port. The good part is that you can write a program in any language which reads stdin and writes data to stdout file descriptors. In this way, you can read data coming from tcp socket simply by stdin file descriptor and send data to tcp client side by writing desired data on stdout file descriptor.

Here is a simple perl code snippet which reads stdin and writes data to stdout.

#!/usr/bin/perl

$| = 1;
print "Hello !\r\n";

while ($line=<STDIN>) {

print "Your input is:".$line."\r\n";

}

Let's call this file as simple.pl
Here is the tcpserver command to run this program on port 9090. tcpserver has other parameters such as given uid and gid to running process. Please see tcpserver man page for details.

# tcpserver -vRH -l test 0 9090 simple.pl

now, you can telnet your server's 9090 port from another machine and test it.

lsof alternatives on FreeBSD

2009-01-29T00:43:00.000-08:00

lsof is a utility which gives information about open sockets/files/pipes on many unix systems. You can easily install lsof and try it on your freebsd installation by using pkg_add command.

# pkg_add -r lsof

But FreeBSD has two utilities coming bundled with default installation. They are called fstat and sockstat.

fstat tells you which user,command and pid opened the file, which mount point the file is and information about the open file descriptor like read/write, inode number and mount point of the opened file.Please see man page of fstat for other options.

sockstat gives you information about the opened sockets like which process/command is using it, user of the process,pid,protocol like tcp/udp,stream,dgram,etc.. and connected ports of local and remote servers.

Freebsd kernel process tracing

2009-01-24T08:22:00.000-08:00

ktrace utility enables to trace and log kernel system calls made by process. By default, it logs to ktrace.out file but this can be overwritten by providing another log filename with -f parameter. You need to pass your command to ktrace or use the pid of a running process.
Also you have to say the kernel what system calls to trace by -t parameter. -t parameter has following options:

c trace system calls
n trace namei translations
i trace I/O
s trace signal processing
t trace various structures
u userland traces
w context switches
+ trace the default set of trace points - c, n, i, s, t, u

While tracing is going in kernel , logging stops when the process stops execution or trace popint ends. The other way is to use -c parameter of the ktrace and provide the pid of the process to stop tracing any further.

to trace and log any running processes simply use the -p parameter and pass the process number (pid) to ktrace. here is a simple example with ktrace to trace find command:

# ktrace -t+w /usr/bin/find /

and following is a short snippet from the log created by ktrace and dispayled using kdump utility

# kdump -f ktrace.out

The log file created by ktrace can be read with kdump utility. Simply pass your ktrace log filename to kdump with -f parameter.

freebsd network tuning

2009-01-22T01:46:00.000-08:00

Few days ago, I couldn't reach on of my freebsd 7.1 servers via ssh. Machine was not giving any response to any packets on the network. This is a server which gets moderate network traffic created by short and long lived network connections. After log in from console, I ran the "vmstat -z" command.
Looking closely to vmstat -z output, I figured out that some kernel zone allocations failed for following zones:
mbuf_cluster: 2048, 25600, 1278, 24322, 393553280, 1384
tcptw: 52, 5184, 0, 5184, 3348441, 1304539
tcpreass: 20, 1690, 0, 1690, 10759020, 503124

simply,
tcptw -> tcp timewait
tcpreass -> tcp reassembly
mbuf_cluser -> network buffer data stored by freebsd kernel

I decided to bump the default numbers. I have used /etc/sysctl.conf to increase
net.inet.tcp.maxtcptw = 12000
kern.ipc.nmbclusters = 32768

numbers. then run /etc/rc.d/sysctl restart. you can run vmstat -z to see if the numbers are in effect. tcpreass is a bit different. It should be written in /boot/loader.conf file and will take effect after rebooting your machine.

net.inet.tcp.reass.maxsegments = 4096

After rebooting my machine I checked the results but I didn't get tcpreass numbers as I wrote to boot loader.conf file. I decided to look what happened. I examined the freebsd 7.1 kernel sources and see tcp_reass_init() has an EVENTHANDLER_REGISTER which calls tcp_reass_zone_change() function when nmbclusters numbers changed by sysctl. The rule is simple, when the machine boots
tcp_reass_init functions calculates tcpreass default value as nmbclusters / 16. But if you add net.inet.tcp.reass.maxsegments to your boot loader it skips the auto calculated default and gets the number you've given. Then as default register a event handler to watch nmbclusters changes by users for auto calculate tcpreass again.In my case it gets the given number but when systcl gets into the account kernel detects that nmbclusters changed and recalculates it. So, I decied to remove my parameter from boot loader.conf and instead of that increase the mbuf cluster number to a little bit higer via sysctl.conf. The reasons here is that tcpreass queue uses mbufs, therefore it's auto tuned by freebsd kernel to not run out of mbuf clusters on system.

NOTE: put this accounting that each mbuf cluster allocates 2KB in memory.

profiling performance of an erlang application with eprof

2009-01-09T03:47:00.000-08:00

Sometimes it's hard to pinpoint performance problems. In this cases profiling tools come handy. One of the profiling tools in Erlang is called eprof. Here is a simple example how to profile your erlang application with eprof.
First of all, you need to find which pid (module) to profile. You can use "nregs()." command to find your application's pid number in erl shell. In my case pid number is "<0.34.0>".
After finding the pid number start eprof module:

eprof:start().

now tell eprof to start profiling your application by giving the pid number.

eprof:start_profiling([pid(0,34,0)]).

note here that I have changed the dot in pid number to commas.

now wait sometime for eprof to collect some data or did something where you think the application was responding slow or stalling the cpu or whatever.

now stop profiling.

eprof:stop_profiling().

ok now we can see which functions was occupying our cpu with following command:

eprof:total_analyse().

Here is some screen output for total_analyse from my case:
6> eprof:total_analyse().
FUNCTION CALLS TIME
file:file_name_1/1 3450 13 %
erlang:port_control/3 250 7 %
prim_inet:enc_opts/2 300 5 %
erlang:port_command/2 50 4 %
erlang:port_close/1 50 4 %
prim_inet:enc_opt_val/4 300 3 %
gen:wait_resp_mon/3 200 3 %
filename:do_flatten/2 650 3 %
gen:do_call/4 200 2 %
prim_inet:dec_opt_val/3 300 2 %

if you like to see results for process by process use "analyse()" instead of "total_analyse()"
Also,It's possible to log these results to file by eprof's "log" function.

Caching filesystems on linux with fs-cache (cachefilesd)

2009-01-01T05:51:00.000-08:00

fs-cache can be use to cache nfs,local or other network filesystems. Here is an example howto cache a nfs mount point on Centos linux distribution. First of make sure that cachefilesd package is installed. Then make sure that cachefilesd (/etc/init.d/cachefilesd) runs on every boot. You can set it to run on startup by chkconfig utility if you like.
Cachefilesd has a config file named /etc/cachefilesd.conf, where you can set where the cache content will be written. The option is called "dir".

eg. dir /cache

Note: caching directory should be an ext3 filesystem.

Let's assume that we have a nfs mount point called mydisk on server 192.168.0.2. Let's create an entry in our fstab for this mount point

192.168.0.2:/mydisk /mydisk nfs tcp,fsc 0 0

the only option here is "fsc" for letting the caching start. After using check your cache directory in this case our directory is /cache to see it's working. There will be 2 directories there called graveyard and cache. Here is some cache filenames created on my cache directory.

./cache/@4a/I03nfs/@4f/Jk0M00wg00000000000000Y--knmon200/@08
./cache/@4a/I03nfs/@4f/Jk0M00wg00000000000000Y--knmon200/@08/Eo0g000M00E60180000gA5Ac0gmgO00FY0100
./cache/@4a/I03nfs/@4f/Jk0M00wg00000000000000Y--knmon200/@10
./cache/@4a/I03nfs/@4f/Jk0M00wg00000000000000Y--knmon200/@10/Eo0g000M00E601800000D5Ac0cmgO04FY0100
./cache/@4a/I03nfs/@4f/Jk0M00wg00000000000000Y--knmon200/@77
./cache/@4a/I03nfs/@4f/Jk0M00wg00000000000000Y--knmon200/@77/Eo0g000M00E60180000wcAv20fak00Ad12000

logging errors with erlang and mochiweb

2008-12-15T02:56:00.000-08:00

If you would like to run mochiweb or any erlang application and log messages to disk to review later, all you have to do is create a simple config file and wrote something in it like as follows:

[{sasl, [
%% minimise shell error logging
{sasl_error_logger, false},
%% only report errors
{errlog_type, error},
%% define the parameters of the rotating log
%% the log file directory
{error_logger_mf_dir,"/var/logs/error_logs"},
%% # bytes per logfile
{error_logger_mf_maxbytes,10485760}, % 10 MB
%% maximum number of
{error_logger_mf_maxfiles, 10}
]}].

let's say we call this filename erlang_log.config. The trick here is that we have to create the log directory for erlang and make sure it's writable by the user which runs erlang process.
# mkdir /var/logs/error_logs/
you can also make it world writable if you don't care about who reads it on the server.
# chmod 777 /var/log/error_logs/

now pass the config file as parameter to your erl command (on mochiweb part, you can edit start.sh or start-dev.sh file).

erl -config erlang_log

when you first run your erlang process or mochiweb, first of all run on your erlang command prompt
rb:start().
to create index files in log directory and load rb module.Here is some useful commands to see log messages created by erlang. The log files in the directory are binary so you cannot simple read them.You have to read them through erlang (rb module).
rb:list().
%% shows log messages index
rb:show(number).
%% shows given indexed numbered message details
rb:rescan().
%% re-reads the error log directory index.

running mochiweb or any erlang application in background

2008-12-15T02:11:00.000-08:00

Erlang is Ericsson's concurrent programming language.See Erlang Wikipedia for more information. Mochiweb is a web framework to create lightweight http servers written in Erlang language.
After creating a skeleton project with mochiweb (./scripts/new_mochiweb.erl). You can write your own code and compile it. Here is an article shows this in detail.
After all, running your new http server is done by star.sh or start-dev.sh in your application directory where you created by new_mochiweb.erl script. The script runs erl (The Erlang Emulator) command with some parameters. But it runs in foreground, if you want to run the erlang application in background simply add "-detached" parameter to your erl command. But, there is a drawback here. The erlang application runs in background and we cannot see the error and any information log messages now. I'll will explain how to log theses messages in log file and read them in another post.

profiling mysql queries

2008-11-25T10:45:00.000-08:00

Mysql database server has a new query profiling option. It's a very useful feature to profile queries on server and find bottlenecks.
There is an article on mysql site which describes this feature in detail.
Using the New MySQL Query Profiler

finding backup superblocks on ext2 and ext3 filesystems

2008-11-06T11:06:00.000-08:00

If you have a situation where your ext2 or ext3 filesystem's primary superblock is corrupted, you need to pass one of the backup superblocks to fsck program. In this case you can find the backup superblocks by using mkfs program. All you have to do is, run mkfs with "-n" option, where mkfs is run in dry mode. It will not destroy your filesystem, it only prints out what it will do on real mkfs procedure.By the way, you have to add the parameters if you have used when your filesystem originally created to get correct values.

eg. mke2fs -n /dev/sda1

and now you can pass your backup superblock number to fsck (fsck.ext2,fsck.ext3) with "-b" parameter.

freeing cache on linux with drop_caches

2008-11-06T10:55:00.000-08:00

Linux kernel 2.6.16 and later has a option which let's you free inode/dentry and pagecache caches on your system. The file is located on /proc filesystem.
It has 3 options. For example:

# echo 1 > /proc/sys/vm/drop_caches

frees pagecache on your system.

# echo 2 > /proc/sys/vm/drop_caches

frees inodes and dentry caches

# echo 3 > /proc/sys/vm/drop_caches

frees pagecache and inode/dentry caches.

tip: some dirty caches are not freeable so it's better to use "sync" command before using drop_caches for freeing much more memory on your linux system.

disabling bell beep sound on linux console

2008-10-29T03:53:00.000-07:00

Sometimes its annoying to hear a beep sound everytime you hit on tab button for completion on linux console. Here is the quickway how to disable that beep sound.
All you have to do is add the following line in /etc/inputrc file or uncomment it if it presents.

set bell-style none

the following image is the default inputrc file on centos system. You can uncomment the line which says "set bell-style none" for disabling tab completion beep sound.

resolving latency problems on linux with latencytop utility

2008-10-27T01:56:00.000-07:00

Finding out what is causing unresponsive behaviour, audio delays or stalls on your linux machines can be a hard task.
Latencytop utility gives information where latency is occured to diagnose problem.

Here is an example how to use latencytop on fedora core 9 distribution.

yum install latencytop

By default kernel does not enable latency statistics reporting.You have to enable it by hand.

echo 1 > /proc/sys/kernel/latencytop

this command enables it. disabling it is easy by echo 0 > /proc/sys/kernel/latencytop
latencytop uses the statistics from proc file system. statistics is in /proc/latency_stats file.

here is a screenshot where it shows latency caused disk io on ext3 filesystem and with which pid caused it.

note: if your linux system does not have /proc/latency_stats file. it's either is disabled when kernel is compiled or your kernel does not have latency stats support. In that case you have to patch your kernel an recompile it.
for patch see http://www.latencytop.org/download.php

extending laptop battery lifetime with Intel's Powertop utility

2008-10-20T11:10:00.000-07:00

Powertop is designed to pinpoint resources (device drivers, software,etc..) that wakes up idle cpu and lets you extend battery life time by fixing or stopping the application. It's useful for laptop users and also for datacenters who wants to save power consumption.

Here is a sample output disk I/O which showing by powetop:

display system statistics with systat on bsd systems

2008-10-18T03:33:00.000-07:00

systat utility displays system statistics on bsd systems (freebsd, dragonfly, etc.).
system statistics includes swap, network interface, disk I/O , virtual memory (vmstat), network utilitzation (mbuf), TCP/IP statistics, network connections (netstat), icmp.

here is some systat options:
pigs,icmp,ip,tcp,ifstat,iostat,swap,mbufs,vmstat,netstat.
following is sample output for "systat -iostat" command:

editing text files with more

2008-08-29T16:41:00.000-07:00

You can edit documents when using more. if you press "v" , your default editor will be started on current line. After saving and exiting your editor, you can continue using more.
Default editor is chosen by VISUAL and EDITOR variables. VISUAL environment overrides EDITOR variable.

block I/O debugging in Linux

2008-07-26T05:44:00.000-07:00

block_dump option enables you to see all read/write and block dirtying operations on files. It's default value is 0. Setting value to 1 enables block I/O debugging.

echo "1" > /proc/sys/vm/block_dump

You can see the log via dmesg, here is a sample output.

to set if off:

echo "0" > /proc/sys/vm/block_dump

Want to see which memory bank is in use on your system?

2008-07-19T03:49:00.000-07:00

System Management BIOS (SMBIOS) pecification reports information of your hardware. In short terms, it tells you what your system bios detects and supports.

In Linux, you can use dmidecode for accessing smbios information. For example, if you like to see which memory banks is in use and what the ram module sizes, you simply call dmidecode with memory parameter.

# dmidecode -t memory

Here is the output on my system, If you look sample output below, you will see total memory supported by your system and max memory module size which can be installed. Also which banks are installed and memory module details like size and speed.

# dmidecode 2.9
SMBIOS 2.4 present.

Handle 0x0005, DMI type 5, 24 bytes
Memory Controller Information
Error Detecting Method: 8-bit Parity
Error Correcting Capabilities:
None
Supported Interleave: One-way Interleave
Current Interleave: One-way Interleave
Maximum Memory Module Size: 1024 MB
Maximum Total Memory Size: 4096 MB
Supported Speeds:
Other
Supported Memory Types:
Other
Memory Module Voltage: 5.0 V
Associated Memory Slots: 4
0x0006
0x0007
0x0008
0x0009
Enabled Error Correcting Capabilities:
None

Handle 0x0006, DMI type 6, 12 bytes
Memory Module Information
Socket Designation: A0
Bank Connections: 1
Current Speed: Unknown
Type: DIMM SDRAM
Installed Size: 1024 MB (Single-bank Connection)
Enabled Size: 1024 MB (Single-bank Connection)
Error Status: OK

Handle 0x0007, DMI type 6, 12 bytes
Memory Module Information
Socket Designation: A1
Bank Connections: 2
Current Speed: Unknown
Type: Unknown
Installed Size: Not Installed
Enabled Size: Not Installed
Error Status: OK

Handle 0x0008, DMI type 6, 12 bytes
Memory Module Information
Socket Designation: A2
Bank Connections: 3
Current Speed: Unknown
Type: Unknown
Installed Size: Not Installed
Enabled Size: Not Installed
Error Status: OK

Handle 0x0009, DMI type 6, 12 bytes
Memory Module Information
Socket Designation: A3
Bank Connections: 4
Current Speed: Unknown
Type: Unknown
Installed Size: Not Installed
Enabled Size: Not Installed
Error Status: OK

Handle 0x001B, DMI type 16, 15 bytes
Physical Memory Array
Location: System Board Or Motherboard
Use: System Memory
Error Correction Type: None
Maximum Capacity: 4 GB
Error Information Handle: Not Provided
Number Of Devices: 4

Handle 0x001C, DMI type 17, 27 bytes
Memory Device
Array Handle: 0x001B
Error Information Handle: Not Provided
Total Width: 64 bits
Data Width: 64 bits
Size: 1024 MB
Form Factor: DIMM
Set: None
Locator: A0
Bank Locator: Bank0/1
Type: Unknown
Type Detail: None
Speed: 667 MHz (1.5 ns)
Manufacturer: None
Serial Number: None
Asset Tag: None
Part Number: None

Handle 0x001D, DMI type 17, 27 bytes
Memory Device
Array Handle: 0x001B
Error Information Handle: Not Provided
Total Width: Unknown
Data Width: Unknown
Size: No Module Installed
Form Factor: DIMM
Set: None
Locator: A1
Bank Locator: Bank2/3
Type: Unknown
Type Detail: None
Speed: Unknown
Manufacturer: None
Serial Number: None
Asset Tag: None
Part Number: None

Handle 0x001E, DMI type 17, 27 bytes
Memory Device
Array Handle: 0x001B
Error Information Handle: Not Provided
Total Width: Unknown
Data Width: Unknown
Size: No Module Installed
Form Factor: DIMM
Set: None
Locator: A2
Bank Locator: Bank4/5
Type: Unknown
Type Detail: None
Speed: Unknown
Manufacturer: None
Serial Number: None
Asset Tag: None
Part Number: None

Handle 0x001F, DMI type 17, 27 bytes
Memory Device
Array Handle: 0x001B
Error Information Handle: Not Provided
Total Width: Unknown
Data Width: Unknown
Size: No Module Installed
Form Factor: DIMM
Set: None
Locator: A3
Bank Locator: Bank6/7
Type: Unknown
Type Detail: None
Speed: Unknown
Manufacturer: None
Serial Number: None
Asset Tag: None
Part Number: None

dmidecode also gives you information about your system's cache, bios and cpu.
Here is a sample output for processor:

# dmidecode -t processor

# dmidecode 2.9
SMBIOS 2.4 present.

Handle 0x0004, DMI type 4, 35 bytes
Processor Information
Socket Designation: Socket 775
Type: Central Processor
Family: Other
Manufacturer: Intel
ID: F6 06 00 00 FF FB EB BF
Version: Intel(R) Core(TM)2 CPU 630
Voltage: 1.3 V
External Clock: 266 MHz
Max Speed: 4000 MHz
Current Speed: 1866 MHz
Status: Populated, Enabled
Upgrade: Socket 478
L1 Cache Handle: 0x000A
L2 Cache Handle: 0x000B
L3 Cache Handle: Not Provided
Serial Number:
Asset Tag:
Part Number:

information about your system cache
# dmidecode -t cache

# dmidecode 2.9
SMBIOS 2.4 present.

Handle 0x000A, DMI type 7, 19 bytes
Cache Information
Socket Designation: Internal Cache
Configuration: Enabled, Not Socketed, Level 1
Operational Mode: Write Back
Location: Internal
Installed Size: 64 KB
Maximum Size: 64 KB
Supported SRAM Types:
Synchronous
Installed SRAM Type: Synchronous
Speed: Unknown
Error Correction Type: Unknown
System Type: Unknown
Associativity: Unknown

Handle 0x000B, DMI type 7, 19 bytes
Cache Information
Socket Designation: External Cache
Configuration: Enabled, Not Socketed, Level 2
Operational Mode: Write Back
Location: Internal
Installed Size: 2048 KB
Maximum Size: 1024 KB
Supported SRAM Types:
Synchronous
Installed SRAM Type: Synchronous
Speed: Unknown
Error Correction Type: Unknown
System Type: Unknown
Associativity: Unknown

# dmidecode -t bios

bios information includes vendor of your bios and it's version.Useful, to see if you are using and old bios and needing and bios update. Also, which devices are supported and which ones can be use for booting up your system.

Read more!

# dmidecode 2.9
SMBIOS 2.4 present.

Handle 0x0000, DMI type 0, 20 bytes
BIOS Information
Vendor: Award Software International, Inc.
Version: F6
Release Date: 05/28/2007
Address: 0xE0000
Runtime Size: 128 kB
ROM Size: 512 kB
Characteristics:
PCI is supported
PNP is supported
APM is supported
BIOS is upgradeable
BIOS shadowing is allowed
Boot from CD is supported
Selectable boot is supported
EDD is supported
5.25"/360 KB floppy services are supported (int 13h)
5.25"/1.2 MB floppy services are supported (int 13h)
3.5"/720 KB floppy services are supported (int 13h)
3.5"/2.88 MB floppy services are supported (int 13h)
Print screen service is supported (int 5h)
8042 keyboard services are supported (int 9h)
Serial services are supported (int 14h)
Printer services are supported (int 17h)
CGA/mono video services are supported (int 10h)
ACPI is supported
USB legacy is supported
LS-120 boot is supported
ATAPI Zip drive boot is supported
BIOS boot specification is supported

Handle 0x001A, DMI type 13, 22 bytes
BIOS Language Information
Installable Languages: 3
n|US|iso8859-1
n|US|iso8859-1
r|CA|iso8859-1
Currently Installed Language: n|US|iso8859-1

fscking large ext2/ext3 volumes

2008-07-13T01:59:00.000-07:00

If you got following memory allocation error while fscking a large ext2/ext3 volume on a 32-bit system

/dev/sdaX contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Error allocating directory block array: Memory allocation failed
e2fsck: aborted

You can try one of the following methods:
1- boot with a 64-bit live cd and fsck your volume
2- if you don't have a 64-bit cpu, use your one of the partitions or disks for fsck cache.
edit following file
/etc/e2fsck.conf

and add following lines to point fsck to a directory where it can store it's cache files during fsck.
[scratch_files]
directory = /home/myuser/tmp/

be sure that the directory exists, before running fsck.By this way you will reduce the needed memory for fsck. The disadvantage is disk is slow, therefore fsck will take more time to finish it's job.

Regular expression in Bash

2008-07-07T07:12:00.000-07:00

Regular expressions are handy way of quickly manipulating text data or searching any substring or character in it. Almost every language supports it. For example: perl,C,php,java,etc...

Here is some quick tips on regexp with bash.

This if statement is true if string in data variable starts with numeric values.
if [[ $data =~ ^[0-9] ]]; then

^ means start of the string
[0-9] is range from 0 to 9.
[a-z] from a to z but in lower case.
[A-Z] from A to Z but in capital. as you see it's case sensitive.

if string lasts with a digit
if [[ $data =~ [0-9]$ ]]; then

if string starts with a digit and it's not more than 3 digits
if [[ $data =~ ^[0-9]{1,3} ]]; then

{1,3} means 1 character or at most 3 characters of previous declaration

this one means that string starts with lower case letter and ends again with lower case letter
any other character or numeric value fails this if statement

if [[ $data =~ ^[a-z]+*$ ]]; then

$ means end of line
+ means one or more of the preceding element

here is search and replace statements

this one searches character "a" in data variable and changes its first match with "b| character
data=${data/a/b}

this one is exact with previous example with one difference, it searches for all "a" characters and replace all of them with "b" character, please notice double slash.
data=${data//a/b}

Analyzing linux resources with Systemtap

2008-07-06T02:08:00.000-07:00

Systemp lets you gather information about your running linux system. So you can diagnose performance and problems on your system(I/O activity, network, sockets,etc.). It provides a command line interface and a scripting language. There is also a gui interface called stapgui on sourceforge.net. Installing it on Ubuntu is straight forward

sudo apt-get install systemtap
sudo apt-get install linux-image-debug-generic
sudo ln -s /boot/vmlinux-debug-$(uname -r) /lib/modules/$(uname -r)/vmlinux

Systemtap uses kprobes to trace events. Kprobes are markers which are placed on predefined points in kernel. For example if you like to see which programs are run by your system you call syscall exec probe.
Here is a simple stap script which shows which programs are run by your system.

#!/usr/bin/env stap
probe syscall.exec* {
printf("exec %s %s\n", execname(), argstr)
}

As you can see, stap is similiar to C language.
The key point is to choose which probe point you would like to use.
It is also possible to probe only which process you want by pid or it's name.For example:
probe process("/bin/bash").syscall

Here is another example, suppose you have a disk I/O activity and want to know which processes are causing this.

#!/usr/bin/env stap

global reads, writes, total_io
probe kernel.function("vfs_read").return {
reads[execname()] = total_io[execname()] += $return
}
probe kernel.function("vfs_write").return {
writes[execname()] = total_io[execname()] += $return
}
probe timer.s(1) {
foreach(p in total_io- limit 10)
printf("%15s r: %8d KiB w: %8d KiB\n",p, reads[p]/1024,writes[p]/1024)
printf("\n")
}

You can read further at systemtap site
http://sourceware.org/systemtap/langref/

Linux Block Tracing

2008-06-05T14:09:00.000-07:00

blktrace let's you see detailed information about I/O traffic on a disk devices.

You need to have a 2.6.17-rc1 or newer kernel with "Block io tracing" and debugfs enabled in kernel to use blktrace command.

You can download and install blkrace from http://brick.kernel.dk/snaps/ site or if you use ubuntu you can install it with apt-get.
on ubuntu:

$ sudo apt-get install blktrace

blktrace requires debugfs to run. run mount command to see if it's mounted.
If you can't see debugfs in mount command output for example create a directory called /debugfs and mount it by following command as root

# mount -t debugfs none /debugfs

now you can run blkrace command.

blktrace -r /debugfs/ -d /dev/sda -o - | blkparse -i -

default format of blkparse output is as follows (you can change output format if you like with -f option):

by this way you can get detailed information about ongoing disk activity on your system.

Description of Events:

A IO was remapped to a different device

B IO bounced

C IO completion

D IO issued to driver

F IO front merged with request on queue

G Get request

I IO inserted onto request queue

M IO back merged with request on queue

P Plug request

Q IO handled by request queue code

S Sleep request

T Unplug due to timeout

U Unplug request

X Split

simple load balancing with iptables

2008-05-25T00:48:00.000-07:00

iptables has an extension called clusterip. Clusterip extension uses multicast arp feature to achieve load balancing. Let's say we have two web servers called web1(192.168.0.1) and web2(192.168.0.2) and a virtual ip (192.168.0.10) which will be accepting requests for these machines.

virtual ip:192.168.0.10
web1:192.168.0.1
web2:192.168.0.2

Virtual ip will accept the requests and load balance them between these two web servers.

on web1 server run:

# iptables -I INPUT -d 192.168.0.10 -i eth0 -p tcp --dport 80 -j CLUSTERIP --new --clustermac 01:02:03:04:05:06 --total-nodes 2 --local-node 1 --hashmode sourceip
# ifconfig eth0:1 192.168.0.10 netmask 255.255.255.0 up

on web2 server run:
# iptables -I INPUT -d 192.168.0.10 -i eth0 -p tcp --dport 80 -j CLUSTERIP --new --clustermac 01:02:03:04:05:06 --total-nodes 2 --local-node 2 --hashmode sourceip
# ifconfig eth0:1 192.168.0.10 netmask 255.255.255.0 up

only difference between web1 and web2 commands is local-node option as seen above.
now any web requests coming to 192.168.0.10 will be load balanced between web1 and web2.

clusterip supports three hashmodes (sourceip,sourceip-sourceport and sourceip-sourceport-destport) to determine how to route requests to each servers.

This configuration has one drawback. If one of the nodes fall, the other one does not serves incoming requests for the other one. You need to install linux-ha.

If you want to see which requests served by web1 for example, simply run
# cat /proc/net/ipt_CLUSTERIP/192.168.0.1

Let's say web2 is crashed and we would like to first web server (web1) to take care the requests coming to web2.
on web1 server run:
# echo "+2" >> /proc/net/ipt_CLUSTERIP/192.168.0.1
now on, web1 will take the requests coming to web1.
When you up the web2 server, just run
# echo "-2" >> /proc/net/ipt_CLUSTERIP/192.168.0.1
and web1 will not serve the request for web2.

clamsmtp rbl support patch

2008-05-10T04:57:00.000-07:00

clamsmtp is lightweight anti virus smtp proxy. You can put it in front or back of your smtp server. But if you run it in front of your smtp server and relay traffic back to your original smtp server, you cannot use any rbl server.

So, I decided to write a patch which adds this new rbl option to clamsmtpd server. You can download patch from here clamsmtp rbl patch

qmail linefeed problems hotmail , dcc and others

2008-05-06T10:11:00.000-07:00

While I was testing DCC Checksum , I discovered that some email messages are truncated and dcc reports that body part is missing with an error message "missing message body". I figured out that not only the body part is missing also the headers also truncated.

A quick investigation with tcpdump showed that spamc is sending truncated messages to spamd server which is passed to dcc server. A second tcpdump run on smtp server showed that some messages is rejected by qmail server with "451 See http://pobox.com/~djb/docs/smtplf.html." message. This was hitting a small amount of mails but the ones especially coming from hotmail.com.

I have quickly fixed the problem with djb's fixcrio application which comes with DJB's ucspi-tcp package. No more "missing message body" error messages anymore with dcc :)

Running fixcrio is easy. Just put the command before qmail-smtpd where you run it.

VI tricks part one

2008-05-01T12:04:00.000-07:00

I have finished first part of my VI editor tricks article. Part one explains main commands and tricks like sort data, replacing selected lines and feeding vi document content as input to any program while getting output as new document.
You can read VI tricks part one here.

FreeBSD GJournal (Filesystem Journal)

2008-04-20T10:03:00.000-07:00

Freebsd 7.0 has a new geom extenstion called gjournal. You can easily create a journaled filesystem with gjournal on bsd systems.
First of all, gjournal support has to be built in kernel. It's currently built in 7.x series. The kernel option is called UFS_GJOURNAL

You should disable soft updates on filesystems where gjournal is used. Because journalling takes place and there is no need to use soft updates for data recovery on a crash.

While the filesystem is journalled, the mount option async will boost the performance.

For example, let's say we have a disk called ad1 and it has existing filesystem on it.

All we have to do is create a label.

# gjournal label -f /dev/ad1

notice the -f option. It will force convert operation of an existing file system to journal. Journal device will be separate and called in this example as /dev/ad1.journal.

now it's time to load geom gjournal kernel module.

# gjournal load

now add journalling support to our existing filesystem.

# tunefs -J enable /dev/ad1.journal

it's better now to disable soft updates.

# tunefs -n disable /dev/ad1.journal

now we mount our new journalled filesystem with async option.

# mount -o asyn /dev/ad1.journal /mnt

in our example, we don't have any slices on our disk. In slice case new devices will be created (eg. /dev/ad1s1.journal)

Sqlite compression

2008-02-05T12:33:00.000-08:00

Sqlite database has no built-in compression support. But it has a feature called loadable extension, which let's developers to create any addon feature for sqlite database. I have wrote some documentation about sqlite loadable extension and how to add compress/uncompress ability to sqlite database to compress your data's on the fly.
Here is my article

ucspi-tcp mysql patch

2008-01-13T00:04:00.000-08:00

Finally, I had spare time to finish my ucspi-tcp mysql patch . Putting tcpserver rules in a mysql table has many advantages like centralized access rule lists and easy management.

prevent caching of swf,flv or other files

2007-09-12T05:02:00.000-07:00

The header fields Cache-Control and Pragma can be used to cache or prevent caching of web content. Using this header fields on html meta tags doesn't seem to work. Forcing them from http server, seems the best way that works with Internet Explorer.

Following example shows how to configure no cache option on Apache web servers
You should add following statements into your httpd.conf file (mod_header should be enabled of course)

Header set Cache-Control "no-cache, must-revalidate"
Header set Pragma no-cache

Another example in php language

header("Cache-Control: no-cache, must-revalidate");
header("Pragma: no-cache");

For further details and how to configure IIS server please see this article on Microsoft's site.