Process IO Top utility with Solaris DTrace

DTrace is dynamic tracing framework created by Sun Microsystems for Solaris OS. DTrace let's any system administrator or developer to get internal view of how an operating system works, find any system bottlenecks, examine a live system or process, get information on system calls, network internals and many other things. For this purpose DTrace has many providers for specific tasks. You can get list of the DTrace providers and probe function names by running following command:

# dtrace -l

DTrace also adopted to other operating systems like FreeBSD and Mac OS X. DTrace uses D Language which is subset of C programming language. You can find more information about DTrace on Sun's BigAdmin Portal.

Following example demostrates you, how easy to find the information you want on Solaris with DTrace.I wrote it to see which files are most written or read by which processes on the Solaris.

#!/usr/bin/bash
# process_io_top -  show procesess by top read/write KB I/O per file
#      Written by Levent Serinol (lserinol@gmail.com)
#      http://lserinol.blogspot.com
# Apr/14/2009
#
# USAGE: process_io_top [-s interval] [-p pid]
#   
#  -s interval  # gather and show statistics in given interval (seconds)
#  -p pid  # show read/write KB I/O just for given PID
#  -h   # show usage information
#
# eg:
# process_io_top -s 10 
#
#
#
####################################################################################
interval=5
show_pid=0
pid=0

function usage()
{
echo "
USAGE: io.sh [-s interval] [-p pid]
         -s             # set interval, default is 5 seconds
         -p pid         # pid
  eg, 
         io -p 630                # io activity of pid 630
         io -s 10                 # refresh output in every 10 seconds";
}

while getopts h:p:s:a name
do
        case $name in
        p)      show_pid=1; pid=$OPTARG ;;
        s)      interval=$OPTARG ;;
        h|?)    usage;
               exit 1
        esac
done

/usr/sbin/dtrace -Cws <( cat <<EOF

 inline int PID  = $pid;
 inline int SHOW_PID    = $show_pid;
 inline int INTERVAL    = $interval;


#pragma D option quiet
#pragma D option aggsortrev


dtrace:::BEGIN 
{
  secs = INTERVAL;
  printf("Please wait....\n");
}

io:::start 
/ (SHOW_PID == 1) && ( pid == PID) /
{
 self->rw=args[0]->b_flags & B_READ ? "R" : "W";
 @files[pid,execname,self->rw,args[2]->fi_pathname] = sum (args[0]->b_bcount);
 @total_blks[self->rw]=count();
 @total_bytes[self->rw]=sum (args[0]->b_bcount);
 self->rw=0;
}

io:::start 
/ SHOW_PID == 0 /
/* SHOW_PID == 0 && args[2]->fi_pathname != "<none>" */
{
 self->rw=args[0]->b_flags & B_READ ? "R" : "W";
 @files[pid,execname,self->rw,args[2]->fi_pathname] = sum (args[0]->b_bcount);
 @total_blks[self->rw]=count();
 @total_bytes[self->rw]=sum (args[0]->b_bcount);
 self->rw=0;
}

profile:::tick-1s
{
        secs--;
}


profile:::tick-1s
/secs == 0/
{

 trunc(@files,30);
 normalize(@files,1024);
 system("/usr/bin/clear"); 
 printf("%Y ",walltimestamp);
 printa("%s %@11d blocks, ",@total_blks);
 printa("%s %@11d bytes, ",@total_bytes);
 printf("\n%6s %-12s %3s %8s %3s\n", "PID", "CMD","R/W", "KB", "FILE");
 printa("%6d %-12.12s %1s %@10d %s\n",@files);
 secs = INTERVAL;

}
dtrace:::END
{
        trunc(@files);
}
EOF
)

You can download it here process_io_top. This scripts briefly shows you how to pass arguments from shell to DTrace. Using aggregations and sort them in reverse order by using pragma "aggsortrev". Defining probes (io:::start) more than once with different predicate conditions. Following is a sample output from process_io_top script.

speeding up your nginx server with memcached

Nginx is a high performance web and proxy (web and mail proxy) server. Generally, nginx is used as a front-end proxy server to Apache webserver. Nginx is known to be slow while serving dynamic pages like php. Normally, nginx is using fast-cgi method which is slow. Therefore, it's a good idea to run Apache as back-end server to Nginx and serve dynamic php pages from Apache. If your website's php pages suitable to cache for a certain time, you can use Nginx proxy module and proxy_store command to cache Apache served php pages output in Nginx automatically as html. Here, I'll give you instructions how to use Nginx's memcache module and Danga Software's memcached deamon to store your content in memory and serve it. Serving content from memory will be faster than serving it from disk. memcached's default listening port is 11211. You can find instructions on Danga Software's website how to compile and run memcached.

Now, we can look our Nginx configuration for memcache implementation. Let's suppose we have two Apache webservers running on two different physical servers. IP addresses of the Apache webservers are 192.168.2.3 and 192.168.2.4. We'll use those Apache webservers as back-end servers. We have a Nginx server as front-end to them on 192.168.2.1 ip adress. First of all, we have to tell Nginx about those back-end servers. We use Nginx upstream module for this purpose. As you can see below, we defined a upstream named "backend". The configuration has our two Apache webservers ip addresses. Upstream module let's you also give weight to each server in configuration. Our first server's hardware configuration is better than the second one, so we gave the first one weight value 2. This configuration should be in http section of Nginx configration file (nginx.conf).

upstream backend {
     server 192.168.2.3 weight=2;
     server 192.168.2.4;
}

We have created our upstream configuration. Now, we have to tell Nginx, which files will be server by memcache module. I have decided to only serve some image types by memcache. The following configuration part should be in server section of Nginx configuration. The "location" directive tell's the nginx to handle every file which ends with given extensions like .jpg,.png and .gif in url. As first step, Nginx will check the url in memcached. Memcached is simple key value memory database. Every row has a unique key.In our case the key is our url. If Nginx, finds the key (url) in memcached, it will get contents of the key from mecached and send it back to client. This operation is running completely from memory. In case that the key (url) not found, it will fallback to 404 and as you can see, we catch 404 error and send request to our back-end Apache servers. Nginx will then send Apache's response to client.

location ~* \.(jpg|png|gif)$ {
access_log   off;
expires      max;
add_header   Last-Modified "Thu, 26 Mar 2000 17:35:45 GMT";
set $memcached_key $uri;
memcached_pass     127.0.0.1:11211;
error_page         404 = /fetch;
}

location /fetch {
internal;
access_log   off;
expires      max;
add_header   Last-Modified "Thu, 26 Mar 2000 17:35:45 GMT";
proxy_pass http://backend;
break;
}

Of course, we have a drawback here. Nginx's memcache module never put anything automatically in memcached. You have to store your information in it manually by using something like a script. Considering our example, if we forget to store information about a file in memcached, it will be always served by back-end Apache servers. Here is a simple php script, which finds given image types and deploy it into memcached for Nginx.

<?php

function rscandir($base='', &$data=array()) {
$array = array_diff(scandir($base), array('.', '..'));

foreach($array as $value) :
  if (is_dir($base.$value)) {
    $data = rscandir($base.$value.'/', $data);

  }
  elseif (is_file($base.$value)) {
   $rest = substr($value, -4);
   if ((!strcmp($rest,'.jpg')) || (!strcmp($rest,'.png'))
                                || (!strcmp($rest,'.gif')) ){
         $data[] = $base.$value;
   }
 }

endforeach;
return $data;
}

$mylist=rscandir("/var/www/mysite");

$srch = array('/var/www/mysite');
$newval = array('');

$memcache_obj = memcache_connect("192.168.2.1", 11211);

while (list($key, $val) = each($mylist)) {
  $url=str_replace($srch,$newval,$val);
  echo "$key => $val -> ".filesize($val)."\n";
  $value = file_get_contents($val);
  memcache_add($memcache_obj, $url, $value, false, 0);
}
?>

You need to run this script one time, it will find all given image types and store them into memcached. I run this on one of the Apache back-end servers. It will store data into memcached. This memcached is located on Nginx server which ip address is 192.168.2.1 .

Hammer Filesystem

On Feb 17 2009, new release 2.2 of DragonFly BSD released which includes a new filesystem called hammer. Hammer is a new filesystem which intented to replace ffs on DragonFly systems. Hammer filesystem has many advanced futures compared to ffs filesystem like

• instant crash recovery
• large file systems and multivolume support
• data integrity checking
• history and snapshots
• reblocking (on the fly defragmentation)
• mirroring
  

Let's start by creating a new hammer filesystem on a new disk device by using hammer's newfs_hammer command. As you can see below, we used the '-L' parameter. Every hammer filesystem requires a label name and it's mandatory.




Let's mount in under /mnt directory by using "noatime" option. mount_hammer command must be used for this operation.


Looking to df output you can see our new hammer filesystem labeled as "mydata". Now, let's see where hammer filesystem differs from ffs.

History feature

History metadata of a hammer filesystem is written on every 30 seconds. You can use hammer utility to access your files history data. Usage is simple as, calling hammer utility with history command and given your file name which you want to see whole history.

# hammer history /mnt/test.txt



You may be noticed that I have used "sync" command, because I didn't want to wait the the kernel sync operation occur (30 seconds) and forced it by using sync comand to get history result fast.


Snapshot feature

Now let's see hammer's snapshot feature. I have created a directory called "/snapshots" where I want to keep all of my hammer filesystem snaphosts. We'll use hammer utility for taking snapshots.

# hammer snapshot /mnt /snapshots/mnt_snapshot1

I have created a file on /mnt filesystem and took a snapshot. After a sucsessful snapshot, I deleted the original file from /mnt and accessed it by using the snapshot I took before deleting the file (/snapshots/mnt_snapshot1/test.txt).


Undo Feature

You can see and rollback every change between the taken snapshots and original files by using hammer's undo command. You can use "-d" option to get diff output to see every changes on the given file line by line between every snapshot and original one. Also, "-a" option gives you whole change history of given file. Here, I created a test file called "myfile.txt" under my new hammer filesystem (/mnt) and then took a snapshot. After taking snapshot, I added one more line to my test file and used "undo -d /mnt/myfile.txt" to get diff output and see what changes made the file between snapshots.





Mutivolume Feature

As I mentioned above, you can use multi volumes and create a big disk volume with hammer filesystem. Now, we'll create a single hammer volume by using two disks (ad1 and ad3).
You just call newfs_hammer and mount_hammer commands by only giving two or more disk device names instead of one.Commands are straight forward. First, we need to create new hammer filesystem on disks.

# newfs_hammer -L big /dev/ad1 /dev/ad3



I have used to disks and each size is 8GB. After creating new hammer filesystems on disks , we can mount them as one big volume. (I know 8GB is not big :) ).

# mount_hammer /dev/ad1 /dev/ad3 /mnt

You can see on /mnt mount point we have a disk in size 16GB, total of two 8GB disks.

Mirror Feature

Hammer uses Pseudo File Systems (PFS) to duplicate inode numbers to slaves. Therefore, mount_null command required for this operation. This duplication is triggered by using hammer utlity with mirror-copy parameter. First we need to create two pfs, called for master and slave. I created them as /mymaster and /myslave. Master volume must be created with pfs-master and slave with pfs-slave parameter by using hammer utility. One thing to take in accounting here is that slave pfs must use master's shared-uuid number.




After creating mymaster and myslave pfs we have to use mirror-copy to start initial mirroring operation by using these pfs links , because mount_null can't access them without this copy operation.



After, running first mirror copy operation on our pfs links, we can mount our original hamme volumes where we want and associate them with the pfs links we created and sync (mirror-copy) our mounted hammer volumes.


You can use mount_null mount points to do your filesystems operations. You can see that slave volume (myslave) is read-only and you don't have write access to it. You can only use master volume (mymaster) to make your write operations.If you're looking an alternative filesystem to ffs on BSD systems, I suggest you to evaluate both ZFS on FreeBSD and Hammer on DragonFly BSD.

xfs filesystem fragmentation

Many filesystems use many techniques to overcome filesystem fragmentation and XFS filesystem is one of them. But in long usage, you may face with fragmentation problems.Here, I will show you how to find fragmentation ratio in your xfs filesystem and how to re-organize your xfs filesystem with xfs_fsr utility to reduce fragmentation on a live xfs filesystem. First, let's check our xfs filesystem's fragmenation ratio using xfs_db (xfs debugger) utility. As shown below, we used two parameters. First one is "-r", this means that we use debugger in read-only mode. We do not want to accidently do anything bad to our filesystem. Also, this option is useful if we are debugging a mounted filesystem. Second option is "-c" command, which simply takes it's parameter as xfs_db command and run it. Without "-c" parameter, xfs_db will run it's interactive shell where you can run xfs_db commands.



We see that our xfs filesystem on /dev/sdb device has 0.14% fragmentation. This is very low fragmentation. We don't need to defrag it. But here how to do it with xfs_fsr. This utility re-organizes a xfs filesystem while it's mounted file by file. As you can see in sample below, it runs with "-t" parameter, this tells xfs_fsr utility to run maximum that given seconds and quit.


There is one more option you can use while using xfs filesystem to avoid future fragmentations. It's called "allocsize", where it preallocates disk space before writing to a file. You have to set this preallocation size while mounting your xfs filesystem. This will prevent fragmentation on your xfs filesystem if you give a reasonable size depending on your average file sizes.

mount -t xfs -o noatime,allocsize=8M /dev/sdb /mydata

you can also set this option in your /etc/fstab.

stopping and resuming a process run in Linux

You can stop (pause) a running process by sending "-STOP" signal via kill command in Linux. You can re-run same stopped process again by sending "-CONT" signal to it. Assume that we have a process to stop and it's pid is 8674. Following, command will stop the running process id 8674.

# kill -STOP 8674

Now the process is stopped, to resume it's running state, you have to send "-CONT" signal.

# kill -CONT 8674

In Solaris, you can do the same task by using pstop and prun utilities or you can use kill command and use signals "SIGSTOP" and "SIGCONT".

measuring your network throughput on Solaris

You can use nicstat utility to measure your network device utilization and throughput. The utility itself uses solaris kstat for network device statistics information. Statistics include network read and write in kilobytes, packets and device utilization.You can compile nicstat with gcc as shown below.

# gcc nicstat.c -o nicstat -lkstat -lgen -lsocket

Following image shows a sample output from nicstat utility.

getting network interface statistics on Solaris

There are many ways to get network interface status and statistics on Solaris. I'll show some of them. First one is well known command netstat. netstat's "-i" parameter will give you statistics of devices used for ip traffic. Statistics include input/output total packets,errors and collisions for each network device.


Next method is by using solaris ndd command. First, we have to find our device name. This can be done by ifconfig command.
My network device name is "e1000g0" as seen in sample output above. Now, it's time to ask ndd which parameters are supported by this device. We do this by calling ndd (ndd /dev/e1000g0 \?) with our device name and "\?" parameter to display every supported parameter.


ndd let's you change some interface settings. They're marked as "read and write". The parameter names clearly shows what they are for. As you can see in sample output below, ndd can give us link speed, link status (cable plugged/unplugged),duplex mode and autoneg status. Following sample output shows us that my network interface link status is 1, which means the network cable is plugged,Network speed is 1000mbit and auto negotiation is on.
The other command is called dladm. dladm can give us above results with one command and in easily readable format. Also, it can give network interface statistics like netstat with "-s" parameter except collisions.