FreeBSD and procfs

Many Unix systems have support for proc file system (process file system).Procfs filesystem type is pseudo. FreeBSD is one of that Unix systems. Unlike Linux, which has information other than processes, FreeBSD procfs support is only about the processes on the system. FreeBSD doesn't mount the procfs on boot by default. You need to manually add it to fstab for auto mount on boot or mount it by command for temporarily usage.Common mount point for procfs is /proc on Unix systems.

echo "none            /proc        procfs  rw   0 0" >> /etc/fsab

mount -t procfs none /proc

Every process is presented as directories named by it's pid number on the /proc mount point.
Procfs gives information on running processes on the system like memory mapping,command line arguments of running process, process resource limits and many other.Following is a sample procfs directory structure on a FreeBSD machine.



As you can see, every pid is represented as a directory in procfs. Every directory contains following files. Some of the files are write only or read only where you read information or send information to process.

- status  (read-only) : returns process status
- mem     (read/write): virtual memory image of the process
- file    (depends)   : symbolic link to running process
- regs    (read/write): process registers
- ctl     (write-only): used to send signal to process or 
                        attach/deattach it for debugging
- cmdline (read-only) : command line arguments of running process
- rlimits (read-only) : current resource limits of running process
- map     (read-only) : memory mappings of the running process.
- etype   (read-only) : type of the executable (eg. FreeBSD ELF32)
- fpregs  (read/write): floating point registers 

Some of the information provided by these files are in binary format.For example "regs" and "fpregs" files are in binary format. They depend on the architecture of the underlying machine (i386, amd64,sparc64,etc..). Following is the format of the "regs" file on the i386 machine.

struct reg {
        unsigned int    r_fs;
        unsigned int    r_es;
        unsigned int    r_ds;
        unsigned int    r_edi;
        unsigned int    r_esi;
        unsigned int    r_ebp;
        unsigned int    r_isp;
        unsigned int    r_ebx;
        unsigned int    r_edx;
        unsigned int    r_ecx;
        unsigned int    r_eax;
        unsigned int    r_trapno;
        unsigned int    r_err;
        unsigned int    r_eip;
        unsigned int    r_cs;
        unsigned int    r_eflags;
        unsigned int    r_esp;
        unsigned int    r_ss;
        unsigned int    r_gs;
};

You can use "cat" command to read information provided by procfs for text based information unlike the ones I mentioned above in binary format like "regs","fpregs" and "mem".

[root@freebsd ]# cat cmdline 
/usr/sbin/moused-p/dev/ums0-tauto-I/var/run/moused.ums0.pid

You can check a running process's resource limits by looking into rlimit file.

[root@freebsd ]# cat rlimit 
cpu -1 -1
fsize -1 -1
data 536870912 536870912
stack 67108864 67108864
core -1 -1
rss -1 -1
memlock -1 -1
nproc 5547 5547
nofile 11095 11095
sbsize -1 -1
vmem -1 -1

First digits is minimum and last one is maximum values of the given resource name. "-1" means infinite. For examle, nofile (open file descriptor) limit for this process is 11095 as minimum and maximum.

"status" file gives information about process status as follows.

- command name
- pid
- parent pid
- process group id
- session id
- major/minor of the terminal, "-" if no terminal is in action
- process flags
- process start time in seconds and microseconds separated by comma
- user time in seconds and microseconds separated by comma
- system time in seconds and microseconds separated by comma
- wait channel name
- effective userid and group lists separated by comma

Following is "cat status" result of a process.

[root@freebsd ]# cat status 
svscan 28246 1 28245 0 ttyp0 noflags 1242921860,839572 0,318052 3,263576 nanslp 0 0 0,0,0,5 -

Configuring ILOM manager on Sun X4540 Thumper

Sun X4540 Thumper storage server provides 48TB hard disk capacity on one server. It has 2 AMD Quad x64 Opteron processors and 32GB of memory. Thumper has 48 hard disk slots. You can use 1TB sata hard disks in each slot. This will give you 48TB hard disk area as you can see in following photo.


You can run Solaris,OpenSolaris, Linux and Windows operating systems on Thumper. Thumper includes Sun Microsystem's ILOM (Sun Integrated Lights Out Manager) manager. You can use ILOM to upgrade your server's firmware, remotely accessing your server's console by using your web browser, modify and see sensors, see hardware components and so on. You can access ILOM by using a serial console,ssh client or a web browser.You need to set an ip address for your ILOM manager if you need to use ssh or a web browser.If you don't have a dhcp server around you'll need to configure your ILOM manager with a static ip address. I have used a serial console connected on Thumpers "Ser MGT" port and on the other side a linux desktop and minicom application. You need to change your minicom settings to 9600 baud rate , 8 data bits, no parity and one stop bit (9600-8-N-1) with no hardware and software flow control.


Yes, that's right, It's Linux :) . Sun ILOM uses embeded Linux. When you connect to ILOM by a serial console you will need to login in by using ILOM's factory default login name "root" and password "changeme". ILOM has a simple built-in command interface. You can use "help" command for details when you logged in ILOM manager. Setting the static ip address of ILOM is simple. I captured the screenshot when I gave one of my local ip addresses to Thumper's ILOM.



After, you commit your static ip network settings settings with "set commitpending=true" command, you need to connect Thumper's "net mgt" ethernet interface to your network.Now, you can access ILOM by using any ssh client or a web browser. My plan is to use OpenSolaris and 48TB ZFS Raid-Z on this storage server and see the results on a production environment :)

getting detailed process information on Freebsd

Freebsd procstat utility gives detailed information about all of the processes on the system or just for a given process id number, such as virtual memory mapping, thread stack, command line arguments and open files.

Running procstat with "-a" argument prints information like pid,pid,login, process name, wchan (which event the process waiting) of all processes on the system.

# procstat -a



Procstat "-c" option shows you the command line arguments of a process and "-f" option shows opened files by given process.Following sample output shows the command line arguments and the files currently opened with their permissions by vi process.



You can access virtual memory mapping information a process with "-v" option. Following sample shows vi process virtual memory mappings.



Finally, "-k" option shows kernel threads stacks details of given process.

watching interrupt usage on Freebsd

You can easily see how many interrupts taken by each device on Linux by simply looking in /proc/interrupts file. Freebsd has no such information in procfs. But you can access the same information by using "vmstat -i" command in Freebsd.But you can't see which cpu handling which irq as shown by Linux. Following is a sample output of "vmstat -i" command from a Freebsd machine.



You can write a simple c shell script to watch interrupt usage like this.

Vmstat uses sysctl interface to gather interrupt usage information by using hw.intrnames and hw.intrcnt oid names. As names suggest intrnames holds all interrupt names and intrcnt holds their irq counts since system startup for each interrupt. 

Linux NFS and "permission denied" mount problem

Today, I have faced with following strange nfs mount problem.

[root@nfsclient ~]# mount /mymount
mount: XXX.XXX.XXX.XXX:/mymount failed, 
reason given by server: Permission denied

My nfs client is FC4 and nfs server is Centos 4.6.First, I have checked the server from nfs client side by using "showmount -e myserver" command. Exports was okay.After that, I decided to check rpc services on server with "rpcinfo -p" command.

[root@nfsclient ~]# rpcinfo -p myserver

  program vers proto   port
   100000    2   tcp    111  portmapper
   100000    2   udp    111  portmapper
   100024    1   udp    638  status
   100024    1   tcp    641  status
   100021    1   udp  32768  nlockmgr
   100021    3   udp  32768  nlockmgr
   100021    4   udp  32768  nlockmgr
   100021    1   tcp  32772  nlockmgr
   100021    3   tcp  32772  nlockmgr
   100021    4   tcp  32772  nlockmgr
   100011    1   udp    903  rquotad
   100011    2   udp    903  rquotad
   100011    1   tcp    906  rquotad
   100011    2   tcp    906  rquotad
   100003    2   udp   2049  nfs
   100003    3   udp   2049  nfs
   100003    4   udp   2049  nfs
   100003    2   tcp   2049  nfs
   100003    3   tcp   2049  nfs
   100003    4   tcp   2049  nfs
   100005    1   udp    925  mountd
   100005    1   tcp    928  mountd
   100005    2   udp    925  mountd
   100005    2   tcp    928  mountd
   100005    3   udp    925  mountd
   100005    3   tcp    928  mountd

Mandatory NFS services was working correctly as seen above rpc report.I didn't have a firewall between these machines or iptables on any of them.This was not the problem also here.I haven't check nfs client side because it was mounting other nfs server's mount points without a problem at all. So, I have focused on nfs server side. Restarting nfsd service or re-exporting nfs file systems with exportfs didn't work out. I can see nfs clients that mounted server's nfs mount by looking in /var/lib/nfs/rmtab file (showmount -a) on server. But I couldn't access /proc/fs/nfsd/ directory. It was not mounted on nfs server.So,I manually mounted it by using following mount command.

mount -t nfsd nodev /proc/fs/nfsd

After mounting /proc/fs/nfsd manually, I was able to mount the nfs server from client side again.

If you look at into /etc/modprobe.conf.dist file, you can see that when kernel installs module nfsd , it mounts /proc/fs/nfsd and /var/lib/nfs/rpc_pipefs mount points.

install nfsd /sbin/modprobe --first-time --ignore-install nfsd && 
{ /bin/mount -t nfsd nfsd /proc/fs/nfsd > /dev/null 2>&1 || :; }
install sunrpc /sbin/modprobe --first-time --ignore-install sunrpc && 
{ /bin/mount -t rpc_pipefs sunrpc /var/lib/nfs/rpc_pipefs > 
/dev/null 2>&1 || :; }

Process IO Top utility with Solaris DTrace

DTrace is dynamic tracing framework created by Sun Microsystems for Solaris OS. DTrace let's any system administrator or developer to get internal view of how an operating system works, find any system bottlenecks, examine a live system or process, get information on system calls, network internals and many other things. For this purpose DTrace has many providers for specific tasks. You can get list of the DTrace providers and probe function names by running following command:

# dtrace -l

DTrace also adopted to other operating systems like FreeBSD and Mac OS X. DTrace uses D Language which is subset of C programming language. You can find more information about DTrace on Sun's BigAdmin Portal.

Following example demostrates you, how easy to find the information you want on Solaris with DTrace.I wrote it to see which files are most written or read by which processes on the Solaris.

#!/usr/bin/bash
# process_io_top -  show procesess by top read/write KB I/O per file
#      Written by Levent Serinol (lserinol@gmail.com)
#      http://lserinol.blogspot.com
# Apr/14/2009
#
# USAGE: process_io_top [-s interval] [-p pid]
#   
#  -s interval  # gather and show statistics in given interval (seconds)
#  -p pid  # show read/write KB I/O just for given PID
#  -h   # show usage information
#
# eg:
# process_io_top -s 10 
#
#
#
####################################################################################
interval=5
show_pid=0
pid=0

function usage()
{
echo "
USAGE: io.sh [-s interval] [-p pid]
         -s             # set interval, default is 5 seconds
         -p pid         # pid
  eg, 
         io -p 630                # io activity of pid 630
         io -s 10                 # refresh output in every 10 seconds";
}

while getopts h:p:s:a name
do
        case $name in
        p)      show_pid=1; pid=$OPTARG ;;
        s)      interval=$OPTARG ;;
        h|?)    usage;
               exit 1
        esac
done

/usr/sbin/dtrace -Cws <( cat <<EOF

 inline int PID  = $pid;
 inline int SHOW_PID    = $show_pid;
 inline int INTERVAL    = $interval;


#pragma D option quiet
#pragma D option aggsortrev


dtrace:::BEGIN 
{
  secs = INTERVAL;
  printf("Please wait....\n");
}

io:::start 
/ (SHOW_PID == 1) && ( pid == PID) /
{
 self->rw=args[0]->b_flags & B_READ ? "R" : "W";
 @files[pid,execname,self->rw,args[2]->fi_pathname] = sum (args[0]->b_bcount);
 @total_blks[self->rw]=count();
 @total_bytes[self->rw]=sum (args[0]->b_bcount);
 self->rw=0;
}

io:::start 
/ SHOW_PID == 0 /
/* SHOW_PID == 0 && args[2]->fi_pathname != "<none>" */
{
 self->rw=args[0]->b_flags & B_READ ? "R" : "W";
 @files[pid,execname,self->rw,args[2]->fi_pathname] = sum (args[0]->b_bcount);
 @total_blks[self->rw]=count();
 @total_bytes[self->rw]=sum (args[0]->b_bcount);
 self->rw=0;
}

profile:::tick-1s
{
        secs--;
}


profile:::tick-1s
/secs == 0/
{

 trunc(@files,30);
 normalize(@files,1024);
 system("/usr/bin/clear"); 
 printf("%Y ",walltimestamp);
 printa("%s %@11d blocks, ",@total_blks);
 printa("%s %@11d bytes, ",@total_bytes);
 printf("\n%6s %-12s %3s %8s %3s\n", "PID", "CMD","R/W", "KB", "FILE");
 printa("%6d %-12.12s %1s %@10d %s\n",@files);
 secs = INTERVAL;

}
dtrace:::END
{
        trunc(@files);
}
EOF
)

You can download it here process_io_top. This scripts briefly shows you how to pass arguments from shell to DTrace. Using aggregations and sort them in reverse order by using pragma "aggsortrev". Defining probes (io:::start) more than once with different predicate conditions. Following is a sample output from process_io_top script.

speeding up your nginx server with memcached

Nginx is a high performance web and proxy (web and mail proxy) server. Generally, nginx is used as a front-end proxy server to Apache webserver. Nginx is known to be slow while serving dynamic pages like php. Normally, nginx is using fast-cgi method which is slow. Therefore, it's a good idea to run Apache as back-end server to Nginx and serve dynamic php pages from Apache. If your website's php pages suitable to cache for a certain time, you can use Nginx proxy module and proxy_store command to cache Apache served php pages output in Nginx automatically as html. Here, I'll give you instructions how to use Nginx's memcache module and Danga Software's memcached deamon to store your content in memory and serve it. Serving content from memory will be faster than serving it from disk. memcached's default listening port is 11211. You can find instructions on Danga Software's website how to compile and run memcached.

Now, we can look our Nginx configuration for memcache implementation. Let's suppose we have two Apache webservers running on two different physical servers. IP addresses of the Apache webservers are 192.168.2.3 and 192.168.2.4. We'll use those Apache webservers as back-end servers. We have a Nginx server as front-end to them on 192.168.2.1 ip adress. First of all, we have to tell Nginx about those back-end servers. We use Nginx upstream module for this purpose. As you can see below, we defined a upstream named "backend". The configuration has our two Apache webservers ip addresses. Upstream module let's you also give weight to each server in configuration. Our first server's hardware configuration is better than the second one, so we gave the first one weight value 2. This configuration should be in http section of Nginx configration file (nginx.conf).

upstream backend {
     server 192.168.2.3 weight=2;
     server 192.168.2.4;
}

We have created our upstream configuration. Now, we have to tell Nginx, which files will be server by memcache module. I have decided to only serve some image types by memcache. The following configuration part should be in server section of Nginx configuration. The "location" directive tell's the nginx to handle every file which ends with given extensions like .jpg,.png and .gif in url. As first step, Nginx will check the url in memcached. Memcached is simple key value memory database. Every row has a unique key.In our case the key is our url. If Nginx, finds the key (url) in memcached, it will get contents of the key from mecached and send it back to client. This operation is running completely from memory. In case that the key (url) not found, it will fallback to 404 and as you can see, we catch 404 error and send request to our back-end Apache servers. Nginx will then send Apache's response to client.

location ~* \.(jpg|png|gif)$ {
access_log   off;
expires      max;
add_header   Last-Modified "Thu, 26 Mar 2000 17:35:45 GMT";
set $memcached_key $uri;
memcached_pass     127.0.0.1:11211;
error_page         404 = /fetch;
}

location /fetch {
internal;
access_log   off;
expires      max;
add_header   Last-Modified "Thu, 26 Mar 2000 17:35:45 GMT";
proxy_pass http://backend;
break;
}

Of course, we have a drawback here. Nginx's memcache module never put anything automatically in memcached. You have to store your information in it manually by using something like a script. Considering our example, if we forget to store information about a file in memcached, it will be always served by back-end Apache servers. Here is a simple php script, which finds given image types and deploy it into memcached for Nginx.

<?php

function rscandir($base='', &$data=array()) {
$array = array_diff(scandir($base), array('.', '..'));

foreach($array as $value) :
  if (is_dir($base.$value)) {
    $data = rscandir($base.$value.'/', $data);

  }
  elseif (is_file($base.$value)) {
   $rest = substr($value, -4);
   if ((!strcmp($rest,'.jpg')) || (!strcmp($rest,'.png'))
                                || (!strcmp($rest,'.gif')) ){
         $data[] = $base.$value;
   }
 }

endforeach;
return $data;
}

$mylist=rscandir("/var/www/mysite");

$srch = array('/var/www/mysite');
$newval = array('');

$memcache_obj = memcache_connect("192.168.2.1", 11211);

while (list($key, $val) = each($mylist)) {
  $url=str_replace($srch,$newval,$val);
  echo "$key => $val -> ".filesize($val)."\n";
  $value = file_get_contents($val);
  memcache_add($memcache_obj, $url, $value, false, 0);
}
?>

You need to run this script one time, it will find all given image types and store them into memcached. I run this on one of the Apache back-end servers. It will store data into memcached. This memcached is located on Nginx server which ip address is 192.168.2.1 .