Freebsd ciss driver logical drive limit

Lately, I was faced with a strange problem on a HP server which uses HP Smart Array controller P800 which is running Freebsd. All my 24 hard drives which is hosted on externel HP enclosures was presented as 24 single raid0 logical volumes. Surprisingly, when the freebsd booted it doesn't presented the disks and disabled the ciss driver (HP Smart Array Controller) with following message

Mar 12 16:00:41 freebsd kernel: ciss1: adapter claims to report absurd
number of logical drives (24 > 15)

A quick check on the driver source code showed up that there is a hard limit defined to 15.(/usr/src/sys/dev/ciss/cissvar.h)

 #define CISS_MAX_LOGICAL        15

So I wanted to be sure if I change the number to a higher value , won't do any unwanted behavior later on the system. In that point I contacted Paul Saab where he explained the limit as  below.

 really that's done to limit the amount of memory needed upfront by the
driver.  I believe you can easily increase the number of drives
without issue as long as you have enough memory below 4GB.  Parts of
the ciss driver require that the memory you DMA from be under 4G

After the clarification, I just set the number to 32 and rebuild/install the kernel, restart the machine and a quick check on dmesg showed up all my 24 logical volumes presented on HP P800 controller.

Mar 13 10:32:37 freebsd kernel: da19: 286070MB (585871964 512 byte sectors: 255H 32S/T 65535C)
Mar 13 10:32:37 freebsd kernel: da20 at ciss1 bus 0 scbus2 target 19 lun 0 
Mar 13 10:32:37 freebsd kernel: da20:  Fixed Direct Access SCSI-5 device
Mar 13 10:32:37 freebsd kernel: da20: 135.168MB/s transfers
Mar 13 10:32:37 freebsd kernel: da20: Command Queueing enabled
Mar 13 10:32:37 freebsd kernel: da20: 286070MB (585871964 512 byte sectors: 255H 32S/T 65535C)
Mar 13 10:32:37 freebsd kernel: da21 at ciss1 bus 0 scbus2 target 20 lun 0 
Mar 13 10:32:37 freebsd kernel: da21:  Fixed Direct Access SCSI-5 device
Mar 13 10:32:37 freebsd kernel: da21: 135.168MB/s transfers
Mar 13 10:32:37 freebsd kernel: da21: Command Queueing enabled
Mar 13 10:32:37 freebsd kernel: da21: 286070MB (585871964 512 byte sectors: 255H 32S/T 65535C)
Mar 13 10:32:37 freebsd kernel: da22 at ciss1 bus 0 scbus2 target 21 lun 0 
Mar 13 10:32:37 freebsd kernel: da22:  Fixed Direct Access SCSI-5 device
Mar 13 10:32:37 freebsd kernel: da22: 135.168MB/s transfers
Mar 13 10:32:37 freebsd kernel: da22: Command Queueing enabled
Mar 13 10:32:37 freebsd kernel: da22: 286070MB (585871964 512 byte sectors: 255H 32S/T 65535C)
Mar 13 10:32:37 freebsd kernel: da23 at ciss1 bus 0 scbus2 target 22 lun 0 
Mar 13 10:32:37 freebsd kernel: da23:  Fixed Direct Access SCSI-5 device
Mar 13 10:32:37 freebsd kernel: da23: 135.168MB/s transfers
Mar 13 10:32:37 freebsd kernel: da23: Command Queueing enabled
Mar 13 10:32:37 freebsd kernel: da23: 286070MB (585871964 512 byte sectors: 255H 32S/T 65535C)
Mar 13 10:32:37 freebsd kernel: da24 at ciss1 bus 0 scbus2 target 23 lun 0 
Mar 13 10:32:37 freebsd kernel: da24:  Fixed Direct Access SCSI-5 device
Mar 13 10:32:37 freebsd kernel: da24: 135.168MB/s transfers
Mar 13 10:32:37 freebsd kernel: da24: Command Queueing enabled
Mar 13 10:32:37 freebsd kernel: da24: 286070MB (585871964 512 byte sectors: 255H 32S/T 65535C)

Linux like dmesg in Freebsd

UPDATE:  this patch is applied to Freebsd repository on  10 May 2013

http://svnweb.freebsd.org/base?view=revision&revision=250430

In my previous post, I was talking about the "dmesg -c" equivalent in FreeBSD. Simply it was a sysctl command. So, I decided to simply change FreeBSD dmesg command and make it act like in as Linux. All you have to do is download my patch and apply it as shown below.I was tested the patch on FreeBSD 9 btw.

mybsd# pwd
/usr/src/sbin/dmesg
mybsd# patch < dmesg.patch 
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|--- dmesg.c.org        2012-01-03 19:04:44.000000000 +0200
|+++ dmesg.c    2012-02-28 11:36:22.000000000 +0200
--------------------------
Patching file dmesg.c using Plan A...
Hunk #1 succeeded at 81.
Hunk #2 succeeded at 194.
Hunk #3 succeeded at 204.
done
mybsd# make 
Warning: Object directory not changed from original /usr/src/sbin/dmesg
cc -O2 -pipe  -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -c dmesg.c
cc -O2 -pipe  -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign  -o dmesg dmesg.o -lkvm

Clearing FreeBSD Kernel Message Buffer

As you can clear kernel message buffer on Linux with dmesg command " dmesg -c ", it's a little bit different on a BSD system.It's done by sysctl as following

# sysctl kern.msgbuf_clear=1

FreeBSD and procfs

Many Unix systems have support for proc file system (process file system).Procfs filesystem type is pseudo. FreeBSD is one of that Unix systems. Unlike Linux, which has information other than processes, FreeBSD procfs support is only about the processes on the system. FreeBSD doesn't mount the procfs on boot by default. You need to manually add it to fstab for auto mount on boot or mount it by command for temporarily usage.Common mount point for procfs is /proc on Unix systems.

echo "none            /proc        procfs  rw   0 0" >> /etc/fsab

mount -t procfs none /proc

Every process is presented as directories named by it's pid number on the /proc mount point.
Procfs gives information on running processes on the system like memory mapping,command line arguments of running process, process resource limits and many other.Following is a sample procfs directory structure on a FreeBSD machine.



As you can see, every pid is represented as a directory in procfs. Every directory contains following files. Some of the files are write only or read only where you read information or send information to process.

- status  (read-only) : returns process status
- mem     (read/write): virtual memory image of the process
- file    (depends)   : symbolic link to running process
- regs    (read/write): process registers
- ctl     (write-only): used to send signal to process or 
                        attach/deattach it for debugging
- cmdline (read-only) : command line arguments of running process
- rlimits (read-only) : current resource limits of running process
- map     (read-only) : memory mappings of the running process.
- etype   (read-only) : type of the executable (eg. FreeBSD ELF32)
- fpregs  (read/write): floating point registers 

Some of the information provided by these files are in binary format.For example "regs" and "fpregs" files are in binary format. They depend on the architecture of the underlying machine (i386, amd64,sparc64,etc..). Following is the format of the "regs" file on the i386 machine.

struct reg {
        unsigned int    r_fs;
        unsigned int    r_es;
        unsigned int    r_ds;
        unsigned int    r_edi;
        unsigned int    r_esi;
        unsigned int    r_ebp;
        unsigned int    r_isp;
        unsigned int    r_ebx;
        unsigned int    r_edx;
        unsigned int    r_ecx;
        unsigned int    r_eax;
        unsigned int    r_trapno;
        unsigned int    r_err;
        unsigned int    r_eip;
        unsigned int    r_cs;
        unsigned int    r_eflags;
        unsigned int    r_esp;
        unsigned int    r_ss;
        unsigned int    r_gs;
};

You can use "cat" command to read information provided by procfs for text based information unlike the ones I mentioned above in binary format like "regs","fpregs" and "mem".

[root@freebsd ]# cat cmdline 
/usr/sbin/moused-p/dev/ums0-tauto-I/var/run/moused.ums0.pid

You can check a running process's resource limits by looking into rlimit file.

[root@freebsd ]# cat rlimit 
cpu -1 -1
fsize -1 -1
data 536870912 536870912
stack 67108864 67108864
core -1 -1
rss -1 -1
memlock -1 -1
nproc 5547 5547
nofile 11095 11095
sbsize -1 -1
vmem -1 -1

First digits is minimum and last one is maximum values of the given resource name. "-1" means infinite. For examle, nofile (open file descriptor) limit for this process is 11095 as minimum and maximum.

"status" file gives information about process status as follows.

- command name
- pid
- parent pid
- process group id
- session id
- major/minor of the terminal, "-" if no terminal is in action
- process flags
- process start time in seconds and microseconds separated by comma
- user time in seconds and microseconds separated by comma
- system time in seconds and microseconds separated by comma
- wait channel name
- effective userid and group lists separated by comma

Following is "cat status" result of a process.

[root@freebsd ]# cat status 
svscan 28246 1 28245 0 ttyp0 noflags 1242921860,839572 0,318052 3,263576 nanslp 0 0 0,0,0,5 -

getting detailed process information on Freebsd

Freebsd procstat utility gives detailed information about all of the processes on the system or just for a given process id number, such as virtual memory mapping, thread stack, command line arguments and open files.

Running procstat with "-a" argument prints information like pid,pid,login, process name, wchan (which event the process waiting) of all processes on the system.

# procstat -a



Procstat "-c" option shows you the command line arguments of a process and "-f" option shows opened files by given process.Following sample output shows the command line arguments and the files currently opened with their permissions by vi process.



You can access virtual memory mapping information a process with "-v" option. Following sample shows vi process virtual memory mappings.



Finally, "-k" option shows kernel threads stacks details of given process.

watching interrupt usage on Freebsd

You can easily see how many interrupts taken by each device on Linux by simply looking in /proc/interrupts file. Freebsd has no such information in procfs. But you can access the same information by using "vmstat -i" command in Freebsd.But you can't see which cpu handling which irq as shown by Linux. Following is a sample output of "vmstat -i" command from a Freebsd machine.



You can write a simple c shell script to watch interrupt usage like this.

Vmstat uses sysctl interface to gather interrupt usage information by using hw.intrnames and hw.intrcnt oid names. As names suggest intrnames holds all interrupt names and intrcnt holds their irq counts since system startup for each interrupt. 

FreeBSD kernel profiling with kgmon

Sometimes, it's not possible to find a system bottleneck by profiling and debugging user space processess. You have to profile the kernel to pinpoint the perfomance bottleneck. Here, I will show how to configure a freebsd system for kernel profiling and using kgmon utility to gather kernel profiling data. kgmon produces gprof compatible output. The default output file is called gmon.out .

First of all, we need a profiling enabled kernel configured and build. First step is to use config utility. Now, we need to run config utility from our kernel source configuration directory. I'm using a i386 system. The conf file directory depends on your architecture.
The default directory is /usr/src/sys/ARCH/conf. In my case it's /usr/src/sys/i386/conf/.
We run config utility from where our kernel conf file resides with parameter -p. This tell's the compiler to compile our kernel for low resolution profiling. If we need a high resolution profiling we need to run config with -pp parameter instead of -p. config utility needs a second parameter called SYSTEM_NAME. This is our default system name. FreeBSD's default system name is GENERIC. This can be different on your system, if you build a new kernel with different config file. Older versions of freebsd default GENERIC kernels was not using smp. So it's possible to you have build a new kernel for smp.Therefore you can have a custom build kernel on your system or for another reason. You can see your default system name by using "uname -a" command.
Here is mine is called TESTKERNEL. You'll probably see GENERIC in your case.





After learning you SYSTEM NAME, it's now time to configure our kernel source for profiling. The following picture shows you how to run config with -p option on i386 arch with GENERIC system name.




After running "make cleandepend && depend" command, we run the "make" command on the same directory and it will built a profiling enabled kernel for us.



After the make command, we have build our kernel, now run "make install.debug" command and it will install our debug kernel.

Now, it's time to reboot our machine. After rebooting check kernel messages for following message to see if build is successful for profiling enabled kernel Addresses should be different but seeing message "Profiling kernel" means it's ok.

Profiling kernel, textsize=6845824 [c045cbe0..c0ae4160]

It's time to run kgmon to collect profiling data on our kernel.

# kgmon -b

Now kernel profiling is running for low profiling. For high profiling -B option must be used and don't forget you have to use config -pp for high profiling.
You can now, run the application which was causing the problems on your system to profile it, or create the same situation on server where you were having problems.
Now, after creating and seeing the problem, let's kgmon to dump the profiled kernel data.

# kgmon -p

As I said before, kgmon will create a file called gmon.out for profiled data in gprof format.

Now, we gathered our data, we can now stop profiling.

# kgmon -h

We collected the data we need. Let's see what the gmon.out file contains.

# gprof /boot/kernel/kernel gmon.out

Here is a snippet from gprof output.




I'll explain the meaning of the gprof output in another post.

NOTE: you can apply this technique to other freebsd deriatives like dragonfly bsd and netbsd.

lsof alternatives on FreeBSD

lsof is a utility which gives information about open sockets/files/pipes on many unix systems. You can easily install lsof and try it on your freebsd installation by using pkg_add command.

# pkg_add -r lsof

But FreeBSD has two utilities coming bundled with default installation. They are called fstat and sockstat.

fstat tells you which user,command and pid opened the file, which mount point the file is and information about the open file descriptor like read/write, inode number and mount point of the opened file.Please see man page of fstat for other options.

sockstat gives you information about the opened sockets like which process/command is using it, user of the process,pid,protocol like tcp/udp,stream,dgram,etc.. and connected ports of local and remote servers.

Freebsd kernel process tracing

ktrace utility enables to trace and log kernel system calls made by process. By default, it logs to ktrace.out file but this can be overwritten by providing another log filename with -f parameter. You need to pass your command to ktrace or use the pid of a running process.
Also you have to say the kernel what system calls to trace by -t parameter. -t parameter has following options:

c trace system calls
n trace namei translations
i trace I/O
s trace signal processing
t trace various structures
u userland traces
w context switches
+ trace the default set of trace points - c, n, i, s, t, u

While tracing is going in kernel , logging stops when the process stops execution or trace popint ends. The other way is to use -c parameter of the ktrace and provide the pid of the process to stop tracing any further.

to trace and log any running processes simply use the -p parameter and pass the process number (pid) to ktrace. here is a simple example with ktrace to trace find command:

# ktrace -t+w /usr/bin/find /

and following is a short snippet from the log created by ktrace and dispayled using kdump utility

# kdump -f ktrace.out





The log file created by ktrace can be read with kdump utility. Simply pass your ktrace log filename to kdump with -f parameter.

freebsd network tuning

Few days ago, I couldn't reach on of my freebsd 7.1 servers via ssh. Machine was not giving any response to any packets on the network. This is a server which gets moderate network traffic created by short and long lived network connections. After log in from console, I ran the "vmstat -z" command.
Looking closely to vmstat -z output, I figured out that some kernel zone allocations failed for following zones:
mbuf_cluster: 2048, 25600, 1278, 24322, 393553280, 1384
tcptw: 52, 5184, 0, 5184, 3348441, 1304539
tcpreass: 20, 1690, 0, 1690, 10759020, 503124

simply,
tcptw -> tcp timewait
tcpreass -> tcp reassembly
mbuf_cluser -> network buffer data stored by freebsd kernel

I decided to bump the default numbers. I have used /etc/sysctl.conf to increase
net.inet.tcp.maxtcptw = 12000
kern.ipc.nmbclusters = 32768

numbers. then run /etc/rc.d/sysctl restart. you can run vmstat -z to see if the numbers are in effect. tcpreass is a bit different. It should be written in /boot/loader.conf file and will take effect after rebooting your machine.

net.inet.tcp.reass.maxsegments = 4096

After rebooting my machine I checked the results but I didn't get tcpreass numbers as I wrote to boot loader.conf file. I decided to look what happened. I examined the freebsd 7.1 kernel sources and see tcp_reass_init() has an EVENTHANDLER_REGISTER which calls tcp_reass_zone_change() function when nmbclusters numbers changed by sysctl. The rule is simple, when the machine boots
tcp_reass_init functions calculates tcpreass default value as nmbclusters / 16. But if you add net.inet.tcp.reass.maxsegments to your boot loader it skips the auto calculated default and gets the number you've given. Then as default register a event handler to watch nmbclusters changes by users for auto calculate tcpreass again.In my case it gets the given number but when systcl gets into the account kernel detects that nmbclusters changed and recalculates it. So, I decied to remove my parameter from boot loader.conf and instead of that increase the mbuf cluster number to a little bit higer via sysctl.conf. The reasons here is that tcpreass queue uses mbufs, therefore it's auto tuned by freebsd kernel to not run out of mbuf clusters on system.


NOTE: put this accounting that each mbuf cluster allocates 2KB in memory.

display system statistics with systat on bsd systems

systat utility displays system statistics on bsd systems (freebsd, dragonfly, etc.).
system statistics includes swap, network interface, disk I/O , virtual memory (vmstat), network utilitzation (mbuf), TCP/IP statistics, network connections (netstat), icmp.

here is some systat options:
pigs,icmp,ip,tcp,ifstat,iostat,swap,mbufs,vmstat,netstat.
following is sample output for "systat -iostat" command:

FreeBSD GJournal (Filesystem Journal)

Freebsd 7.0 has a new geom extenstion called gjournal. You can easily create a journaled filesystem with gjournal on bsd systems.
First of all, gjournal support has to be built in kernel. It's currently built in 7.x series. The kernel option is called UFS_GJOURNAL

You should disable soft updates on filesystems where gjournal is used. Because journalling takes place and there is no need to use soft updates for data recovery on a crash.

While the filesystem is journalled, the mount option async will boost the performance.


For example, let's say we have a disk called ad1 and it has existing filesystem on it.

All we have to do is create a label.

# gjournal label -f /dev/ad1

notice the -f option. It will force convert operation of an existing file system to journal. Journal device will be separate and called in this example as /dev/ad1.journal.

now it's time to load geom gjournal kernel module.

# gjournal load

now add journalling support to our existing filesystem.

# tunefs -J enable /dev/ad1.journal

it's better now to disable soft updates.

# tunefs -n disable /dev/ad1.journal


now we mount our new journalled filesystem with async option.

# mount -o asyn /dev/ad1.journal /mnt

in our example, we don't have any slices on our disk. In slice case new devices will be created (eg. /dev/ad1s1.journal)