lsof alternatives on FreeBSD

lsof is a utility which gives information about open sockets/files/pipes on many unix systems. You can easily install lsof and try it on your freebsd installation by using pkg_add command.

# pkg_add -r lsof

But FreeBSD has two utilities coming bundled with default installation. They are called fstat and sockstat.

fstat tells you which user,command and pid opened the file, which mount point the file is and information about the open file descriptor like read/write, inode number and mount point of the opened file.Please see man page of fstat for other options.

sockstat gives you information about the opened sockets like which process/command is using it, user of the process,pid,protocol like tcp/udp,stream,dgram,etc.. and connected ports of local and remote servers.

Freebsd kernel process tracing

ktrace utility enables to trace and log kernel system calls made by process. By default, it logs to ktrace.out file but this can be overwritten by providing another log filename with -f parameter. You need to pass your command to ktrace or use the pid of a running process.
Also you have to say the kernel what system calls to trace by -t parameter. -t parameter has following options:

c trace system calls
n trace namei translations
i trace I/O
s trace signal processing
t trace various structures
u userland traces
w context switches
+ trace the default set of trace points - c, n, i, s, t, u

While tracing is going in kernel , logging stops when the process stops execution or trace popint ends. The other way is to use -c parameter of the ktrace and provide the pid of the process to stop tracing any further.

to trace and log any running processes simply use the -p parameter and pass the process number (pid) to ktrace. here is a simple example with ktrace to trace find command:

# ktrace -t+w /usr/bin/find /

and following is a short snippet from the log created by ktrace and dispayled using kdump utility

# kdump -f ktrace.out





The log file created by ktrace can be read with kdump utility. Simply pass your ktrace log filename to kdump with -f parameter.

freebsd network tuning

Few days ago, I couldn't reach on of my freebsd 7.1 servers via ssh. Machine was not giving any response to any packets on the network. This is a server which gets moderate network traffic created by short and long lived network connections. After log in from console, I ran the "vmstat -z" command.
Looking closely to vmstat -z output, I figured out that some kernel zone allocations failed for following zones:
mbuf_cluster: 2048, 25600, 1278, 24322, 393553280, 1384
tcptw: 52, 5184, 0, 5184, 3348441, 1304539
tcpreass: 20, 1690, 0, 1690, 10759020, 503124

simply,
tcptw -> tcp timewait
tcpreass -> tcp reassembly
mbuf_cluser -> network buffer data stored by freebsd kernel

I decided to bump the default numbers. I have used /etc/sysctl.conf to increase
net.inet.tcp.maxtcptw = 12000
kern.ipc.nmbclusters = 32768

numbers. then run /etc/rc.d/sysctl restart. you can run vmstat -z to see if the numbers are in effect. tcpreass is a bit different. It should be written in /boot/loader.conf file and will take effect after rebooting your machine.

net.inet.tcp.reass.maxsegments = 4096

After rebooting my machine I checked the results but I didn't get tcpreass numbers as I wrote to boot loader.conf file. I decided to look what happened. I examined the freebsd 7.1 kernel sources and see tcp_reass_init() has an EVENTHANDLER_REGISTER which calls tcp_reass_zone_change() function when nmbclusters numbers changed by sysctl. The rule is simple, when the machine boots
tcp_reass_init functions calculates tcpreass default value as nmbclusters / 16. But if you add net.inet.tcp.reass.maxsegments to your boot loader it skips the auto calculated default and gets the number you've given. Then as default register a event handler to watch nmbclusters changes by users for auto calculate tcpreass again.In my case it gets the given number but when systcl gets into the account kernel detects that nmbclusters changed and recalculates it. So, I decied to remove my parameter from boot loader.conf and instead of that increase the mbuf cluster number to a little bit higer via sysctl.conf. The reasons here is that tcpreass queue uses mbufs, therefore it's auto tuned by freebsd kernel to not run out of mbuf clusters on system.


NOTE: put this accounting that each mbuf cluster allocates 2KB in memory.

profiling performance of an erlang application with eprof

Sometimes it's hard to pinpoint performance problems. In this cases profiling tools come handy. One of the profiling tools in Erlang is called eprof. Here is a simple example how to profile your erlang application with eprof.
First of all, you need to find which pid (module) to profile. You can use "nregs()." command to find your application's pid number in erl shell. In my case pid number is "<0.34.0>".
After finding the pid number start eprof module:

eprof:start().

now tell eprof to start profiling your application by giving the pid number.

eprof:start_profiling([pid(0,34,0)]).

note here that I have changed the dot in pid number to commas.

now wait sometime for eprof to collect some data or did something where you think the application was responding slow or stalling the cpu or whatever.

now stop profiling.

eprof:stop_profiling().

ok now we can see which functions was occupying our cpu with following command:

eprof:total_analyse().


Here is some screen output for total_analyse from my case:
6> eprof:total_analyse().
FUNCTION CALLS TIME
file:file_name_1/1 3450 13 %
erlang:port_control/3 250 7 %
prim_inet:enc_opts/2 300 5 %
erlang:port_command/2 50 4 %
erlang:port_close/1 50 4 %
prim_inet:enc_opt_val/4 300 3 %
gen:wait_resp_mon/3 200 3 %
filename:do_flatten/2 650 3 %
gen:do_call/4 200 2 %
prim_inet:dec_opt_val/3 300 2 %

if you like to see results for process by process use "analyse()" instead of "total_analyse()"
Also,It's possible to log these results to file by eprof's "log" function.

Caching filesystems on linux with fs-cache (cachefilesd)

fs-cache can be use to cache nfs,local or other network filesystems. Here is an example howto cache a nfs mount point on Centos linux distribution. First of make sure that cachefilesd package is installed. Then make sure that cachefilesd (/etc/init.d/cachefilesd) runs on every boot. You can set it to run on startup by chkconfig utility if you like.
Cachefilesd has a config file named /etc/cachefilesd.conf, where you can set where the cache content will be written. The option is called "dir".

eg. dir /cache

Note: caching directory should be an ext3 filesystem.

Let's assume that we have a nfs mount point called mydisk on server 192.168.0.2. Let's create an entry in our fstab for this mount point

192.168.0.2:/mydisk /mydisk nfs tcp,fsc 0 0

the only option here is "fsc" for letting the caching start. After using check your cache directory in this case our directory is /cache to see it's working. There will be 2 directories there called graveyard and cache. Here is some cache filenames created on my cache directory.

./cache/@4a/I03nfs/@4f/Jk0M00wg00000000000000Y--knmon200/@08
./cache/@4a/I03nfs/@4f/Jk0M00wg00000000000000Y--knmon200/@08/Eo0g000M00E60180000gA5Ac0gmgO00FY0100
./cache/@4a/I03nfs/@4f/Jk0M00wg00000000000000Y--knmon200/@10
./cache/@4a/I03nfs/@4f/Jk0M00wg00000000000000Y--knmon200/@10/Eo0g000M00E601800000D5Ac0cmgO04FY0100
./cache/@4a/I03nfs/@4f/Jk0M00wg00000000000000Y--knmon200/@77
./cache/@4a/I03nfs/@4f/Jk0M00wg00000000000000Y--knmon200/@77/Eo0g000M00E60180000wcAv20fak00Ad12000

logging errors with erlang and mochiweb

If you would like to run mochiweb or any erlang application and log messages to disk to review later, all you have to do is create a simple config file and wrote something in it like as follows:

[{sasl, [
%% minimise shell error logging
{sasl_error_logger, false},
%% only report errors
{errlog_type, error},
%% define the parameters of the rotating log
%% the log file directory
{error_logger_mf_dir,"/var/logs/error_logs"},
%% # bytes per logfile
{error_logger_mf_maxbytes,10485760}, % 10 MB
%% maximum number of
{error_logger_mf_maxfiles, 10}
]}].

let's say we call this filename erlang_log.config. The trick here is that we have to create the log directory for erlang and make sure it's writable by the user which runs erlang process.
# mkdir /var/logs/error_logs/
you can also make it world writable if you don't care about who reads it on the server.
# chmod 777 /var/log/error_logs/

now pass the config file as parameter to your erl command (on mochiweb part, you can edit start.sh or start-dev.sh file).

erl -config erlang_log

when you first run your erlang process or mochiweb, first of all run on your erlang command prompt
rb:start().
to create index files in log directory and load rb module.Here is some useful commands to see log messages created by erlang. The log files in the directory are binary so you cannot simple read them.You have to read them through erlang (rb module).
rb:list().
%% shows log messages index
rb:show(number).
%% shows given indexed numbered message details
rb:rescan().
%% re-reads the error log directory index.

running mochiweb or any erlang application in background

Erlang is Ericsson's concurrent programming language.See Erlang Wikipedia for more information. Mochiweb is a web framework to create lightweight http servers written in Erlang language.
After creating a skeleton project with mochiweb (./scripts/new_mochiweb.erl). You can write your own code and compile it. Here is an article shows this in detail.
After all, running your new http server is done by star.sh or start-dev.sh in your application directory where you created by new_mochiweb.erl script. The script runs erl (The Erlang Emulator) command with some parameters. But it runs in foreground, if you want to run the erlang application in background simply add "-detached" parameter to your erl command. But, there is a drawback here. The erlang application runs in background and we cannot see the error and any information log messages now. I'll will explain how to log theses messages in log file and read them in another post.