logging errors with erlang and mochiweb

If you would like to run mochiweb or any erlang application and log messages to disk to review later, all you have to do is create a simple config file and wrote something in it like as follows:

[{sasl, [
%% minimise shell error logging
{sasl_error_logger, false},
%% only report errors
{errlog_type, error},
%% define the parameters of the rotating log
%% the log file directory
{error_logger_mf_dir,"/var/logs/error_logs"},
%% # bytes per logfile
{error_logger_mf_maxbytes,10485760}, % 10 MB
%% maximum number of
{error_logger_mf_maxfiles, 10}
]}].

let's say we call this filename erlang_log.config. The trick here is that we have to create the log directory for erlang and make sure it's writable by the user which runs erlang process.
# mkdir /var/logs/error_logs/
you can also make it world writable if you don't care about who reads it on the server.
# chmod 777 /var/log/error_logs/

now pass the config file as parameter to your erl command (on mochiweb part, you can edit start.sh or start-dev.sh file).

erl -config erlang_log

when you first run your erlang process or mochiweb, first of all run on your erlang command prompt
rb:start().
to create index files in log directory and load rb module.Here is some useful commands to see log messages created by erlang. The log files in the directory are binary so you cannot simple read them.You have to read them through erlang (rb module).
rb:list().
%% shows log messages index
rb:show(number).
%% shows given indexed numbered message details
rb:rescan().
%% re-reads the error log directory index.

running mochiweb or any erlang application in background

Erlang is Ericsson's concurrent programming language.See Erlang Wikipedia for more information. Mochiweb is a web framework to create lightweight http servers written in Erlang language.
After creating a skeleton project with mochiweb (./scripts/new_mochiweb.erl). You can write your own code and compile it. Here is an article shows this in detail.
After all, running your new http server is done by star.sh or start-dev.sh in your application directory where you created by new_mochiweb.erl script. The script runs erl (The Erlang Emulator) command with some parameters. But it runs in foreground, if you want to run the erlang application in background simply add "-detached" parameter to your erl command. But, there is a drawback here. The erlang application runs in background and we cannot see the error and any information log messages now. I'll will explain how to log theses messages in log file and read them in another post.

profiling mysql queries

Mysql database server has a new query profiling option. It's a very useful feature to profile queries on server and find bottlenecks.
There is an article on mysql site which describes this feature in detail.
Using the New MySQL Query Profiler

finding backup superblocks on ext2 and ext3 filesystems

If you have a situation where your ext2 or ext3 filesystem's primary superblock is corrupted, you need to pass one of the backup superblocks to fsck program. In this case you can find the backup superblocks by using mkfs program. All you have to do is, run mkfs with "-n" option, where mkfs is run in dry mode. It will not destroy your filesystem, it only prints out what it will do on real mkfs procedure.By the way, you have to add the parameters if you have used when your filesystem originally created to get correct values.

eg. mke2fs -n /dev/sda1

and now you can pass your backup superblock number to fsck (fsck.ext2,fsck.ext3) with "-b" parameter.

freeing cache on linux with drop_caches

Linux kernel 2.6.16 and later has a option which let's you free inode/dentry and pagecache caches on your system. The file is located on /proc filesystem.
It has 3 options. For example:

# echo 1 > /proc/sys/vm/drop_caches

frees pagecache on your system.

# echo 2 > /proc/sys/vm/drop_caches

frees inodes and dentry caches

# echo 3 > /proc/sys/vm/drop_caches

frees pagecache and inode/dentry caches.

tip: some dirty caches are not freeable so it's better to use "sync" command before using drop_caches for freeing much more memory on your linux system.

disabling bell beep sound on linux console

Sometimes its annoying to hear a beep sound everytime you hit on tab button for completion on linux console. Here is the quickway how to disable that beep sound.
All you have to do is add the following line in /etc/inputrc file or uncomment it if it presents.

set bell-style none

the following image is the default inputrc file on centos system. You can uncomment the line which says "set bell-style none" for disabling tab completion beep sound.

resolving latency problems on linux with latencytop utility

Finding out what is causing unresponsive behaviour, audio delays or stalls on your linux machines can be a hard task.
Latencytop utility gives information where latency is occured to diagnose problem.

Here is an example how to use latencytop on fedora core 9 distribution.

yum install latencytop


By default kernel does not enable latency statistics reporting.You have to enable it by hand.


echo 1 > /proc/sys/kernel/latencytop

this command enables it. disabling it is easy by echo 0 > /proc/sys/kernel/latencytop
latencytop uses the statistics from proc file system. statistics is in /proc/latency_stats file.

here is a screenshot where it shows latency caused disk io on ext3 filesystem and with which pid caused it.





note: if your linux system does not have /proc/latency_stats file. it's either is disabled when kernel is compiled or your kernel does not have latency stats support. In that case you have to patch your kernel an recompile it.
for patch see http://www.latencytop.org/download.php

extending laptop battery lifetime with Intel's Powertop utility

Powertop is designed to pinpoint resources (device drivers, software,etc..) that wakes up idle cpu and lets you extend battery life time by fixing or stopping the application. It's useful for laptop users and also for datacenters who wants to save power consumption.

Here is a sample output disk I/O which showing by powetop:

display system statistics with systat on bsd systems

systat utility displays system statistics on bsd systems (freebsd, dragonfly, etc.).
system statistics includes swap, network interface, disk I/O , virtual memory (vmstat), network utilitzation (mbuf), TCP/IP statistics, network connections (netstat), icmp.

here is some systat options:
pigs,icmp,ip,tcp,ifstat,iostat,swap,mbufs,vmstat,netstat.
following is sample output for "systat -iostat" command:

editing text files with more

You can edit documents when using more. if you press "v" , your default editor will be started on current line. After saving and exiting your editor, you can continue using more.
Default editor is chosen by VISUAL and EDITOR variables. VISUAL environment overrides EDITOR variable.

block I/O debugging in Linux

block_dump option enables you to see all read/write and block dirtying operations on files. It's default value is 0. Setting value to 1 enables block I/O debugging.


echo "1" > /proc/sys/vm/block_dump

You can see the log via dmesg, here is a sample output.



to set if off:

echo "0" > /proc/sys/vm/block_dump

Want to see which memory bank is in use on your system?

System Management BIOS (SMBIOS) pecification reports information of your hardware. In short terms, it tells you what your system bios detects and supports.

In Linux, you can use dmidecode for accessing smbios information. For example, if you like to see which memory banks is in use and what the ram module sizes, you simply call dmidecode with memory parameter.

# dmidecode -t memory

Here is the output on my system, If you look sample output below, you will see total memory supported by your system and max memory module size which can be installed. Also which banks are installed and memory module details like size and speed.


# dmidecode 2.9
SMBIOS 2.4 present.

Handle 0x0005, DMI type 5, 24 bytes
Memory Controller Information
Error Detecting Method: 8-bit Parity
Error Correcting Capabilities:
None
Supported Interleave: One-way Interleave
Current Interleave: One-way Interleave
Maximum Memory Module Size: 1024 MB
Maximum Total Memory Size: 4096 MB
Supported Speeds:
Other
Supported Memory Types:
Other
Memory Module Voltage: 5.0 V
Associated Memory Slots: 4
0x0006
0x0007
0x0008
0x0009
Enabled Error Correcting Capabilities:
None

Handle 0x0006, DMI type 6, 12 bytes
Memory Module Information
Socket Designation: A0
Bank Connections: 1
Current Speed: Unknown
Type: DIMM SDRAM
Installed Size: 1024 MB (Single-bank Connection)
Enabled Size: 1024 MB (Single-bank Connection)
Error Status: OK

Handle 0x0007, DMI type 6, 12 bytes
Memory Module Information
Socket Designation: A1
Bank Connections: 2
Current Speed: Unknown
Type: Unknown
Installed Size: Not Installed
Enabled Size: Not Installed
Error Status: OK

Handle 0x0008, DMI type 6, 12 bytes
Memory Module Information
Socket Designation: A2
Bank Connections: 3
Current Speed: Unknown
Type: Unknown
Installed Size: Not Installed
Enabled Size: Not Installed
Error Status: OK

Handle 0x0009, DMI type 6, 12 bytes
Memory Module Information
Socket Designation: A3
Bank Connections: 4
Current Speed: Unknown
Type: Unknown
Installed Size: Not Installed
Enabled Size: Not Installed
Error Status: OK

Handle 0x001B, DMI type 16, 15 bytes
Physical Memory Array
Location: System Board Or Motherboard
Use: System Memory
Error Correction Type: None
Maximum Capacity: 4 GB
Error Information Handle: Not Provided
Number Of Devices: 4

Handle 0x001C, DMI type 17, 27 bytes
Memory Device
Array Handle: 0x001B
Error Information Handle: Not Provided
Total Width: 64 bits
Data Width: 64 bits
Size: 1024 MB
Form Factor: DIMM
Set: None
Locator: A0
Bank Locator: Bank0/1
Type: Unknown
Type Detail: None
Speed: 667 MHz (1.5 ns)
Manufacturer: None
Serial Number: None
Asset Tag: None
Part Number: None

Handle 0x001D, DMI type 17, 27 bytes
Memory Device
Array Handle: 0x001B
Error Information Handle: Not Provided
Total Width: Unknown
Data Width: Unknown
Size: No Module Installed
Form Factor: DIMM
Set: None
Locator: A1
Bank Locator: Bank2/3
Type: Unknown
Type Detail: None
Speed: Unknown
Manufacturer: None
Serial Number: None
Asset Tag: None
Part Number: None

Handle 0x001E, DMI type 17, 27 bytes
Memory Device
Array Handle: 0x001B
Error Information Handle: Not Provided
Total Width: Unknown
Data Width: Unknown
Size: No Module Installed
Form Factor: DIMM
Set: None
Locator: A2
Bank Locator: Bank4/5
Type: Unknown
Type Detail: None
Speed: Unknown
Manufacturer: None
Serial Number: None
Asset Tag: None
Part Number: None

Handle 0x001F, DMI type 17, 27 bytes
Memory Device
Array Handle: 0x001B
Error Information Handle: Not Provided
Total Width: Unknown
Data Width: Unknown
Size: No Module Installed
Form Factor: DIMM
Set: None
Locator: A3
Bank Locator: Bank6/7
Type: Unknown
Type Detail: None
Speed: Unknown
Manufacturer: None
Serial Number: None
Asset Tag: None
Part Number: None

dmidecode also gives you information about your system's cache, bios and cpu.
Here is a sample output for processor:

# dmidecode -t processor

# dmidecode 2.9
SMBIOS 2.4 present.

Handle 0x0004, DMI type 4, 35 bytes
Processor Information
Socket Designation: Socket 775
Type: Central Processor
Family: Other
Manufacturer: Intel
ID: F6 06 00 00 FF FB EB BF
Version: Intel(R) Core(TM)2 CPU 630
Voltage: 1.3 V
External Clock: 266 MHz
Max Speed: 4000 MHz
Current Speed: 1866 MHz
Status: Populated, Enabled
Upgrade: Socket 478
L1 Cache Handle: 0x000A
L2 Cache Handle: 0x000B
L3 Cache Handle: Not Provided
Serial Number:
Asset Tag:
Part Number:

information about your system cache
# dmidecode -t cache

# dmidecode 2.9
SMBIOS 2.4 present.

Handle 0x000A, DMI type 7, 19 bytes
Cache Information
Socket Designation: Internal Cache
Configuration: Enabled, Not Socketed, Level 1
Operational Mode: Write Back
Location: Internal
Installed Size: 64 KB
Maximum Size: 64 KB
Supported SRAM Types:
Synchronous
Installed SRAM Type: Synchronous
Speed: Unknown
Error Correction Type: Unknown
System Type: Unknown
Associativity: Unknown

Handle 0x000B, DMI type 7, 19 bytes
Cache Information
Socket Designation: External Cache
Configuration: Enabled, Not Socketed, Level 2
Operational Mode: Write Back
Location: Internal
Installed Size: 2048 KB
Maximum Size: 1024 KB
Supported SRAM Types:
Synchronous
Installed SRAM Type: Synchronous
Speed: Unknown
Error Correction Type: Unknown
System Type: Unknown
Associativity: Unknown


# dmidecode -t bios

bios information includes vendor of your bios and it's version.Useful, to see if you are using and old bios and needing and bios update. Also, which devices are supported and which ones can be use for booting up your system.


Read more!

# dmidecode 2.9
SMBIOS 2.4 present.

Handle 0x0000, DMI type 0, 20 bytes
BIOS Information
Vendor: Award Software International, Inc.
Version: F6
Release Date: 05/28/2007
Address: 0xE0000
Runtime Size: 128 kB
ROM Size: 512 kB
Characteristics:
PCI is supported
PNP is supported
APM is supported
BIOS is upgradeable
BIOS shadowing is allowed
Boot from CD is supported
Selectable boot is supported
EDD is supported
5.25"/360 KB floppy services are supported (int 13h)
5.25"/1.2 MB floppy services are supported (int 13h)
3.5"/720 KB floppy services are supported (int 13h)
3.5"/2.88 MB floppy services are supported (int 13h)
Print screen service is supported (int 5h)
8042 keyboard services are supported (int 9h)
Serial services are supported (int 14h)
Printer services are supported (int 17h)
CGA/mono video services are supported (int 10h)
ACPI is supported
USB legacy is supported
LS-120 boot is supported
ATAPI Zip drive boot is supported
BIOS boot specification is supported

Handle 0x001A, DMI type 13, 22 bytes
BIOS Language Information
Installable Languages: 3
n|US|iso8859-1
n|US|iso8859-1
r|CA|iso8859-1
Currently Installed Language: n|US|iso8859-1

fscking large ext2/ext3 volumes

If you got following memory allocation error while fscking a large ext2/ext3 volume on a 32-bit system

/dev/sdaX contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Error allocating directory block array: Memory allocation failed
e2fsck: aborted

You can try one of the following methods:
1- boot with a 64-bit live cd and fsck your volume
2- if you don't have a 64-bit cpu, use your one of the partitions or disks for fsck cache.
edit following file
/etc/e2fsck.conf

and add following lines to point fsck to a directory where it can store it's cache files during fsck.
[scratch_files]
directory = /home/myuser/tmp/

be sure that the directory exists, before running fsck.By this way you will reduce the needed memory for fsck. The disadvantage is disk is slow, therefore fsck will take more time to finish it's job.

Regular expression in Bash

Regular expressions are handy way of quickly manipulating text data or searching any substring or character in it. Almost every language supports it. For example: perl,C,php,java,etc...

Here is some quick tips on regexp with bash.

This if statement is true if string in data variable starts with numeric values.
if [[ $data =~ ^[0-9] ]]; then

^ means start of the string
[0-9] is range from 0 to 9.
[a-z] from a to z but in lower case.
[A-Z] from A to Z but in capital. as you see it's case sensitive.

if string lasts with a digit
if [[ $data =~ [0-9]$ ]]; then

if string starts with a digit and it's not more than 3 digits
if [[ $data =~ ^[0-9]{1,3} ]]; then

{1,3} means 1 character or at most 3 characters of previous declaration

this one means that string starts with lower case letter and ends again with lower case letter
any other character or numeric value fails this if statement

if [[ $data =~ ^[a-z]+*$ ]]; then

$ means end of line
+ means one or more of the preceding element

here is search and replace statements

this one searches character "a" in data variable and changes its first match with "b| character
data=${data/a/b}

this one is exact with previous example with one difference, it searches for all "a" characters and replace all of them with "b" character, please notice double slash.
data=${data//a/b}

Analyzing linux resources with Systemtap

Systemp lets you gather information about your running linux system. So you can diagnose performance and problems on your system(I/O activity, network, sockets,etc.). It provides a command line interface and a scripting language. There is also a gui interface called stapgui on sourceforge.net. Installing it on Ubuntu is straight forward

sudo apt-get install systemtap
sudo apt-get install linux-image-debug-generic
sudo ln -s /boot/vmlinux-debug-$(uname -r) /lib/modules/$(uname -r)/vmlinux

Systemtap uses kprobes to trace events. Kprobes are markers which are placed on predefined points in kernel. For example if you like to see which programs are run by your system you call syscall exec probe.
Here is a simple stap script which shows which programs are run by your system.


#!/usr/bin/env stap
probe syscall.exec* {
printf("exec %s %s\n", execname(), argstr)
}

As you can see, stap is similiar to C language.
The key point is to choose which probe point you would like to use.
It is also possible to probe only which process you want by pid or it's name.For example:
probe process("/bin/bash").syscall

Here is another example, suppose you have a disk I/O activity and want to know which processes are causing this.

#!/usr/bin/env stap

global reads, writes, total_io
probe kernel.function("vfs_read").return {
reads[execname()] = total_io[execname()] += $return
}
probe kernel.function("vfs_write").return {
writes[execname()] = total_io[execname()] += $return
}
probe timer.s(1) {
foreach(p in total_io- limit 10)
printf("%15s r: %8d KiB w: %8d KiB\n",p, reads[p]/1024,writes[p]/1024)
printf("\n")
}


You can read further at systemtap site
http://sourceware.org/systemtap/langref/

Linux Block Tracing

blktrace let's you see detailed information about I/O traffic on a disk devices.

You need to have a 2.6.17-rc1 or newer kernel with "Block io tracing" and debugfs enabled in kernel to use blktrace command.


You can download and install blkrace from http://brick.kernel.dk/snaps/ site or if you use ubuntu you can install it with apt-get.
on ubuntu:

$ sudo apt-get install blktrace


blktrace requires debugfs to run. run mount command to see if it's mounted.
If you can't see debugfs in mount command output for example create a directory called /debugfs and mount it by following command as root

# mount -t debugfs none /debugfs

now you can run blkrace command.


blktrace -r /debugfs/ -d /dev/sda -o - | blkparse -i -

default format of blkparse output is as follows (you can change output format if you like with -f option):





by this way you can get detailed information about ongoing disk activity on your system.

Description of Events:

A IO was remapped to a different device

B IO bounced

C IO completion

D IO issued to driver

F IO front merged with request on queue

G Get request

I IO inserted onto request queue

M IO back merged with request on queue

P Plug request

Q IO handled by request queue code

S Sleep request

T Unplug due to timeout

U Unplug request

X Split

simple load balancing with iptables

iptables has an extension called clusterip. Clusterip extension uses multicast arp feature to achieve load balancing. Let's say we have two web servers called web1(192.168.0.1) and web2(192.168.0.2) and a virtual ip (192.168.0.10) which will be accepting requests for these machines.

virtual ip:192.168.0.10
web1:192.168.0.1
web2:192.168.0.2

Virtual ip will accept the requests and load balance them between these two web servers.

on web1 server run:

# iptables -I INPUT -d 192.168.0.10 -i eth0 -p tcp --dport 80 -j CLUSTERIP --new --clustermac 01:02:03:04:05:06 --total-nodes 2 --local-node 1 --hashmode sourceip
# ifconfig eth0:1 192.168.0.10 netmask 255.255.255.0 up

on web2 server run:
# iptables -I INPUT -d 192.168.0.10 -i eth0 -p tcp --dport 80 -j CLUSTERIP --new --clustermac 01:02:03:04:05:06 --total-nodes 2 --local-node 2 --hashmode sourceip
# ifconfig eth0:1 192.168.0.10 netmask 255.255.255.0 up

only difference between web1 and web2 commands is local-node option as seen above.
now any web requests coming to 192.168.0.10 will be load balanced between web1 and web2.

clusterip supports three hashmodes (sourceip,sourceip-sourceport and sourceip-sourceport-destport) to determine how to route requests to each servers.

This configuration has one drawback. If one of the nodes fall, the other one does not serves incoming requests for the other one. You need to install linux-ha.

If you want to see which requests served by web1 for example, simply run
# cat /proc/net/ipt_CLUSTERIP/192.168.0.1

Let's say web2 is crashed and we would like to first web server (web1) to take care the requests coming to web2.
on web1 server run:
# echo "+2" >> /proc/net/ipt_CLUSTERIP/192.168.0.1
now on, web1 will take the requests coming to web1.
When you up the web2 server, just run
# echo "-2" >> /proc/net/ipt_CLUSTERIP/192.168.0.1
and web1 will not serve the request for web2.

clamsmtp rbl support patch

clamsmtp is lightweight anti virus smtp proxy. You can put it in front or back of your smtp server. But if you run it in front of your smtp server and relay traffic back to your original smtp server, you cannot use any rbl server.

So, I decided to write a patch which adds this new rbl option to clamsmtpd server. You can download patch from here clamsmtp rbl patch

qmail linefeed problems hotmail , dcc and others

While I was testing DCC Checksum , I discovered that some email messages are truncated and dcc reports that body part is missing with an error message "missing message body". I figured out that not only the body part is missing also the headers also truncated.

A quick investigation with tcpdump showed that spamc is sending truncated messages to spamd server which is passed to dcc server. A second tcpdump run on smtp server showed that some messages is rejected by qmail server with "451 See http://pobox.com/~djb/docs/smtplf.html." message. This was hitting a small amount of mails but the ones especially coming from hotmail.com.

I have quickly fixed the problem with djb's fixcrio application which comes with DJB's ucspi-tcp package. No more "missing message body" error messages anymore with dcc :)

Running fixcrio is easy. Just put the command before qmail-smtpd where you run it.

VI tricks part one

I have finished first part of my VI editor tricks article. Part one explains main commands and tricks like sort data, replacing selected lines and feeding vi document content as input to any program while getting output as new document.
You can read VI tricks part one here.

FreeBSD GJournal (Filesystem Journal)

Freebsd 7.0 has a new geom extenstion called gjournal. You can easily create a journaled filesystem with gjournal on bsd systems.
First of all, gjournal support has to be built in kernel. It's currently built in 7.x series. The kernel option is called UFS_GJOURNAL

You should disable soft updates on filesystems where gjournal is used. Because journalling takes place and there is no need to use soft updates for data recovery on a crash.

While the filesystem is journalled, the mount option async will boost the performance.


For example, let's say we have a disk called ad1 and it has existing filesystem on it.

All we have to do is create a label.

# gjournal label -f /dev/ad1

notice the -f option. It will force convert operation of an existing file system to journal. Journal device will be separate and called in this example as /dev/ad1.journal.

now it's time to load geom gjournal kernel module.

# gjournal load

now add journalling support to our existing filesystem.

# tunefs -J enable /dev/ad1.journal

it's better now to disable soft updates.

# tunefs -n disable /dev/ad1.journal


now we mount our new journalled filesystem with async option.

# mount -o asyn /dev/ad1.journal /mnt

in our example, we don't have any slices on our disk. In slice case new devices will be created (eg. /dev/ad1s1.journal)

Sqlite compression

Sqlite database has no built-in compression support. But it has a feature called loadable extension, which let's developers to create any addon feature for sqlite database. I have wrote some documentation about sqlite loadable extension and how to add compress/uncompress ability to sqlite database to compress your data's on the fly.
Here is my article

ucspi-tcp mysql patch

Finally, I had spare time to finish my ucspi-tcp mysql patch . Putting tcpserver rules in a mysql table has many advantages like centralized access rule lists and easy management.