Showing posts with label Solaris. Show all posts
Showing posts with label Solaris. Show all posts

Sunday, May 31, 2009

Mirroring your OpenSolaris root disk with ZFS

Solaris/OpenSolaris supports booting with ZFS filesystem. For example, if you look into your opensolaris insallation you'll notice the zfs rpool (root pool). ZFS is default filesystem on opensolaris (optional with Solaris). If you look into detailts of that pool with zpool utility , you'll see that it's a single disk or a partition (depends how you partitioned your disk on installation process of opensolaris). If that single disk crashes, you may lost your data and operating system.That's one point of failure. The simple solution is to convert our zpool into RAID-1 with a secondary disk (with same capacity or bigger one).









As you can see, our zpool contains a single disk called c4t0d0. Let's add a new disk to our rpool to convert it into mirror. zpool will do it automatically for us. One important point here is that Solaris doesn't support boot devices to be in EFI label format. So, if your new disk label is in EFI format, you need to convert it into SMI label. So, I'll use fdisk command to create a Solaris2 partition and remove any EFI labels if present. My new disk name is c4t1d0.






After creating the partition which covers whole disk area, I copied partition map of original rpool member c4t0d0 to our new disk c4t1d0.




Now, it's time to attach our brand new disk into rpool. After attaching the disk, our rpool pool will be converted into mirror (raid-1). Except one thing, the grub boot manager. We need to install it into new disk by hand using installgrub command.





Now, we have created our mirror on rpool. It will take time to sync original disk to new disk. You can watch this process by using zpool's status command.

Saturday, May 16, 2009

Configuring ILOM manager on Sun X4540 Thumper

Sun X4540 Thumper storage server provides 48TB hard disk capacity on one server. It has 2 AMD Quad x64 Opteron processors and 32GB of memory. Thumper has 48 hard disk slots. You can use 1TB sata hard disks in each slot. This will give you 48TB hard disk area as you can see in following photo.


You can run Solaris,OpenSolaris, Linux and Windows operating systems on Thumper. Thumper includes Sun Microsystem's ILOM (Sun Integrated Lights Out Manager) manager. You can use ILOM to upgrade your server's firmware, remotely accessing your server's console by using your web browser, modify and see sensors, see hardware components and so on. You can access ILOM by using a serial console,ssh client or a web browser.You need to set an ip address for your ILOM manager if you need to use ssh or a web browser.If you don't have a dhcp server around you'll need to configure your ILOM manager with a static ip address. I have used a serial console connected on Thumpers "Ser MGT" port and on the other side a linux desktop and minicom application. You need to change your minicom settings to 9600 baud rate , 8 data bits, no parity and one stop bit (9600-8-N-1) with no hardware and software flow control.


Yes, that's right, It's Linux :) . Sun ILOM uses embeded Linux. When you connect to ILOM by a serial console you will need to login in by using ILOM's factory default login name "root" and password "changeme". ILOM has a simple built-in command interface. You can use "help" command for details when you logged in ILOM manager. Setting the static ip address of ILOM is simple. I captured the screenshot when I gave one of my local ip addresses to Thumper's ILOM.



After, you commit your static ip network settings settings with "set commitpending=true" command, you need to connect Thumper's "net mgt" ethernet interface to your network.Now, you can access ILOM by using any ssh client or a web browser. My plan is to use OpenSolaris and 48TB ZFS Raid-Z on this storage server and see the results on a production environment :)

Tuesday, April 14, 2009

Process IO Top utility with Solaris DTrace

DTrace is dynamic tracing framework created by Sun Microsystems for Solaris OS. DTrace let's any system administrator or developer to get internal view of how an operating system works, find any system bottlenecks, examine a live system or process, get information on system calls, network internals and many other things. For this purpose DTrace has many providers for specific tasks. You can get list of the DTrace providers and probe function names by running following command:

# dtrace -l

DTrace also adopted to other operating systems like FreeBSD and Mac OS X. DTrace uses D Language which is subset of C programming language. You can find more information about DTrace on Sun's BigAdmin Portal.

Following example demostrates you, how easy to find the information you want on Solaris with DTrace.I wrote it to see which files are most written or read by which processes on the Solaris.


#!/usr/bin/bash
# process_io_top - show procesess by top read/write KB I/O per file
# Written by Levent Serinol (lserinol@gmail.com)
# http://lserinol.blogspot.com
# Apr/14/2009
#
# USAGE: process_io_top [-s interval] [-p pid]
#
# -s interval # gather and show statistics in given interval (seconds)
# -p pid # show read/write KB I/O just for given PID
# -h # show usage information
#
# eg:
# process_io_top -s 10
#
#
#
####################################################################################
interval=5
show_pid=0
pid=0

function usage()
{
echo "
USAGE: io.sh [-s interval] [-p pid]
-s # set interval, default is 5 seconds
-p pid # pid
eg,
io -p 630 # io activity of pid 630
io -s 10 # refresh output in every 10 seconds";
}

while getopts h:p:s:a name
do
case $name in
p) show_pid=1; pid=$OPTARG ;;
s) interval=$OPTARG ;;
h|?) usage;
exit 1
esac
done

/usr/sbin/dtrace -Cws <( cat <<EOF

inline int PID = $pid;
inline int SHOW_PID = $show_pid;
inline int INTERVAL = $interval;


#pragma D option quiet
#pragma D option aggsortrev


dtrace:::BEGIN
{
secs = INTERVAL;
printf("Please wait....\n");
}

io:::start
/ (SHOW_PID == 1) && ( pid == PID) /
{
self->rw=args[0]->b_flags & B_READ ? "R" : "W";
@files[pid,execname,self->rw,args[2]->fi_pathname] = sum (args[0]->b_bcount);
@total_blks[self->rw]=count();
@total_bytes[self->rw]=sum (args[0]->b_bcount);
self->rw=0;
}

io:::start
/ SHOW_PID == 0 /
/* SHOW_PID == 0 && args[2]->fi_pathname != "<none>" */
{
self->rw=args[0]->b_flags & B_READ ? "R" : "W";
@files[pid,execname,self->rw,args[2]->fi_pathname] = sum (args[0]->b_bcount);
@total_blks[self->rw]=count();
@total_bytes[self->rw]=sum (args[0]->b_bcount);
self->rw=0;
}

profile:::tick-1s
{
secs--;
}


profile:::tick-1s
/secs == 0/
{

trunc(@files,30);
normalize(@files,1024);
system("/usr/bin/clear");
printf("%Y ",walltimestamp);
printa("%s %@11d blocks, ",@total_blks);
printa("%s %@11d bytes, ",@total_bytes);
printf("\n%6s %-12s %3s %8s %3s\n", "PID", "CMD","R/W", "KB", "FILE");
printa("%6d %-12.12s %1s %@10d %s\n",@files);
secs = INTERVAL;

}
dtrace:::END
{
trunc(@files);
}
EOF
)



You can download it here process_io_top. This scripts briefly shows you how to pass arguments from shell to DTrace. Using aggregations and sort them in reverse order by using pragma "aggsortrev". Defining probes (io:::start) more than once with different predicate conditions. Following is a sample output from process_io_top script.

Saturday, March 7, 2009

measuring your network throughput on Solaris

You can use nicstat utility to measure your network device utilization and throughput. The utility itself uses solaris kstat for network device statistics information. Statistics include network read and write in kilobytes, packets and device utilization.You can compile nicstat with gcc as shown below.

# gcc nicstat.c -o nicstat -lkstat -lgen -lsocket

Following image shows a sample output from nicstat utility.


getting network interface statistics on Solaris

There are many ways to get network interface status and statistics on Solaris. I'll show some of them. First one is well known command netstat. netstat's "-i" parameter will give you statistics of devices used for ip traffic. Statistics include input/output total packets,errors and collisions for each network device.


Next method is by using solaris ndd command. First, we have to find our device name. This can be done by ifconfig command.
My network device name is "e1000g0" as seen in sample output above. Now, it's time to ask ndd which parameters are supported by this device. We do this by calling ndd (ndd /dev/e1000g0 \?) with our device name and "\?" parameter to display every supported parameter.


ndd let's you change some interface settings. They're marked as "read and write". The parameter names clearly shows what they are for. As you can see in sample output below, ndd can give us link speed, link status (cable plugged/unplugged),duplex mode and autoneg status. Following sample output shows us that my network interface link status is 1, which means the network cable is plugged,Network speed is 1000mbit and auto negotiation is on.
The other command is called dladm. dladm can give us above results with one command and in easily readable format. Also, it can give network interface statistics like netstat with "-s" parameter except collisions.

Saturday, February 21, 2009

Solaris Interrupts

In this post, I'll be shown you how to access irq table on a Solaris system and how to get statistics about the number of interrupts produced on Solaris. We'll use two commands intrstat and mdb.
intrstat command shows how many interrupts produced by each device and how many of them handled by each cpu on the system. "%tim" column shows the time where spend by that driver's interrupt handler on processing cpu. You can pass a parameter to intrstat and tell it to gather information in given intervals (in seconds). Following is a sample output of intrstat where it gets statistics in 5 seconds interval.

Now, after getting interrupt statistics we can now examine the irq table on Solaris system. For this job, we'll use solaris mdb (Solaris modular debugger). We can access irq table through the kernel. So, we'll pass the "-k" parameter to mdb. This will tell it to get in kernel debugging mode. mdb will use /dev/ksyms and /dev/kmem to access Solaris kernel. After running "mdb -k", type "::interrupts" command to acess irq table in Solaris system. Here is a sample output:

This table shows us which driver(s) (ISR) using which irq. Share column shows how many devices sharing the same irq. IPL gives irq priority. Type column show irq type used Fixed (legacy), MSI, IPI interprocessor interrupt (xcalls - cross call).

disk error statistics with Solaris

Solaris iostat command normally shows you disk activity statistics and cpu utilization information. But it has another useful option which shows you disk error statistics. Those parameters are "-e" and "-E"."-e" options show summary disk error statistics for soft, hard and trasnportation errors. The other option "-E" shows all errors on the disks and information about the devices like the vendor,model,size and media erros.

Tuesday, February 17, 2009

Stopping and Running process on the fly with Solaris/OpenSolaris

Solaris/OpenSolaris has two utilities called pstop and prun which has ability to stop a running process and run it again. Both of the ulities takes pid or pid/lwp (light weight process - thread) as parameter. As the name says, pstop stops a running process when given an pid of a process. Given the same process id of a stopped process to prun , makes the process runnable again.

Booting OpenSolaris in verbose mode

When OpenSolaris boots it does not show kernel messages on the console by default. You can see them by passing an argument to kernel when booting. OpenSolaris uses grub boot loader. On the grub menu press "e" to enter edit mode.


Move to the line where kernel parameters are written with cursor keys.


Press "e" again for edit mode for the grub. Than add "-v" parameter to the end of kernel parameters.



After adding "-v" verbose parameter to kernel press "enter" and than "b" to boot OpenSolaris with verbose mode enabled.


If you need this option to be permanent, you need to modify grub config file in OpenSolaris. It's located in /boot/grub/menu.lst . If you use zfs root, than it's located in /rpool/boot/grub/menu.lst.You can do this on a Sun Sparc machine with Solaris when the openboot procedure begins by pressing "STOP-A" combination and type "boot -v"

Tuesday, February 3, 2009

Solaris pargs command

Solaris pargs gives you environment and arguments information for any given process id. If you pass just a process id (pid) to pargs, it gives you all argument variables of the running process. If you need to have arguments as a single line for using it with for example on shell use -l argument with it. Also pargs can give you environment information of any running process if you run it with -e parameter.
Sometimes this utility can be handy if you cannot see the parameters or environment variables of a process with other methods like ps or any other well known utilities.

pargs