Thursday, March 3, 2016

Few custom scala hive udfs I wrote to simply some my sql queries on Hive.
Git repo link : https://github.com/lserinol/Hive-udfs


Hive-udfs

Collection of my Scala Hive UDFs

Requirements

  • SBT
  • Scala
  • Hive

Compile

  1. git clone/fork project https://github.com/lserinol/Hive-udfs.git 
  2. sbt assembly 
  3. add jar 'target/scala-version number/levent-hive-funcs-assembly-version number.jar' to HIVE_AUX_JARS_FILE_PATH 
    or use "add jar" command for Hive-CLI/Beeline

Registering UDFs

create temporary function lcrc32 as 'com.levent.hive.udfs.lcrc32'; 
create temporary function ldays_between as 'com.levent.hive.udfs.ldays_between'; 
create temporary function lmonths_between as 'com.levent.hive.udfs.lmonths_between'; 
create temporary function ilike as 'com.levent.hive.udfs.liLIKE'; 

com.levent.hive.udfs.lcrc32

  • takes a string input and returns crc32(long) value 

com.levent.hive.udfs.ldays_between

  • calculates days difference between two dates and returns number of days(int) value 

com.levent.hive.udfs.lmonths_between

  • calculates months difference between two dates and returns number of months(int) value 

com.levent.hive.udfs.liLIKE

  • Hive ilike function in-case sensitive like between two strings retuns boolean TRUE/FALSE 

Monday, May 6, 2013

Which processes are using your swap ? (Linux)

Here is a simple bash script which briefly shows you the processes using your Linux's swap area.
#!/bin/bash

exec 2>/dev/null
total=0
printf "pid\t\tprocess\t\tmemory (kB)\n"
while read -r me
do
 ps=$(cat $me/comm)
 res1=$?
 val=($(grep "^VmSwap" $me/status))
 res2=$?
 p=$(basename $me)
 if [[ $res1 -eq 0 && $res2 -eq 0 ]]; then
  if [ ${val[1]} -ne 0 ]; then
   printf "%d\t\t%s\t\t%s\n" "$p" "$ps" "${val[1]}"
   let total=$total+${val[1]}

  fi
 fi
done < <(find /proc/ -mindepth 1 -maxdepth 1 -type d -name "[0-9]*" )
echo "Total(kB):$total"


Sample output (processes not sorted by their memory usage)

Thursday, September 20, 2012

Ext4 sysfs parameters

Have you ever wondered how much data written to your Ext4 filesystem since it's first creation time or last mount time. Here is two ext4 sysfs parameters that will tell you the magic numbers you need.
/sys/fs/ext4//lifetime_write_kbytes
/sys/fs/ext4//session_write_kbytes
Here is some more ext4 sysfs paramters and their meanings, but mostly tuning of ext4 multiblock allocator. ext4 sysfs paramters

Tuesday, March 13, 2012

Freebsd ciss driver logical drive limit

Lately, I was faced with a strange problem on a HP server which uses HP Smart Array controller P800 which is running Freebsd. All my 24 hard drives which is hosted on externel HP enclosures was presented as 24 single raid0 logical volumes. Surprisingly, when the freebsd booted it doesn't presented the disks and disabled the ciss driver (HP Smart Array Controller) with following message

Mar 12 16:00:41 freebsd kernel: ciss1: adapter claims to report absurd
number of logical drives (24 > 15)

A quick check on the driver source code showed up that there is a hard limit defined to 15.(/usr/src/sys/dev/ciss/cissvar.h)

 #define CISS_MAX_LOGICAL        15


So I wanted to be sure if I change the number to a higher value , won't do any unwanted behavior later on the system. In that point I contacted Paul Saab where he explained the limit as  below.
 really that's done to limit the amount of memory needed upfront by the
driver.  I believe you can easily increase the number of drives
without issue as long as you have enough memory below 4GB.  Parts of
the ciss driver require that the memory you DMA from be under 4G


After the clarification, I just set the number to 32 and rebuild/install the kernel, restart the machine and a quick check on dmesg showed up all my 24 logical volumes presented on HP P800 controller.
Mar 13 10:32:37 freebsd kernel: da19: 286070MB (585871964 512 byte sectors: 255H 32S/T 65535C)
Mar 13 10:32:37 freebsd kernel: da20 at ciss1 bus 0 scbus2 target 19 lun 0 
Mar 13 10:32:37 freebsd kernel: da20:  Fixed Direct Access SCSI-5 device
Mar 13 10:32:37 freebsd kernel: da20: 135.168MB/s transfers
Mar 13 10:32:37 freebsd kernel: da20: Command Queueing enabled
Mar 13 10:32:37 freebsd kernel: da20: 286070MB (585871964 512 byte sectors: 255H 32S/T 65535C)
Mar 13 10:32:37 freebsd kernel: da21 at ciss1 bus 0 scbus2 target 20 lun 0 
Mar 13 10:32:37 freebsd kernel: da21:  Fixed Direct Access SCSI-5 device
Mar 13 10:32:37 freebsd kernel: da21: 135.168MB/s transfers
Mar 13 10:32:37 freebsd kernel: da21: Command Queueing enabled
Mar 13 10:32:37 freebsd kernel: da21: 286070MB (585871964 512 byte sectors: 255H 32S/T 65535C)
Mar 13 10:32:37 freebsd kernel: da22 at ciss1 bus 0 scbus2 target 21 lun 0 
Mar 13 10:32:37 freebsd kernel: da22:  Fixed Direct Access SCSI-5 device
Mar 13 10:32:37 freebsd kernel: da22: 135.168MB/s transfers
Mar 13 10:32:37 freebsd kernel: da22: Command Queueing enabled
Mar 13 10:32:37 freebsd kernel: da22: 286070MB (585871964 512 byte sectors: 255H 32S/T 65535C)
Mar 13 10:32:37 freebsd kernel: da23 at ciss1 bus 0 scbus2 target 22 lun 0 
Mar 13 10:32:37 freebsd kernel: da23:  Fixed Direct Access SCSI-5 device
Mar 13 10:32:37 freebsd kernel: da23: 135.168MB/s transfers
Mar 13 10:32:37 freebsd kernel: da23: Command Queueing enabled
Mar 13 10:32:37 freebsd kernel: da23: 286070MB (585871964 512 byte sectors: 255H 32S/T 65535C)
Mar 13 10:32:37 freebsd kernel: da24 at ciss1 bus 0 scbus2 target 23 lun 0 
Mar 13 10:32:37 freebsd kernel: da24:  Fixed Direct Access SCSI-5 device
Mar 13 10:32:37 freebsd kernel: da24: 135.168MB/s transfers
Mar 13 10:32:37 freebsd kernel: da24: Command Queueing enabled
Mar 13 10:32:37 freebsd kernel: da24: 286070MB (585871964 512 byte sectors: 255H 32S/T 65535C)

Tuesday, February 28, 2012

Linux like dmesg in Freebsd

UPDATE:  this patch is applied to Freebsd repository on  10 May 2013
http://svnweb.freebsd.org/base?view=revision&revision=250430


In my previous post, I was talking about the "dmesg -c" equivalent in FreeBSD. Simply it was a sysctl command. So, I decided to simply change FreeBSD dmesg command and make it act like in as Linux. All you have to do is download my patch and apply it as shown below.I was tested the patch on FreeBSD 9 btw.
mybsd# pwd
/usr/src/sbin/dmesg
mybsd# patch < dmesg.patch 
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|--- dmesg.c.org        2012-01-03 19:04:44.000000000 +0200
|+++ dmesg.c    2012-02-28 11:36:22.000000000 +0200
--------------------------
Patching file dmesg.c using Plan A...
Hunk #1 succeeded at 81.
Hunk #2 succeeded at 194.
Hunk #3 succeeded at 204.
done
mybsd# make 
Warning: Object directory not changed from original /usr/src/sbin/dmesg
cc -O2 -pipe  -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -c dmesg.c
cc -O2 -pipe  -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign  -o dmesg dmesg.o -lkvm


Friday, February 17, 2012

Clearing FreeBSD Kernel Message Buffer

As you can clear kernel message buffer on Linux with dmesg command " dmesg -c ", it's a little bit different on a BSD system.It's done by sysctl as following
# sysctl kern.msgbuf_clear=1

Monday, February 8, 2010

Detailed network interface statistics on Linux

Here is a helpful command for getting interface statistics on your linux box. It's called ip. It's bundled with iproute package. You can see information about dropped, collisions,crc,transmit and other info about your network interface.


root@helga:~# ip -s -s link show eth0
2: eth0: mtu 1500 qdi
WN qlen 1000
link/ether 00:24:8c:d5:6a:a7 brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped overrun mcast
1633058095 16933162 0 0 0 0
RX errors: length crc frame fifo missed
0 0 0 0 0
TX: bytes packets errors dropped carrier collsns
1994496524 5588574 0 0 0 0
TX errors: aborted fifo window heartbeat
0 0 0 0

Friday, September 11, 2009

I/O usage per process on Linux

Linux kernel 2.6.20 and later supports per process I/O accounting. You can access every process/thread's I/O read/write values by using /proc filesystem. You can check if your kernel has built with I/O account by just simply checking /proc/self/io file. If it exists then you have I/O accounting built-in.


$ cat /proc/self/io
rchar: 3809
wchar: 0
syscr: 10
syscw: 0
read_bytes: 0
write_bytes: 0
cancelled_write_bytes: 0


Field Descriptions:

rchar - bytes read
wchar - byres written
syscr - number of read syscalls
syscw - number of write syscalls
read_bytes - number of bytes caused by this process to read
from underlying storage
write_bytes - number of bytes caused by this process to written from
underlying storage
As you know, ever process is presented by it's pid number under /proc directory. You can access any process's I/O accounting values by just looking /proc/#pid/io file. There is a utility called iotop which collects these values and shows you in like top utility. You see your processes I/O activity with iotop utility.

Friday, September 4, 2009

problem compiling Mysql 5.1 with sphinx engine support

Mysql's full-text search engine is not very powerful and support comes only with Myisam engine. Sphinx is a full-text search engine replacement for Mysql and Postgresql. Also, It's possible to use Sphinx full-text search engine with both Myisam and InnoDB engines. You can read detailed information about mysql compilation with sphinx on Sphinx site.

I have faced with following error message, while compiling Mysql with innodb and sphinx engine support ( ./configure --prefix=/data/mysql3/ --with-plugins=sphinx,innobase). I used the innodb (base) which bundled with Mysql. You can also compile innodb plugin which has additional features. For example, on the fly compression, speed improvements.

configure: error: unknown plugin: sphinx

The trick was to run autorun.sh command in Mysql's source tree before running configure with sphinx engine support. This script will create a new configure script which knows how to compile sphinx engine.


# ./BUILD/autorun.sh


Thursday, September 3, 2009

nethogs network utility

Nethogs is a small but very useful utility, if you like to see your machine's bandwidth usage per processes. Here is a simple screenshot from my linuxbox.

Sunday, July 12, 2009

Fast Copy on Unix systems

I needed to copy so many small files and directories (1.2 TB) to another machine with netcat (nc). But I faced with a problem when I tried to run nc program on the background with bash. Bash was suspending standard input when I move nc process to background job, this was causing nc processess to die when standard input was blocked. So, I have found another netcat like utility called socat. Socat is not stop execution while it was working as background job on bash shell. I was able to utilize 100mbit ethernet with few socat processes on server and client side. First of all, I run socat processes on destination server as daemons to listen specific ports. Following, command will run socat in daemon mode to listen on tcp port 4123 and pipe incoming data to tar command on the background.

# (socat -u tcp4-listen:4123 - | tar xzp -C .) &

On source server, I run socat to feed incoming tar data to destination socat daemon where it listens on tcp port 4123. My destination server's ip adddess was 192.168.36.199 .

# (tar -czp - /mydir/1/ | socat -u - tcp4:192.168.3.199:4123)&

So, with a simple bash script I was able to run few socat daemons on destination server with different ports and feed them from source server to achieve higher data transer rate.

Monday, June 8, 2009

socket programming with Bash

I have many memcached servers around. So, I needed a tool to check status information on these servers like top utility. I decided to write it in bash scripting language (100% in bash except printf,date and sleep commands :) ). You can simply enhance the script for example getting server names and ports from a config file or display interval parameter from command line. Memcached supports tcp and udp protocols for communication with clients. I used tcp communication in my bash script. Here is information about Memcached's supported server commands. My example script uses "stats" command. Here is sample output for "stats" command from one memcached server.




I have used the information provided by "stats" command. Script uses pure bash builtin commands to access a tcp server like memcached.

You can download the script from here.


Here is sample output from the script.



If you're using Ubuntu, probably you will not be able to run this script, because bash is not compiled with "--enable-net-redirections" parameter on Ubuntu systems. RedHat based distributions has no problem with bash net-directions like Fedora,Centos and RedHat ES.

Sunday, May 31, 2009

Mirroring your OpenSolaris root disk with ZFS

Solaris/OpenSolaris supports booting with ZFS filesystem. For example, if you look into your opensolaris insallation you'll notice the zfs rpool (root pool). ZFS is default filesystem on opensolaris (optional with Solaris). If you look into detailts of that pool with zpool utility , you'll see that it's a single disk or a partition (depends how you partitioned your disk on installation process of opensolaris). If that single disk crashes, you may lost your data and operating system.That's one point of failure. The simple solution is to convert our zpool into RAID-1 with a secondary disk (with same capacity or bigger one).









As you can see, our zpool contains a single disk called c4t0d0. Let's add a new disk to our rpool to convert it into mirror. zpool will do it automatically for us. One important point here is that Solaris doesn't support boot devices to be in EFI label format. So, if your new disk label is in EFI format, you need to convert it into SMI label. So, I'll use fdisk command to create a Solaris2 partition and remove any EFI labels if present. My new disk name is c4t1d0.






After creating the partition which covers whole disk area, I copied partition map of original rpool member c4t0d0 to our new disk c4t1d0.




Now, it's time to attach our brand new disk into rpool. After attaching the disk, our rpool pool will be converted into mirror (raid-1). Except one thing, the grub boot manager. We need to install it into new disk by hand using installgrub command.





Now, we have created our mirror on rpool. It will take time to sync original disk to new disk. You can watch this process by using zpool's status command.

Saturday, May 23, 2009

FreeBSD and procfs

Many Unix systems have support for proc file system (process file system).Procfs filesystem type is pseudo. FreeBSD is one of that Unix systems. Unlike Linux, which has information other than processes, FreeBSD procfs support is only about the processes on the system. FreeBSD doesn't mount the procfs on boot by default. You need to manually add it to fstab for auto mount on boot or mount it by command for temporarily usage.Common mount point for procfs is /proc on Unix systems.


echo "none /proc procfs rw 0 0" >> /etc/fsab



mount -t procfs none /proc


Every process is presented as directories named by it's pid number on the /proc mount point.
Procfs gives information on running processes on the system like memory mapping,command line arguments of running process, process resource limits and many other.Following is a sample procfs directory structure on a FreeBSD machine.



As you can see, every pid is represented as a directory in procfs. Every directory contains following files. Some of the files are write only or read only where you read information or send information to process.



- status (read-only) : returns process status
- mem (read/write): virtual memory image of the process
- file (depends) : symbolic link to running process
- regs (read/write): process registers
- ctl (write-only): used to send signal to process or
attach/deattach it for debugging
- cmdline (read-only) : command line arguments of running process
- rlimits (read-only) : current resource limits of running process
- map (read-only) : memory mappings of the running process.
- etype (read-only) : type of the executable (eg. FreeBSD ELF32)
- fpregs (read/write): floating point registers



Some of the information provided by these files are in binary format.For example "regs" and "fpregs" files are in binary format. They depend on the architecture of the underlying machine (i386, amd64,sparc64,etc..). Following is the format of the "regs" file on the i386 machine.


struct reg {
unsigned int r_fs;
unsigned int r_es;
unsigned int r_ds;
unsigned int r_edi;
unsigned int r_esi;
unsigned int r_ebp;
unsigned int r_isp;
unsigned int r_ebx;
unsigned int r_edx;
unsigned int r_ecx;
unsigned int r_eax;
unsigned int r_trapno;
unsigned int r_err;
unsigned int r_eip;
unsigned int r_cs;
unsigned int r_eflags;
unsigned int r_esp;
unsigned int r_ss;
unsigned int r_gs;
};


You can use "cat" command to read information provided by procfs for text based information unlike the ones I mentioned above in binary format like "regs","fpregs" and "mem".



[root@freebsd ]# cat cmdline
/usr/sbin/moused-p/dev/ums0-tauto-I/var/run/moused.ums0.pid


You can check a running process's resource limits by looking into rlimit file.



[root@freebsd ]# cat rlimit
cpu -1 -1
fsize -1 -1
data 536870912 536870912
stack 67108864 67108864
core -1 -1
rss -1 -1
memlock -1 -1
nproc 5547 5547
nofile 11095 11095
sbsize -1 -1
vmem -1 -1


First digits is minimum and last one is maximum values of the given resource name. "-1" means infinite. For examle, nofile (open file descriptor) limit for this process is 11095 as minimum and maximum.

"status" file gives information about process status as follows.

- command name
- pid
- parent pid
- process group id
- session id
- major/minor of the terminal, "-" if no terminal is in action
- process flags
- process start time in seconds and microseconds separated by comma
- user time in seconds and microseconds separated by comma
- system time in seconds and microseconds separated by comma
- wait channel name
- effective userid and group lists separated by comma


Following is "cat status" result of a process.
[root@freebsd ]# cat status 
svscan 28246 1 28245 0 ttyp0 noflags 1242921860,839572 0,318052 3,263576 nanslp 0 0 0,0,0,5 -

Saturday, May 16, 2009

Configuring ILOM manager on Sun X4540 Thumper

Sun X4540 Thumper storage server provides 48TB hard disk capacity on one server. It has 2 AMD Quad x64 Opteron processors and 32GB of memory. Thumper has 48 hard disk slots. You can use 1TB sata hard disks in each slot. This will give you 48TB hard disk area as you can see in following photo.


You can run Solaris,OpenSolaris, Linux and Windows operating systems on Thumper. Thumper includes Sun Microsystem's ILOM (Sun Integrated Lights Out Manager) manager. You can use ILOM to upgrade your server's firmware, remotely accessing your server's console by using your web browser, modify and see sensors, see hardware components and so on. You can access ILOM by using a serial console,ssh client or a web browser.You need to set an ip address for your ILOM manager if you need to use ssh or a web browser.If you don't have a dhcp server around you'll need to configure your ILOM manager with a static ip address. I have used a serial console connected on Thumpers "Ser MGT" port and on the other side a linux desktop and minicom application. You need to change your minicom settings to 9600 baud rate , 8 data bits, no parity and one stop bit (9600-8-N-1) with no hardware and software flow control.


Yes, that's right, It's Linux :) . Sun ILOM uses embeded Linux. When you connect to ILOM by a serial console you will need to login in by using ILOM's factory default login name "root" and password "changeme". ILOM has a simple built-in command interface. You can use "help" command for details when you logged in ILOM manager. Setting the static ip address of ILOM is simple. I captured the screenshot when I gave one of my local ip addresses to Thumper's ILOM.



After, you commit your static ip network settings settings with "set commitpending=true" command, you need to connect Thumper's "net mgt" ethernet interface to your network.Now, you can access ILOM by using any ssh client or a web browser. My plan is to use OpenSolaris and 48TB ZFS Raid-Z on this storage server and see the results on a production environment :)

Sunday, May 3, 2009

getting detailed process information on Freebsd

Freebsd procstat utility gives detailed information about all of the processes on the system or just for a given process id number, such as virtual memory mapping, thread stack, command line arguments and open files.

Running procstat with "-a" argument prints information like pid,pid,login, process name, wchan (which event the process waiting) of all processes on the system.

# procstat -a



Procstat "-c" option shows you the command line arguments of a process and "-f" option shows opened files by given process.Following sample output shows the command line arguments and the files currently opened with their permissions by vi process.



You can access virtual memory mapping information a process with "-v" option. Following sample shows vi process virtual memory mappings.



Finally, "-k" option shows kernel threads stacks details of given process.