tag:blogger.com,1999:blog-16100417787328879942024-03-18T20:04:32.598-07:00Levent Serinol's BlogAnonymoushttp://www.blogger.com/profile/06161379979453710759noreply@blogger.comBlogger71125tag:blogger.com,1999:blog-1610041778732887994.post-44865557205053958062016-03-03T05:29:00.000-08:002016-03-03T05:29:15.512-08:00Few custom scala hive udfs I wrote to simply some my sql queries on Hive.<br />
Git repo link : <a href="https://github.com/lserinol/Hive-udfs">https://github.com/lserinol/Hive-udfs</a><br />
<br />
<br />
<h1 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 2.25em; line-height: 1.2; margin-bottom: 16px; margin-left: 0px; margin-right: 0px; margin-top: 0px !important; padding-bottom: 0.3em;">
Hive-udfs</h1>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
Collection of my Scala Hive UDFs</div>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/lserinol/Hive-udfs#requirements" id="user-content-requirements" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" role="img" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Requirements</h2>
<ul style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px; margin-top: 0px; padding: 0px 0px 0px 2em;">
<li style="box-sizing: border-box;">SBT</li>
<li style="box-sizing: border-box;">Scala</li>
<li style="box-sizing: border-box;">Hive</li>
</ul>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/lserinol/Hive-udfs#compile" id="user-content-compile" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" role="img" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Compile</h2>
<ol style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px; margin-top: 0px; padding: 0px 0px 0px 2em;">
<li style="box-sizing: border-box;">git clone/fork project <a href="https://github.com/lserinol/Hive-udfs.git" style="box-sizing: border-box; color: #4078c0; text-decoration: none;">https://github.com/lserinol/Hive-udfs.git</a> </li>
<li style="box-sizing: border-box;">sbt assembly </li>
<li style="box-sizing: border-box;">add jar 'target/scala-version number/levent-hive-funcs-assembly-version number.jar' to HIVE_AUX_JARS_FILE_PATH <br style="box-sizing: border-box;" />or use "add jar" command for Hive-CLI/Beeline</li>
</ol>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/lserinol/Hive-udfs#registering-udfs" id="user-content-registering-udfs" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" role="img" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>Registering UDFs</h2>
<div style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px;">
create temporary function lcrc32 as 'com.levent.hive.udfs.lcrc32'; <br style="box-sizing: border-box;" />create temporary function ldays_between as 'com.levent.hive.udfs.ldays_between'; <br style="box-sizing: border-box;" />create temporary function lmonths_between as 'com.levent.hive.udfs.lmonths_between'; <br style="box-sizing: border-box;" />create temporary function ilike as 'com.levent.hive.udfs.liLIKE'; </div>
<h3 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.5em; line-height: 1.43; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/lserinol/Hive-udfs#comleventhiveudfslcrc32" id="user-content-comleventhiveudfslcrc32" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" role="img" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>com.levent.hive.udfs.lcrc32</h3>
<ul style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px; margin-top: 0px; padding: 0px 0px 0px 2em;">
<li style="box-sizing: border-box;">takes a string input and returns crc32(long) value </li>
</ul>
<h3 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.5em; line-height: 1.43; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/lserinol/Hive-udfs#comleventhiveudfsldays_between" id="user-content-comleventhiveudfsldays_between" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" role="img" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>com.levent.hive.udfs.ldays_between</h3>
<ul style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px; margin-top: 0px; padding: 0px 0px 0px 2em;">
<li style="box-sizing: border-box;">calculates days difference between two dates and returns number of days(int) value </li>
</ul>
<h3 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.5em; line-height: 1.43; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/lserinol/Hive-udfs#comleventhiveudfslmonths_between" id="user-content-comleventhiveudfslmonths_between" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" role="img" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>com.levent.hive.udfs.lmonths_between</h3>
<ul style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 16px; margin-top: 0px; padding: 0px 0px 0px 2em;">
<li style="box-sizing: border-box;">calculates months difference between two dates and returns number of months(int) value </li>
</ul>
<h3 style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 1.5em; line-height: 1.43; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="anchor" href="https://github.com/lserinol/Hive-udfs#comleventhiveudfslilike" id="user-content-comleventhiveudfslilike" style="box-sizing: border-box; color: #4078c0; display: inline-block; line-height: 1.2; margin-left: -18px; padding-right: 2px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" role="img" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1h-1c-1.5 0-3-1.69-3-3.5s1.55-3.5 3-3.5h4c1.45 0 3 1.69 3 3.5 0 1.41-0.91 2.72-2 3.25v-1.16c0.58-0.45 1-1.27 1-2.09 0-1.28-1.02-2.5-2-2.5H4c-0.98 0-2 1.22-2 2.5s1 2.5 2 2.5z m9-3h-1v1h1c1 0 2 1.22 2 2.5s-1.02 2.5-2 2.5H9c-0.98 0-2-1.22-2-2.5 0-0.83 0.42-1.64 1-2.09v-1.16c-1.09 0.53-2 1.84-2 3.25 0 1.81 1.55 3.5 3 3.5h4c1.45 0 3-1.69 3-3.5s-1.5-3.5-3-3.5z"></path></svg></a>com.levent.hive.udfs.liLIKE</h3>
<ul style="box-sizing: border-box; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px; margin-bottom: 0px !important; margin-top: 0px; padding: 0px 0px 0px 2em;">
<li style="box-sizing: border-box;">Hive ilike function in-case sensitive like between two strings retuns boolean TRUE/FALSE </li>
</ul>
Anonymoushttp://www.blogger.com/profile/06161379979453710759noreply@blogger.com0tag:blogger.com,1999:blog-1610041778732887994.post-25015843312190770942015-11-10T10:09:00.003-08:002015-11-10T10:10:40.960-08:00Simple HDFS Utility to see disk usage graphically on Hadoop 2.x<div style="text-align: center;">
<span class="fullpost"><a href="https://github.com/lserinol/simple-hdfs-du2">
<b>Link to project simple-hdfs-du2</b></a><br />
</span></div>
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<span class="fullpost"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuMJINwE-tQmuXE9A32eTM8APmPRsrK4tPza0823bq4UitSyaDH8MsoYqog1Cr9D-YesI5c6gvuU0lps-Z_lbNVRyhQtaI8IDXMSDY09XJswqjJTDoW3YRIEoWtFY1QWcgO4WAc9lAPvN8/s1600/Sampledu.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuMJINwE-tQmuXE9A32eTM8APmPRsrK4tPza0823bq4UitSyaDH8MsoYqog1Cr9D-YesI5c6gvuU0lps-Z_lbNVRyhQtaI8IDXMSDY09XJswqjJTDoW3YRIEoWtFY1QWcgO4WAc9lAPvN8/s320/Sampledu.png" /></a></span></div>
<span class="fullpost">
</span>Anonymoushttp://www.blogger.com/profile/06161379979453710759noreply@blogger.com0tag:blogger.com,1999:blog-1610041778732887994.post-12974890863190323832015-07-25T03:29:00.000-07:002015-07-25T03:32:01.443-07:00Simple HDFS Utility to see disk usage graphically<div style="text-align: center;">
<span class="fullpost"><a href="https://github.com/lserinol/simple-hdfs-du">
<b>Link to project simple-hdfs-du</b></a><br />
</span></div>
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<span class="fullpost"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuMJINwE-tQmuXE9A32eTM8APmPRsrK4tPza0823bq4UitSyaDH8MsoYqog1Cr9D-YesI5c6gvuU0lps-Z_lbNVRyhQtaI8IDXMSDY09XJswqjJTDoW3YRIEoWtFY1QWcgO4WAc9lAPvN8/s1600/Sampledu.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuMJINwE-tQmuXE9A32eTM8APmPRsrK4tPza0823bq4UitSyaDH8MsoYqog1Cr9D-YesI5c6gvuU0lps-Z_lbNVRyhQtaI8IDXMSDY09XJswqjJTDoW3YRIEoWtFY1QWcgO4WAc9lAPvN8/s320/Sampledu.png" /></a></span></div>
<span class="fullpost">
</span>Anonymoushttp://www.blogger.com/profile/06161379979453710759noreply@blogger.com0tag:blogger.com,1999:blog-1610041778732887994.post-49698672591159023042013-10-10T00:25:00.001-07:002013-10-10T00:25:44.580-07:00Time Lapse Code Virtualization of My Email DB Project<iframe allowfullscreen="" frameborder="0" height="344" src="//www.youtube.com/embed/ko4MQMQ0lxM" width="459"></iframe><br />
<br />
Anonymoushttp://www.blogger.com/profile/06161379979453710759noreply@blogger.com0tag:blogger.com,1999:blog-1610041778732887994.post-16724523331842843362013-05-06T05:16:00.002-07:002013-05-06T05:16:50.685-07:00Which processes are using your swap ? (Linux)Here is a simple bash script which briefly shows you the processes using your Linux's swap area.
<br />
<pre style="border: 1px solid gray; padding: 10px;">#!/bin/bash
exec 2>/dev/null
total=0
printf "pid\t\tprocess\t\tmemory (kB)\n"
while read -r me
do
ps=$(cat $me/comm)
res1=$?
val=($(grep "^VmSwap" $me/status))
res2=$?
p=$(basename $me)
if [[ $res1 -eq 0 && $res2 -eq 0 ]]; then
if [ ${val[1]} -ne 0 ]; then
printf "%d\t\t%s\t\t%s\n" "$p" "$ps" "${val[1]}"
let total=$total+${val[1]}
fi
fi
done < <(find /proc/ -mindepth 1 -maxdepth 1 -type d -name "[0-9]*" )
echo "Total(kB):$total"
</pre>
<br />
Sample output (processes not sorted by their memory usage)<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgyHvCQpGU_WN4SNV86EaV3Gx8Wq7p4H66-pqAr1qXAxyzsqMbKOx-q7sV_AQ0Km8bJ-zg4b9_dkDGCaWjz4HtiibEAv10iS141X3XMqNlPOseQ009J7fH-CCrYdlxi0VCt-wHYHGJny1fX/s1600/swap.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgyHvCQpGU_WN4SNV86EaV3Gx8Wq7p4H66-pqAr1qXAxyzsqMbKOx-q7sV_AQ0Km8bJ-zg4b9_dkDGCaWjz4HtiibEAv10iS141X3XMqNlPOseQ009J7fH-CCrYdlxi0VCt-wHYHGJny1fX/s320/swap.png" width="318" /></a></div>
<br />Anonymoushttp://www.blogger.com/profile/06161379979453710759noreply@blogger.com2tag:blogger.com,1999:blog-1610041778732887994.post-91105070164222197372012-09-20T06:54:00.001-07:002012-09-20T06:55:05.672-07:00Ext4 sysfs parametersHave you ever wondered how much data written to your Ext4 filesystem since it's first creation time or last mount time. Here is two ext4 sysfs parameters that will tell you the magic numbers you need.
<pre style="border: 1px solid gray; padding: 10px;">
/sys/fs/ext4/<disk>/lifetime_write_kbytes
</pre>
<pre style="border: 1px solid gray; padding: 10px;">
/sys/fs/ext4/<disk>/session_write_kbytes
</pre>
Here is some more ext4 sysfs paramters and their meanings, but mostly tuning of ext4 multiblock allocator.
<a href="http://www.kernel.org/doc/Documentation/ABI/testing/sysfs-fs-ext4"> ext4 sysfs paramters </a>Anonymoushttp://www.blogger.com/profile/06161379979453710759noreply@blogger.com0tag:blogger.com,1999:blog-1610041778732887994.post-30058689586802791212012-03-13T03:10:00.000-07:002012-03-13T03:10:56.071-07:00Freebsd ciss driver logical drive limitLately, I was faced with a strange problem on a HP server which uses HP Smart Array controller P800 which is running Freebsd.
All my 24 hard drives which is hosted on externel HP enclosures was presented as 24 single raid0 logical volumes. Surprisingly, when the freebsd booted it doesn't presented the disks and disabled the ciss driver (HP Smart Array Controller) with following message
<p>
<pre style="border: 1px solid gray; padding: 10px;">
Mar 12 16:00:41 freebsd kernel: ciss1: adapter claims to report absurd
number of logical drives (24 > 15)
</pre>
<p>
A quick check on the driver source code showed up that there is a hard limit defined to 15.(/usr/src/sys/dev/ciss/cissvar.h)<br />
<span style="color: #881280; font-family: monospace; white-space: pre-wrap;"></span><br />
<pre style="border: 1px solid gray; padding: 10px;">
#define CISS_MAX_LOGICAL 15
</pre>
<br>
<br>
So I wanted to be sure if I change the number to a higher value , won't do any unwanted behavior later on the system. In that point I contacted <a href="http://www.yogurt.org/me.html">Paul Saab</a> where he explained the limit as below.<br />
<pre style="border: 1px solid gray; padding: 10px;">
really that's done to limit the amount of memory needed upfront by the
driver. I believe you can easily increase the number of drives
without issue as long as you have enough memory below 4GB. Parts of
the ciss driver require that the memory you DMA from be under 4G</pre>
<br>
<br>
After the clarification, I just set the number to 32 and rebuild/install the kernel, restart the machine and a quick check on dmesg showed up all my 24 logical volumes presented on HP P800 controller.
<pre style="border: 1px solid gray; padding: 10px;">
Mar 13 10:32:37 freebsd kernel: da19: 286070MB (585871964 512 byte sectors: 255H 32S/T 65535C)
Mar 13 10:32:37 freebsd kernel: da20 at ciss1 bus 0 scbus2 target 19 lun 0
Mar 13 10:32:37 freebsd kernel: da20: <COMPAQ RAID 0 VOLUME OK> Fixed Direct Access SCSI-5 device
Mar 13 10:32:37 freebsd kernel: da20: 135.168MB/s transfers
Mar 13 10:32:37 freebsd kernel: da20: Command Queueing enabled
Mar 13 10:32:37 freebsd kernel: da20: 286070MB (585871964 512 byte sectors: 255H 32S/T 65535C)
Mar 13 10:32:37 freebsd kernel: da21 at ciss1 bus 0 scbus2 target 20 lun 0
Mar 13 10:32:37 freebsd kernel: da21: <COMPAQ RAID 0 VOLUME OK> Fixed Direct Access SCSI-5 device
Mar 13 10:32:37 freebsd kernel: da21: 135.168MB/s transfers
Mar 13 10:32:37 freebsd kernel: da21: Command Queueing enabled
Mar 13 10:32:37 freebsd kernel: da21: 286070MB (585871964 512 byte sectors: 255H 32S/T 65535C)
Mar 13 10:32:37 freebsd kernel: da22 at ciss1 bus 0 scbus2 target 21 lun 0
Mar 13 10:32:37 freebsd kernel: da22: <COMPAQ RAID 0 VOLUME OK> Fixed Direct Access SCSI-5 device
Mar 13 10:32:37 freebsd kernel: da22: 135.168MB/s transfers
Mar 13 10:32:37 freebsd kernel: da22: Command Queueing enabled
Mar 13 10:32:37 freebsd kernel: da22: 286070MB (585871964 512 byte sectors: 255H 32S/T 65535C)
Mar 13 10:32:37 freebsd kernel: da23 at ciss1 bus 0 scbus2 target 22 lun 0
Mar 13 10:32:37 freebsd kernel: da23: <COMPAQ RAID 0 VOLUME OK> Fixed Direct Access SCSI-5 device
Mar 13 10:32:37 freebsd kernel: da23: 135.168MB/s transfers
Mar 13 10:32:37 freebsd kernel: da23: Command Queueing enabled
Mar 13 10:32:37 freebsd kernel: da23: 286070MB (585871964 512 byte sectors: 255H 32S/T 65535C)
Mar 13 10:32:37 freebsd kernel: da24 at ciss1 bus 0 scbus2 target 23 lun 0
Mar 13 10:32:37 freebsd kernel: da24: <COMPAQ RAID 0 VOLUME OK> Fixed Direct Access SCSI-5 device
Mar 13 10:32:37 freebsd kernel: da24: 135.168MB/s transfers
Mar 13 10:32:37 freebsd kernel: da24: Command Queueing enabled
Mar 13 10:32:37 freebsd kernel: da24: 286070MB (585871964 512 byte sectors: 255H 32S/T 65535C)
</pre>
Anonymoushttp://www.blogger.com/profile/06161379979453710759noreply@blogger.com0tag:blogger.com,1999:blog-1610041778732887994.post-39000202391309681742012-02-28T01:50:00.000-08:002013-05-10T01:10:36.552-07:00Linux like dmesg in Freebsd<pre style="border: 1px solid gray; padding: 10px;">
<b>UPDATE</b>: this patch is applied to Freebsd repository on 10 May 2013<br />
<a href="http://svnweb.freebsd.org/base?view=revision&revision=250430">http://svnweb.freebsd.org/base?view=revision&revision=250430</a><br />
</pre>
<br />
<br />
In my previous <a href="http://lserinol.blogspot.com/2012/02/clearing-freebsd-kernel-message-buffer.html">post</a>, I was talking about the "dmesg -c" equivalent in FreeBSD. Simply it was a sysctl command. So, I decided to simply change FreeBSD dmesg command and make it act like in as Linux. All you have to do is download <a href="http://dl.dropbox.com/u/12053391/dmesg.patch">my patch</a> and apply it as shown below.I was tested the patch on FreeBSD 9 btw.<br />
<pre style="border: 1px solid gray; padding: 10px;">mybsd# pwd
/usr/src/sbin/dmesg
mybsd# patch < dmesg.patch
Hmm... Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|--- dmesg.c.org 2012-01-03 19:04:44.000000000 +0200
|+++ dmesg.c 2012-02-28 11:36:22.000000000 +0200
--------------------------
Patching file dmesg.c using Plan A...
Hunk #1 succeeded at 81.
Hunk #2 succeeded at 194.
Hunk #3 succeeded at 204.
done
mybsd# make
Warning: Object directory not changed from original /usr/src/sbin/dmesg
cc -O2 -pipe -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -c dmesg.c
cc -O2 -pipe -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -o dmesg dmesg.o -lkvm
</pre>
<br />Anonymoushttp://www.blogger.com/profile/06161379979453710759noreply@blogger.com2tag:blogger.com,1999:blog-1610041778732887994.post-22724328344113428442012-02-17T05:02:00.001-08:002012-02-17T05:02:52.710-08:00Clearing FreeBSD Kernel Message BufferAs you can clear kernel message buffer on Linux with <a href="http://en.wikipedia.org/wiki/Dmesg">dmesg</a> command " dmesg -c ", it's a little bit different on a BSD system.It's done by sysctl as following
<pre style="border: 1px solid gray; padding: 10px;">
# sysctl kern.msgbuf_clear=1
</pre>Anonymoushttp://www.blogger.com/profile/06161379979453710759noreply@blogger.com0tag:blogger.com,1999:blog-1610041778732887994.post-60351095280103922472010-06-25T08:34:00.001-07:002010-06-25T08:35:29.896-07:00Boosting Radware Defense Pro's performance ;)<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgWW2Z-F2yx6nZAtOehsA9djCVOZWNgFneKDogpNQipOW76P4c-Kj9M3hAAvCZWCV65EICf3-V-XLSqYMdMYXA44P2j3oeOYrzU5gtr2HfkVsgXh-jfal9SJiHWc2AkJx3CjxReYxG8ImbS/s1600/rad.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 320px; height: 192px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgWW2Z-F2yx6nZAtOehsA9djCVOZWNgFneKDogpNQipOW76P4c-Kj9M3hAAvCZWCV65EICf3-V-XLSqYMdMYXA44P2j3oeOYrzU5gtr2HfkVsgXh-jfal9SJiHWc2AkJx3CjxReYxG8ImbS/s320/rad.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5486735317709473986" /></a>Anonymoushttp://www.blogger.com/profile/06161379979453710759noreply@blogger.com0tag:blogger.com,1999:blog-1610041778732887994.post-60282226789653783002010-02-08T05:36:00.001-08:002010-02-08T05:42:17.954-08:00Detailed network interface statistics on LinuxHere is a helpful command for getting interface statistics on your linux box. It's called ip. It's bundled with iproute package. You can see information about dropped, collisions,crc,transmit and other info about your network interface.<br /><br /><pre style="border: 1px solid gray; padding: 10px;"><br />root@helga:~# ip -s -s link show eth0<br />2: eth0: <broadcast,multicast,up,lower_up> mtu 1500 qdi<br />WN qlen 1000<br /> link/ether 00:24:8c:d5:6a:a7 brd ff:ff:ff:ff:ff:ff<br /> RX: bytes packets errors dropped overrun mcast<br /> 1633058095 16933162 0 0 0 0 <br /> RX errors: length crc frame fifo missed<br /> 0 0 0 0 0 <br /> TX: bytes packets errors dropped carrier collsns<br /> 1994496524 5588574 0 0 0 0 <br /> TX errors: aborted fifo window heartbeat<br /> 0 0 0 0 <br /><br /></pre>Anonymoushttp://www.blogger.com/profile/06161379979453710759noreply@blogger.com3tag:blogger.com,1999:blog-1610041778732887994.post-30526522021787040482009-09-11T03:58:00.000-07:002009-09-23T07:40:52.362-07:00I/O usage per process on LinuxLinux kernel 2.6.20 and later supports per process I/O accounting. You can access every process/thread's I/O read/write values by using /proc filesystem. You can check if your kernel has built with I/O account by just simply checking /proc/self/io file. If it exists then you have I/O accounting built-in.<br /><br /><pre style="border: 1px solid gray; padding: 10px;"><br />$ cat /proc/self/io<br />rchar: 3809<br />wchar: 0<br />syscr: 10<br />syscw: 0<br />read_bytes: 0<br />write_bytes: 0<br />cancelled_write_bytes: 0<br /></pre><br /><pre style="border: 1px solid gray; padding: 10px;"><br /><span style="font-weight: bold;">Field Descriptions:</span><br /><br />rchar - bytes read<br />wchar - byres written<br />syscr - number of read syscalls<br />syscw - number of write syscalls<br />read_bytes - number of bytes caused by this process to read<br /> from underlying storage<br />write_bytes - number of bytes caused by this process to written from<br /> underlying storage<br /></pre>As you know, ever process is presented by it's pid number under /proc directory. You can access any process's I/O accounting values by just looking /proc/#pid/io file. There is a utility called <a href="http://guichaz.free.fr/iotop/">iotop</a> which collects these values and shows you in like top utility. You see your processes I/O activity with iotop utility.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1LPu5HaghyIZWD6hk3TZomu90t1j3gstyH7odC-2nyDvBpEYlzhsjM8HG2nwYnq7VeaL4UGE3YQq0OIpMnkLrRUof-1HX4RPVydyKfm62JSwMqKI9fOiphvH-4OX1tEq6gn-SElN0OCIt/s1600-h/iotop.png"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer; width: 320px; height: 217px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1LPu5HaghyIZWD6hk3TZomu90t1j3gstyH7odC-2nyDvBpEYlzhsjM8HG2nwYnq7VeaL4UGE3YQq0OIpMnkLrRUof-1HX4RPVydyKfm62JSwMqKI9fOiphvH-4OX1tEq6gn-SElN0OCIt/s320/iotop.png" alt="" id="BLOGGER_PHOTO_ID_5380169622024679122" border="0" /></a>Anonymoushttp://www.blogger.com/profile/06161379979453710759noreply@blogger.com3tag:blogger.com,1999:blog-1610041778732887994.post-62086720730806319452009-09-04T12:24:00.000-07:002009-09-06T23:25:13.050-07:00problem compiling Mysql 5.1 with sphinx engine support<div>Mysql's full-text search engine is not very powerful and support comes only with Myisam engine. <a href="http://www.sphinxsearch.com/">Sphinx</a> is a full-text search engine replacement for Mysql and Postgresql. Also, It's possible to use Sphinx full-text search engine with both Myisam and InnoDB engines. You can read detailed information about mysql compilation with sphinx on <a href="http://www.sphinxsearch.com/docs/current.html#sphinxse">Sphinx</a> site.<br /></div><br /><div>I have faced with following error message, while compiling Mysql with innodb and sphinx engine support ( ./configure --prefix=/data/mysql3/ --with-plugins=sphinx,innobase). I used the innodb (base) which bundled with Mysql. You can also compile <a href="http://www.innodb.com/products/innodb_plugin/">innodb plugin</a> which has additional features. For example, on the fly compression, speed improvements.<br /></div><pre style="border: 1px solid gray; padding: 10px;"><br />configure: error: unknown plugin: sphinx</pre><br /><p>The trick was to run autorun.sh command in Mysql's source tree before running configure with sphinx engine support. This script will create a new configure script which knows how to compile sphinx engine.<br /></p><p><pre style="border: 1px solid gray; padding: 10px;"><br /># ./BUILD/autorun.sh</pre><br /></p><p><br /></p>Anonymoushttp://www.blogger.com/profile/06161379979453710759noreply@blogger.com1tag:blogger.com,1999:blog-1610041778732887994.post-24352154905082167062009-09-03T00:31:00.000-07:002009-09-03T00:35:22.526-07:00nethogs network utility<a href="http://nethogs.sourceforge.net/">Nethogs</a> is a small but very useful utility, if you like to see your machine's bandwidth usage per processes. Here is a simple screenshot from my linuxbox.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyvkTawWlS2twF-FcGdFRAvXBChR5puXvrXi_x08P1ZPs3d2aH_dC8yC_AOXy6jHexEUhYXnUqW-7SCE9ZZ_svmIcrjgH9AVld8JHlfaDydhN_CX1j0_fh-QvwGuRm6tqDv3h26reMoqu3/s1600-h/nethogs.png"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer; width: 320px; height: 217px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyvkTawWlS2twF-FcGdFRAvXBChR5puXvrXi_x08P1ZPs3d2aH_dC8yC_AOXy6jHexEUhYXnUqW-7SCE9ZZ_svmIcrjgH9AVld8JHlfaDydhN_CX1j0_fh-QvwGuRm6tqDv3h26reMoqu3/s320/nethogs.png" alt="" id="BLOGGER_PHOTO_ID_5377141131547000098" border="0" /></a>Anonymoushttp://www.blogger.com/profile/06161379979453710759noreply@blogger.com0tag:blogger.com,1999:blog-1610041778732887994.post-77456603728942267732009-07-12T01:41:00.000-07:002009-07-12T02:00:13.920-07:00Fast Copy on Unix systemsI needed to copy so many small files and directories (1.2 TB) to another machine with <a href="http://netcat.sourceforge.net/">netcat</a> (nc). But I faced with a problem when I tried to run nc program on the background with bash. Bash was suspending standard input when I move nc process to background job, this was causing nc processess to die when standard input was blocked. So, I have found another netcat like utility called <a href="http://www.dest-unreach.org/socat/">socat</a>. Socat is not stop execution while it was working as background job on bash shell. I was able to utilize 100mbit ethernet with few socat processes on server and client side. First of all, I run socat processes on destination server as daemons to listen specific ports. Following, command will run socat in daemon mode to listen on tcp port 4123 and pipe incoming data to tar command on the background.<br /><br /># (socat -u tcp4-listen:4123 - | tar xzp -C .) &<br /><br />On source server, I run socat to feed incoming tar data to destination socat daemon where it listens on tcp port 4123. My destination server's ip adddess was 192.168.36.199 .<br /><br /># (tar -czp - /mydir/1/ | socat -u - tcp4:192.168.3.199:4123)&<br /><br /><div>So, with a simple bash script I was able to run few socat daemons on destination server with different ports and feed them from source server to achieve higher data transer rate.</div><div><br /></div>Anonymoushttp://www.blogger.com/profile/06161379979453710759noreply@blogger.com4tag:blogger.com,1999:blog-1610041778732887994.post-59694273296823298672009-06-08T09:13:00.000-07:002009-06-08T10:10:10.296-07:00socket programming with BashI have many memcached servers around. So, I needed a tool to check status information on these servers like top utility. I decided to write it in bash scripting language (100% in bash except printf,date and sleep commands :) ). You can simply enhance the script for example getting server names and ports from a config file or display interval parameter from command line. Memcached supports tcp and udp protocols for communication with clients. I used tcp communication in my bash script. Here is information about <a href="http://code.sixapart.com/svn/memcached/trunk/server/doc/protocol.txt">Memcached's supported server commands</a>. My example script uses "stats" command. Here is sample output for "stats" command from one memcached server. <div><br /></div><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghyjfUhBdHnaIT3VKhhBWfyLtncpCOa8SxRafq_R88y0hjZug3xPVP2xu89XZ7fcpXML3rOk5NZDBF0_QOpV_30FHoo8MoTHynOJYYQTg7Y_5rdP0Kk66QMXV2MmPXxOVXhk0i297J_EEx/s1600-h/memcache.jpg"><img style="display: block; margin-top: 0px; margin-right: auto; margin-bottom: 10px; margin-left: auto; cursor: pointer; width: 320px; height: 224px; text-align: center; " src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghyjfUhBdHnaIT3VKhhBWfyLtncpCOa8SxRafq_R88y0hjZug3xPVP2xu89XZ7fcpXML3rOk5NZDBF0_QOpV_30FHoo8MoTHynOJYYQTg7Y_5rdP0Kk66QMXV2MmPXxOVXhk0i297J_EEx/s320/memcache.jpg" border="0" alt="" id="BLOGGER_PHOTO_ID_5344993970269409506" /></a><br /><br />I have used the information provided by "stats" command. Script uses pure bash builtin commands to access a tcp server like memcached.<br /><br />You can download the script from <a href="http://lserinol.googlepages.com/memcache_stats.sh">here</a>.<br /><br /><br />Here is sample output from the script.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1a-aygfJHkldu8TKtmB5FKrTAYkQItd5qtPMGPSPV3b3MuT0N3RekfheE3WxA6TAt8onG65o1HpUIzfwsYLya85IvcAKD2SgGTergXTdry0ncLKWDQnpIosOqpj120B624vjOvuFVMuuP/s1600-h/stats.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 320px; height: 224px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1a-aygfJHkldu8TKtmB5FKrTAYkQItd5qtPMGPSPV3b3MuT0N3RekfheE3WxA6TAt8onG65o1HpUIzfwsYLya85IvcAKD2SgGTergXTdry0ncLKWDQnpIosOqpj120B624vjOvuFVMuuP/s320/stats.jpg" border="0" alt="" id="BLOGGER_PHOTO_ID_5344995657125518354" /></a><br /><br />If you're using Ubuntu, probably you will not be able to run this script, because bash is not compiled with "--enable-net-redirections" parameter on Ubuntu systems. RedHat based distributions has no problem with bash net-directions like Fedora,Centos and RedHat ES.Anonymoushttp://www.blogger.com/profile/06161379979453710759noreply@blogger.com2tag:blogger.com,1999:blog-1610041778732887994.post-1406253705495861132009-05-31T02:06:00.000-07:002009-05-31T02:58:44.931-07:00Mirroring your OpenSolaris root disk with ZFSSolaris/OpenSolaris supports booting with ZFS filesystem. For example, if you look into your opensolaris insallation you'll notice the zfs rpool (root pool). ZFS is default filesystem on opensolaris (optional with Solaris). If you look into detailts of that pool with zpool utility , you'll see that it's a single disk or a partition (depends how you partitioned your disk on installation process of opensolaris). If that single disk crashes, you may lost your data and operating system.That's one point of failure. The simple solution is to convert our zpool into <a href="http://en.wikipedia.org/wiki/RAID_1#RAID_1">RAID-1</a> with a secondary disk (with same capacity or bigger one). <p><br /><br /><br /><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjdGWXotD0jhKIFlaysj4EDDWUOE2IJlHLsmPq3HobzwiDoIpt3Qxp0HcGBobP5OznFY5b9zE_F3xTg4MY9xbVdwpyO_6H2DknwIeE_S9_NwW_fWC_304ojXWN1bjT-brHTch0rIeEeZla1/s1600-h/zpool1.JPG"><img id="BLOGGER_PHOTO_ID_5341923515110796754" style="DISPLAY: block; MARGIN: 0px auto 10px; WIDTH: 320px; CURSOR: hand; HEIGHT: 256px; TEXT-ALIGN: center" alt="" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjdGWXotD0jhKIFlaysj4EDDWUOE2IJlHLsmPq3HobzwiDoIpt3Qxp0HcGBobP5OznFY5b9zE_F3xTg4MY9xbVdwpyO_6H2DknwIeE_S9_NwW_fWC_304ojXWN1bjT-brHTch0rIeEeZla1/s320/zpool1.JPG" border="0" /></a><br /><br /><br /><br /><br /><p>As you can see, our zpool contains a single disk called c4t0d0. Let's add a new disk to our rpool to convert it into mirror. zpool will do it automatically for us. One important point here is that Solaris doesn't support boot devices to be in EFI label format. So, if your new disk label is in EFI format, you need to convert it into SMI label. So, I'll use fdisk command to create a Solaris2 partition and remove any EFI labels if present. My new disk name is c4t1d0.<br /></p><br /><br /><p><br /><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh23byRE1LuKvqLjZFMTckYDwm352OLhoGzd_4CMuiGoWHvhxFkCm7s9cKmpa9T7CW8LE4wrw1LU4aO3aoOlpptRBHmW_vc-I-gJHj2NdEHcBfT4tt_ElflvxuCyZv8_LRS3oiQVdiRcA9q/s1600-h/zpool3.JPG"><img id="BLOGGER_PHOTO_ID_5341920095333038642" style="DISPLAY: block; MARGIN: 0px auto 10px; WIDTH: 320px; CURSOR: hand; HEIGHT: 256px; TEXT-ALIGN: center" alt="" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh23byRE1LuKvqLjZFMTckYDwm352OLhoGzd_4CMuiGoWHvhxFkCm7s9cKmpa9T7CW8LE4wrw1LU4aO3aoOlpptRBHmW_vc-I-gJHj2NdEHcBfT4tt_ElflvxuCyZv8_LRS3oiQVdiRcA9q/s320/zpool3.JPG" border="0" /></a> </p><br /><br /><p>After creating the partition which covers whole disk area, I copied partition map of original rpool member c4t0d0 to our new disk c4t1d0.</p><br /><br /><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi46zLxge01q2yurna4towqZGa_PRpNACD9F2gwOm4A4Z9ibKGdGklS0r0qJdXS7BgvEc2kMcbwSA0WwEawvcxnX9Ai_ohDSF33b17i5qhAW1lwpXg071K9KoWjxjMpmzs2AupeuWJ9EoBO/s1600-h/zpool4.JPG"><img id="BLOGGER_PHOTO_ID_5341921038872505282" style="DISPLAY: block; MARGIN: 0px auto 10px; WIDTH: 320px; CURSOR: hand; HEIGHT: 256px; TEXT-ALIGN: center" alt="" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi46zLxge01q2yurna4towqZGa_PRpNACD9F2gwOm4A4Z9ibKGdGklS0r0qJdXS7BgvEc2kMcbwSA0WwEawvcxnX9Ai_ohDSF33b17i5qhAW1lwpXg071K9KoWjxjMpmzs2AupeuWJ9EoBO/s320/zpool4.JPG" border="0" /></a><br /><p>Now, it's time to attach our brand new disk into rpool. After attaching the disk, our rpool pool will be converted into mirror (raid-1). Except one thing, the grub boot manager. We need to install it into new disk by hand using installgrub command.<p><br /><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUkkUyug192XBkHERBsIu3fnfJsf0kL3NjFtFQ_ROqznI_tiL67L2V7L38CvLDdJYHiyzBgrHt8V07IPfOm5vbyqgIGfN1oCGxLNese2WSBmn1oHCoxIGNQQ0uDsHKCsck85NAgmoleT45/s1600-h/zpool5.JPG"><img id="BLOGGER_PHOTO_ID_5341922021676787602" style="DISPLAY: block; MARGIN: 0px auto 10px; WIDTH: 320px; CURSOR: hand; HEIGHT: 256px; TEXT-ALIGN: center" alt="" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUkkUyug192XBkHERBsIu3fnfJsf0kL3NjFtFQ_ROqznI_tiL67L2V7L38CvLDdJYHiyzBgrHt8V07IPfOm5vbyqgIGfN1oCGxLNese2WSBmn1oHCoxIGNQQ0uDsHKCsck85NAgmoleT45/s320/zpool5.JPG" border="0" /></a><br /><br /><br />Now, we have created our mirror on rpool. It will take time to sync original disk to new disk. You can watch this process by using zpool's status command.<br /><br /><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuphQIXeie-Iy3ZIKBG-3V9KcBsMpNYMBLOaCteaPkQnIGWXbKuZcaoN3YHLyxtLGlJdy_y_l8icno70Z5NA9i_c1fqwClP6tqG8byBS0H4OQtUhEuPhCkHlRJFDJpoN2t9KcKZ6Vgn43O/s1600-h/zpool6.JPG"><img id="BLOGGER_PHOTO_ID_5341922530639334210" style="DISPLAY: block; MARGIN: 0px auto 10px; WIDTH: 320px; CURSOR: hand; HEIGHT: 256px; TEXT-ALIGN: center" alt="" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuphQIXeie-Iy3ZIKBG-3V9KcBsMpNYMBLOaCteaPkQnIGWXbKuZcaoN3YHLyxtLGlJdy_y_l8icno70Z5NA9i_c1fqwClP6tqG8byBS0H4OQtUhEuPhCkHlRJFDJpoN2t9KcKZ6Vgn43O/s320/zpool6.JPG" border="0" /></a>Anonymoushttp://www.blogger.com/profile/06161379979453710759noreply@blogger.com6tag:blogger.com,1999:blog-1610041778732887994.post-87491170643909375962009-05-23T03:21:00.001-07:002009-05-23T04:24:55.491-07:00FreeBSD and procfsMany Unix systems have support for proc file system (process file system).Procfs filesystem type is pseudo. FreeBSD is one of that Unix systems. Unlike Linux, which has information other than processes, FreeBSD procfs support is only about the processes on the system. FreeBSD doesn't mount the procfs on boot by default. You need to manually add it to fstab for auto mount on boot or mount it by command for temporarily usage.Common mount point for procfs is /proc on Unix systems.<br /><br /><pre style="border: 1px solid gray; padding: 10px;"><br />echo "</span>none /proc procfs rw 0 0" >> /etc/fsab<br /></pre><br /><br /><pre style="border: 1px solid gray; padding: 10px;"><br />mount -t procfs none /proc<br /></pre><br /><br />Every process is presented as directories named by it's pid number on the /proc mount point. <br />Procfs gives information on running processes on the system like memory mapping,command line arguments of running process, process resource limits and many other.Following is a sample procfs directory structure on a FreeBSD machine.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiuu4c4qId2oehVSIP9P4uiraDVAWVE-_BkQScgvmFwUpgT2MNxNXY9LeNgnKyRB0F0yMhbOkBcuLOysysO_9m4LgEn265UNyWhCj10wQe700h76p8XBjwRAFpuYy-Lj0LYWxs2rG01ABZO/s1600-h/mount_procfs_ls.jpeg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 320px; height: 224px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiuu4c4qId2oehVSIP9P4uiraDVAWVE-_BkQScgvmFwUpgT2MNxNXY9LeNgnKyRB0F0yMhbOkBcuLOysysO_9m4LgEn265UNyWhCj10wQe700h76p8XBjwRAFpuYy-Lj0LYWxs2rG01ABZO/s320/mount_procfs_ls.jpeg" border="0" alt=""id="BLOGGER_PHOTO_ID_5338966274327860578" /></a><br /><br />As you can see, every pid is represented as a directory in procfs. Every directory contains following files. Some of the files are write only or read only where you read information or send information to process. <br /><br /><pre style="border: 1px solid gray; padding: 10px;"><br /><br />- status (read-only) : returns process status<br />- mem (read/write): virtual memory image of the process<br />- file (depends) : symbolic link to running process<br />- regs (read/write): process registers<br />- ctl (write-only): used to send signal to process or <br /> attach/deattach it for debugging<br />- cmdline (read-only) : command line arguments of running process<br />- rlimits (read-only) : current resource limits of running process<br />- map (read-only) : memory mappings of the running process.<br />- etype (read-only) : type of the executable (eg. FreeBSD ELF32)<br />- fpregs (read/write): floating point registers <br /></pre><br /><br /><br />Some of the information provided by these files are in binary format.For example "regs" and "fpregs" files are in binary format. They depend on the architecture of the underlying machine (i386, amd64,sparc64,etc..). Following is the format of the "regs" file on the i386 machine.<br /><pre style="border: 1px solid gray; padding: 10px;"><br /><br />struct reg {<br /> unsigned int r_fs;<br /> unsigned int r_es;<br /> unsigned int r_ds;<br /> unsigned int r_edi;<br /> unsigned int r_esi;<br /> unsigned int r_ebp;<br /> unsigned int r_isp;<br /> unsigned int r_ebx;<br /> unsigned int r_edx;<br /> unsigned int r_ecx;<br /> unsigned int r_eax;<br /> unsigned int r_trapno;<br /> unsigned int r_err;<br /> unsigned int r_eip;<br /> unsigned int r_cs;<br /> unsigned int r_eflags;<br /> unsigned int r_esp;<br /> unsigned int r_ss;<br /> unsigned int r_gs;<br />};<br /></pre><br /><br />You can use "cat" command to read information provided by procfs for text based information unlike the ones I mentioned above in binary format like "regs","fpregs" and "mem". <br /><br /><pre style="border: 1px solid gray; padding: 10px;"><br /><br />[root@freebsd ]# cat cmdline <br />/usr/sbin/moused-p/dev/ums0-tauto-I/var/run/moused.ums0.pid<br /></pre><br /><br />You can check a running process's resource limits by looking into rlimit file.<br /><br /><pre style="border: 1px solid gray; padding: 10px;"><br /><br />[root@freebsd ]# cat rlimit <br />cpu -1 -1<br />fsize -1 -1<br />data 536870912 536870912<br />stack 67108864 67108864<br />core -1 -1<br />rss -1 -1<br />memlock -1 -1<br />nproc 5547 5547<br />nofile 11095 11095<br />sbsize -1 -1<br />vmem -1 -1<br /></pre><br /><br />First digits is minimum and last one is maximum values of the given resource name. "-1" means infinite. For examle, nofile (open file descriptor) limit for this process is 11095 as minimum and maximum.<br /><br />"status" file gives information about process status as follows.<br /><pre style="border: 1px solid gray; padding: 10px;"><br />- command name<br />- pid<br />- parent pid<br />- process group id<br />- session id<br />- major/minor of the terminal, "-" if no terminal is in action<br />- process flags<br />- process start time in seconds and microseconds separated by comma<br />- user time in seconds and microseconds separated by comma<br />- system time in seconds and microseconds separated by comma<br />- wait channel name<br />- effective userid and group lists separated by comma<br /></pre><br /><br />Following is "cat status" result of a process.<br /><pre style="BORDER-RIGHT: gray 1px solid; PADDING-RIGHT: 10px; BORDER-TOP: gray 1px solid; PADDING-LEFT: 10px; PADDING-BOTTOM: 10px; BORDER-LEFT: gray 1px solid; PADDING-TOP: 10px; overflow:scroll;white-space:nowrap;height:40px;BORDER-BOTTOM: gray 1px solid">[root@freebsd ]# cat status <br />svscan 28246 1 28245 0 ttyp0 noflags 1242921860,839572 0,318052 3,263576 nanslp 0 0 0,0,0,5 -<br /></pre>Anonymoushttp://www.blogger.com/profile/06161379979453710759noreply@blogger.com0tag:blogger.com,1999:blog-1610041778732887994.post-68234394391136438352009-05-16T04:49:00.000-07:002009-05-16T06:08:44.576-07:00Configuring ILOM manager on Sun X4540 Thumper<a href="http://www.sun.com/servers/x64/x4540/">Sun X4540 Thumper</a> storage server provides 48TB hard disk capacity on one server. It has 2 AMD Quad x64 Opteron processors and 32GB of memory. Thumper has 48 hard disk slots. You can use 1TB sata hard disks in each slot. This will give you 48TB hard disk area as you can see in following photo.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9qH5AgDBo_sPHmzlErmvvPcHS3klwHMV76fSi1cgS_VS3ZNTshMYkPhTNikXMP1Vs9FE_RcNt9GAsXi8xiTaX2_Y23-1ZFEwblNgkWEDs7TbgmuGnIgiaYfXezCmciIuf4ENFknUz-Xb7/s1600-h/IMG_3753.jpg"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer; width: 320px; height: 240px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9qH5AgDBo_sPHmzlErmvvPcHS3klwHMV76fSi1cgS_VS3ZNTshMYkPhTNikXMP1Vs9FE_RcNt9GAsXi8xiTaX2_Y23-1ZFEwblNgkWEDs7TbgmuGnIgiaYfXezCmciIuf4ENFknUz-Xb7/s320/IMG_3753.jpg" alt="" id="BLOGGER_PHOTO_ID_5336393148969781442" border="0" /></a><br />You can run Solaris,OpenSolaris, Linux and Windows operating systems on Thumper. Thumper includes Sun Microsystem's <a href="http://www.sun.com/systemmanagement/ilom.jsp">ILOM</a> (Sun Integrated Lights Out Manager) manager. You can use ILOM to upgrade your server's firmware, remotely accessing your server's console by using your web browser, modify and see sensors, see hardware components and so on. You can access ILOM by using a serial console,ssh client or a web browser.You need to set an ip address for your ILOM manager if you need to use ssh or a web browser.If you don't have a dhcp server around you'll need to configure your ILOM manager with a static ip address. I have used a serial console connected on Thumpers "Ser MGT" port and on the other side a linux desktop and minicom application. You need to change your minicom settings to 9600 baud rate , 8 data bits, no parity and one stop bit (9600-8-N-1) with no hardware and software flow control.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEioGp1qdMfFZaI7_vFQUz58DOV6jTRg_mHB3Tst85vdBOgsjGkIuHPV3nYRqNewAaRdOGiCM58skoOuUvM1MHMGkXjeW3m88WYzcGEA-KA1R9oEvwi7bdz56ztBz0WahNdDRIbkiDi5pL5X/s1600-h/ilom1.png"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer; width: 320px; height: 183px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEioGp1qdMfFZaI7_vFQUz58DOV6jTRg_mHB3Tst85vdBOgsjGkIuHPV3nYRqNewAaRdOGiCM58skoOuUvM1MHMGkXjeW3m88WYzcGEA-KA1R9oEvwi7bdz56ztBz0WahNdDRIbkiDi5pL5X/s320/ilom1.png" alt="" id="BLOGGER_PHOTO_ID_5336397525207414818" border="0" /></a><br />Yes, that's right, It's Linux :) . Sun ILOM uses embeded Linux. When you connect to ILOM by a serial console you will need to login in by using ILOM's factory default login name "root" and password "changeme". ILOM has a simple built-in command interface. You can use "help" command for details when you logged in ILOM manager. Setting the static ip address of ILOM is simple. I captured the screenshot when I gave one of my local ip addresses to Thumper's ILOM.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAtJIJFI6ill2O7gYakgDBeQmW_465Hib1GtTDe9Ctl1ZXoDJTuPyLOrI85S46l8bhwuqCLRjOcofa1RbYtLSSBbnKhZ1cIoK8gG7LirC6tHuJ-_13138FBDDbpvNQ3cguFVZjpCIiWZ90/s1600-h/ilom2.png"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer; width: 320px; height: 183px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAtJIJFI6ill2O7gYakgDBeQmW_465Hib1GtTDe9Ctl1ZXoDJTuPyLOrI85S46l8bhwuqCLRjOcofa1RbYtLSSBbnKhZ1cIoK8gG7LirC6tHuJ-_13138FBDDbpvNQ3cguFVZjpCIiWZ90/s320/ilom2.png" alt="" id="BLOGGER_PHOTO_ID_5336399535168579906" border="0" /></a><br /><br />After, you commit your static ip network settings settings with "set commitpending=true" command, you need to connect Thumper's "net mgt" ethernet interface to your network.Now, you can access ILOM by using any ssh client or a web browser. My plan is to use OpenSolaris and 48TB ZFS Raid-Z on this storage server and see the results on a production environment :)Anonymoushttp://www.blogger.com/profile/06161379979453710759noreply@blogger.com3tag:blogger.com,1999:blog-1610041778732887994.post-60481674917586739062009-05-03T01:35:00.000-07:002009-05-03T04:09:23.402-07:00getting detailed process information on FreebsdFreebsd <a href="http://www.freebsd.org/cgi/man.cgi?query=procstat&apropos=0&sektion=0&manpath=FreeBSD+7.1-RELEASE&format=html">procstat</a> utility gives detailed information about all of the processes on the system or just for a given process id number, such as virtual memory mapping, thread stack, command line arguments and open files.<br /><br />Running procstat with "-a" argument prints information like pid,pid,login, process name, wchan (which event the process waiting) of all processes on the system.<br /><br /># procstat -a <br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj4LrBwZ4IXpDbY_aOVyRC6iXWerHWdA9h_9iX6i4dcxZAUxotGgSkqDuo6nAzoI1yNza9BBYuORbutgYPU_O3pzwvmCeIB8UPohlxMGP57zVFluDeI2y2VXOD22b9Rc2NUIcHyCou9FwVm/s1600-h/procstat_a.JPG"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 320px; height: 178px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj4LrBwZ4IXpDbY_aOVyRC6iXWerHWdA9h_9iX6i4dcxZAUxotGgSkqDuo6nAzoI1yNza9BBYuORbutgYPU_O3pzwvmCeIB8UPohlxMGP57zVFluDeI2y2VXOD22b9Rc2NUIcHyCou9FwVm/s320/procstat_a.JPG" border="0" alt=""id="BLOGGER_PHOTO_ID_5331550357165028002" /></a><br /><br />Procstat "-c" option shows you the command line arguments of a process and "-f" option shows opened files by given process.Following sample output shows the command line arguments and the files currently opened with their permissions by vi process.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1FHAHItlTHVgO71AAux1DKQrp_JBhBkb7PU4TiILUECeTJo4Hz78hXMmQW_pmR06Pcl9-Tgdy8zZBHU1SKCy9S9HQ4YArAJtPYvEiG4HOGj6lTK5Xm9s0RexpBXmpNsl30HTyy0x09m7V/s1600-h/procstat_vi.JPG"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 320px; height: 178px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1FHAHItlTHVgO71AAux1DKQrp_JBhBkb7PU4TiILUECeTJo4Hz78hXMmQW_pmR06Pcl9-Tgdy8zZBHU1SKCy9S9HQ4YArAJtPYvEiG4HOGj6lTK5Xm9s0RexpBXmpNsl30HTyy0x09m7V/s320/procstat_vi.JPG" border="0" alt=""id="BLOGGER_PHOTO_ID_5331550725039255138" /></a><br /><br />You can access virtual memory mapping information a process with "-v" option. Following sample shows vi process virtual memory mappings.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5L8Yj8XXRcB6y_q2VM4Am4M2hcFut3rtEZDFjUTjtTX9Xp9KXQiwBRQO0eejDQTbrf8rtvvJtjU3TlMcUKVHTO9aQ_2T3aBPMnNw0HY4_L-WxKG3fMlpPLdBM0aFATheUQvnqcCkG1ijY/s1600-h/procstat_v.JPG"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 320px; height: 178px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5L8Yj8XXRcB6y_q2VM4Am4M2hcFut3rtEZDFjUTjtTX9Xp9KXQiwBRQO0eejDQTbrf8rtvvJtjU3TlMcUKVHTO9aQ_2T3aBPMnNw0HY4_L-WxKG3fMlpPLdBM0aFATheUQvnqcCkG1ijY/s320/procstat_v.JPG" border="0" alt=""id="BLOGGER_PHOTO_ID_5331551828783527826" /></a><br /><br />Finally, "-k" option shows kernel threads stacks details of given process.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh4ispqVIWjhX4uZiSQdaPWA_VLIo-wf-ip7v5Wq1KnAQHMPDdHQKVXv-pubD_JvY9iygDPlw11MF_GATrbwsja7OAsIhNCLYeekzfeUtSPtczeYbRlYGX0ydeNhyphenhyphenH3Vc-wce5jE4eKMWoD/s1600-h/procstat_k.JPG"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 320px; height: 178px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh4ispqVIWjhX4uZiSQdaPWA_VLIo-wf-ip7v5Wq1KnAQHMPDdHQKVXv-pubD_JvY9iygDPlw11MF_GATrbwsja7OAsIhNCLYeekzfeUtSPtczeYbRlYGX0ydeNhyphenhyphenH3Vc-wce5jE4eKMWoD/s320/procstat_k.JPG" border="0" alt=""id="BLOGGER_PHOTO_ID_5331552730447031298" /></a>Anonymoushttp://www.blogger.com/profile/06161379979453710759noreply@blogger.com0tag:blogger.com,1999:blog-1610041778732887994.post-24463352168754292592009-05-03T00:18:00.000-07:002009-05-03T01:22:15.233-07:00watching interrupt usage on FreebsdYou can easily see how many interrupts taken by each device on Linux by simply looking in /proc/interrupts file. Freebsd has no such information in procfs. But you can access the same information by using "vmstat -i" command in Freebsd.But you can't see which cpu handling which irq as shown by Linux. Following is a sample output of "vmstat -i" command from a Freebsd machine.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhVnTH9LImwZoNsj8i4owJZ10iFZ99FLHhbOkOlBhEUFY2lpL5I1yFx5e-Qe-EyqDeN3edXLpPhnY9ki8o1sB0pZCLIWe2nR9zwd1AK4RJPfJ46ygsvikrhYFmSQKOLY2V-M9u6ytIAolDs/s1600-h/int.JPG"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 320px; height: 178px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhVnTH9LImwZoNsj8i4owJZ10iFZ99FLHhbOkOlBhEUFY2lpL5I1yFx5e-Qe-EyqDeN3edXLpPhnY9ki8o1sB0pZCLIWe2nR9zwd1AK4RJPfJ46ygsvikrhYFmSQKOLY2V-M9u6ytIAolDs/s320/int.JPG" border="0" alt="" id="BLOGGER_PHOTO_ID_5331496433392855122" /></a><br /><br />You can write a simple c shell script to watch interrupt usage like this.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhyWn_Usi6wO_dpfqCnZxjyAZtpBh1LRYAPhvXfbnm0k4GoCWHqrXJNL-3JqtgilR52M3L2WeYao25wAeXXTAIOQ2raTs73pVcCP2eleaF5nlC2IfVPfVM4MdjlqmEi9-mbl4YnvrUBbvu/s1600-h/interval.JPG"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 320px; height: 178px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhyWn_Usi6wO_dpfqCnZxjyAZtpBh1LRYAPhvXfbnm0k4GoCWHqrXJNL-3JqtgilR52M3L2WeYao25wAeXXTAIOQ2raTs73pVcCP2eleaF5nlC2IfVPfVM4MdjlqmEi9-mbl4YnvrUBbvu/s320/interval.JPG" border="0" alt="" id="BLOGGER_PHOTO_ID_5331496797577929266" /></a>Vmstat uses sysctl interface to gather interrupt usage information by using hw.intrnames and hw.intrcnt oid names. As names suggest intrnames holds all interrupt names and intrcnt holds their irq counts since system startup for each interrupt. <br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgmZ2NYeP36XZG8AMLEiVrhvSAeSx0ErPb4AanXtb9sp4ZiJVA01R1QVfR8ROtbKJr8_7V2FiG08r5MLTjvNKJxWeik_zEjyTs8EhIK8qCSdr7Fct6_vOZ0DxTXrdYB5lucsTaitEMBHmpv/s1600-h/sysctl_hw.JPG"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 320px; height: 178px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgmZ2NYeP36XZG8AMLEiVrhvSAeSx0ErPb4AanXtb9sp4ZiJVA01R1QVfR8ROtbKJr8_7V2FiG08r5MLTjvNKJxWeik_zEjyTs8EhIK8qCSdr7Fct6_vOZ0DxTXrdYB5lucsTaitEMBHmpv/s320/sysctl_hw.JPG" border="0" alt="" id="BLOGGER_PHOTO_ID_5331509639837123922" /></a>Anonymoushttp://www.blogger.com/profile/06161379979453710759noreply@blogger.com1tag:blogger.com,1999:blog-1610041778732887994.post-75623771844002807292009-04-24T03:56:00.001-07:002009-04-24T04:34:48.659-07:00Linux NFS and "permission denied" mount problemToday, I have faced with following strange nfs mount problem.<br /><br /><pre style="border: 1px solid gray; padding: 10px;"><br />[root@nfsclient ~]# mount /mymount<br />mount: XXX.XXX.XXX.XXX:/mymount failed, <br />reason given by server: Permission denied<br /></pre><br /><br />My nfs client is FC4 and nfs server is Centos 4.6.First, I have checked the server from nfs client side by using "showmount -e myserver" command. Exports was okay.After that, I decided to check rpc services on server with "rpcinfo -p" command. <br /><br /><pre style="border: 1px solid gray; padding: 10px;"><br />[root@nfsclient ~]# rpcinfo -p myserver<br /><br /> program vers proto port<br /> 100000 2 tcp 111 portmapper<br /> 100000 2 udp 111 portmapper<br /> 100024 1 udp 638 status<br /> 100024 1 tcp 641 status<br /> 100021 1 udp 32768 nlockmgr<br /> 100021 3 udp 32768 nlockmgr<br /> 100021 4 udp 32768 nlockmgr<br /> 100021 1 tcp 32772 nlockmgr<br /> 100021 3 tcp 32772 nlockmgr<br /> 100021 4 tcp 32772 nlockmgr<br /> 100011 1 udp 903 rquotad<br /> 100011 2 udp 903 rquotad<br /> 100011 1 tcp 906 rquotad<br /> 100011 2 tcp 906 rquotad<br /> 100003 2 udp 2049 nfs<br /> 100003 3 udp 2049 nfs<br /> 100003 4 udp 2049 nfs<br /> 100003 2 tcp 2049 nfs<br /> 100003 3 tcp 2049 nfs<br /> 100003 4 tcp 2049 nfs<br /> 100005 1 udp 925 mountd<br /> 100005 1 tcp 928 mountd<br /> 100005 2 udp 925 mountd<br /> 100005 2 tcp 928 mountd<br /> 100005 3 udp 925 mountd<br /> 100005 3 tcp 928 mountd<br /></pre><br /><br />Mandatory NFS services was working correctly as seen above rpc report.I didn't have a firewall between these machines or iptables on any of them.This was not the problem also here.I haven't check nfs client side because it was mounting other nfs server's mount points without a problem at all. So, I have focused on nfs server side. Restarting nfsd service or re-exporting nfs file systems with exportfs didn't work out. I can see nfs clients that mounted server's nfs mount by looking in /var/lib/nfs/rmtab file (showmount -a) on server. But I couldn't access /proc/fs/nfsd/ directory. It was not mounted on nfs server.So,I manually mounted it by using following mount command.<br /><br /><pre style="border: 1px solid gray; padding: 10px;"><br />mount -t nfsd nodev /proc/fs/nfsd<br /></pre><br /><br />After mounting /proc/fs/nfsd manually, I was able to mount the nfs server from client side again.<br /><br />If you look at into /etc/modprobe.conf.dist file, you can see that when kernel installs module nfsd , it mounts /proc/fs/nfsd and /var/lib/nfs/rpc_pipefs mount points.<br /><br /><pre style="border: 1px solid gray; padding: 10px;"><br />install nfsd /sbin/modprobe --first-time --ignore-install nfsd && <br />{ /bin/mount -t nfsd nfsd /proc/fs/nfsd > /dev/null 2>&1 || :; }<br />install sunrpc /sbin/modprobe --first-time --ignore-install sunrpc && <br />{ /bin/mount -t rpc_pipefs sunrpc /var/lib/nfs/rpc_pipefs > <br />/dev/null 2>&1 || :; }<br /></pre>Anonymoushttp://www.blogger.com/profile/06161379979453710759noreply@blogger.com2tag:blogger.com,1999:blog-1610041778732887994.post-79558960922275705202009-04-14T15:18:00.000-07:002009-04-14T16:21:18.168-07:00Process IO Top utility with Solaris DTrace<a href="http://www.sun.com/software/solaris/ds/dtrace.jsp">DTrace</a> is dynamic tracing framework created by Sun Microsystems for Solaris OS. DTrace let's any system administrator or developer to get internal view of how an operating system works, find any system bottlenecks, examine a live system or process, get information on system calls, network internals and many other things. For this purpose DTrace has many providers for specific tasks. You can get list of the DTrace providers and probe function names by running following command:<br /><br /># dtrace -l<br /><br />DTrace also adopted to other operating systems like FreeBSD and Mac OS X. DTrace uses D Language which is subset of C programming language. You can find more information about DTrace on <a href="http://www.sun.com/bigadmin/content/dtrace/">Sun's BigAdmin Portal</a>.<br /><br />Following example demostrates you, how easy to find the information you want on Solaris with DTrace.I wrote it to see which files are most written or read by which processes on the Solaris.<br /><br /><pre style="BORDER-RIGHT: gray 1px solid; PADDING-RIGHT: 10px; BORDER-TOP: gray 1px solid; PADDING-LEFT: 10px; PADDING-BOTTOM: 10px; BORDER-LEFT: gray 1px solid; PADDING-TOP: 10px; overflow:scroll;white-space:nowrap;height:500px;BORDER-BOTTOM: gray 1px solid"><br />#!/usr/bin/bash<br /># process_io_top - show procesess by top read/write KB I/O per file<br /># Written by Levent Serinol (lserinol@gmail.com)<br /># http://lserinol.blogspot.com<br /># Apr/14/2009<br />#<br /># USAGE: process_io_top [-s interval] [-p pid]<br /># <br /># -s interval # gather and show statistics in given interval (seconds)<br /># -p pid # show read/write KB I/O just for given PID<br /># -h # show usage information<br />#<br /># eg:<br /># process_io_top -s 10 <br />#<br />#<br />#<br />####################################################################################<br />interval=5<br />show_pid=0<br />pid=0<br /><br />function usage()<br />{<br />echo "<br />USAGE: io.sh [-s interval] [-p pid]<br /> -s # set interval, default is 5 seconds<br /> -p pid # pid<br /> eg, <br /> io -p 630 # io activity of pid 630<br /> io -s 10 # refresh output in every 10 seconds";<br />}<br /><br />while getopts h:p:s:a name<br />do<br /> case $name in<br /> p) show_pid=1; pid=$OPTARG ;;<br /> s) interval=$OPTARG ;;<br /> h|?) usage;<br /> exit 1<br /> esac<br />done<br /><br />/usr/sbin/dtrace -Cws <( cat <<EOF<br /><br /> inline int PID = $pid;<br /> inline int SHOW_PID = $show_pid;<br /> inline int INTERVAL = $interval;<br /><br /><br />#pragma D option quiet<br />#pragma D option aggsortrev<br /><br /><br />dtrace:::BEGIN <br />{<br /> secs = INTERVAL;<br /> printf("Please wait....\n");<br />}<br /><br />io:::start <br />/ (SHOW_PID == 1) && ( pid == PID) /<br />{<br /> self->rw=args[0]->b_flags & B_READ ? "R" : "W";<br /> @files[pid,execname,self->rw,args[2]->fi_pathname] = sum (args[0]->b_bcount);<br /> @total_blks[self->rw]=count();<br /> @total_bytes[self->rw]=sum (args[0]->b_bcount);<br /> self->rw=0;<br />}<br /><br />io:::start <br />/ SHOW_PID == 0 /<br />/* SHOW_PID == 0 && args[2]->fi_pathname != "<none>" */<br />{<br /> self->rw=args[0]->b_flags & B_READ ? "R" : "W";<br /> @files[pid,execname,self->rw,args[2]->fi_pathname] = sum (args[0]->b_bcount);<br /> @total_blks[self->rw]=count();<br /> @total_bytes[self->rw]=sum (args[0]->b_bcount);<br /> self->rw=0;<br />}<br /><br />profile:::tick-1s<br />{<br /> secs--;<br />}<br /><br /><br />profile:::tick-1s<br />/secs == 0/<br />{<br /><br /> trunc(@files,30);<br /> normalize(@files,1024);<br /> system("/usr/bin/clear"); <br /> printf("%Y ",walltimestamp);<br /> printa("%s %@11d blocks, ",@total_blks);<br /> printa("%s %@11d bytes, ",@total_bytes);<br /> printf("\n%6s %-12s %3s %8s %3s\n", "PID", "CMD","R/W", "KB", "FILE");<br /> printa("%6d %-12.12s %1s %@10d %s\n",@files);<br /> secs = INTERVAL;<br /><br />}<br />dtrace:::END<br />{<br /> trunc(@files);<br />}<br />EOF<br />)<br /><br /></pre><br /><br />You can download it here <a href="http://lserinol.googlepages.com/process_io_top">process_io_top</a>. This scripts briefly shows you how to pass arguments from shell to DTrace. Using aggregations and sort them in reverse order by using pragma "aggsortrev". Defining probes (io:::start) more than once with different predicate conditions. Following is a sample output from process_io_top script.<br /><br /><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjc2BT_lsS-ZqHqWRXGXK_dU_N04j-s7meHEN0shZD1P5jABsMWhINx-aBvfBaFfTGrNp6HjhCPkx6VVSyEUKEBt64aDRemSuVPIOyHfxrw7zT4heNxFqK-2m0GBESCROenG2QppxxIil4o/s1600-h/process_io_top.JPG"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 320px; height: 240px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjc2BT_lsS-ZqHqWRXGXK_dU_N04j-s7meHEN0shZD1P5jABsMWhINx-aBvfBaFfTGrNp6HjhCPkx6VVSyEUKEBt64aDRemSuVPIOyHfxrw7zT4heNxFqK-2m0GBESCROenG2QppxxIil4o/s320/process_io_top.JPG" border="0" alt=""id="BLOGGER_PHOTO_ID_5324691022870726338" /></a>Anonymoushttp://www.blogger.com/profile/06161379979453710759noreply@blogger.com2tag:blogger.com,1999:blog-1610041778732887994.post-10041794343011930192009-03-30T23:51:00.000-07:002009-03-31T02:13:15.522-07:00speeding up your nginx server with memcached<a href="http://nginx.net/">Nginx</a> is a high performance web and proxy (web and mail proxy) server. Generally, nginx is used as a front-end proxy server to <a href="http://httpd.apache.org">Apache</a> webserver. Nginx is known to be slow while serving dynamic pages like php. Normally, nginx is using fast-cgi method which is slow. Therefore, it's a good idea to run Apache as back-end server to Nginx and serve dynamic php pages from Apache. If your website's php pages suitable to cache for a certain time, you can use Nginx proxy module and proxy_store command to cache Apache served php pages output in Nginx automatically as html. Here, I'll give you instructions how to use <a href="http://wiki.nginx.org/NginxHttpMemcachedModule">Nginx's memcache module</a> and Danga Software's <a href="http://www.danga.com/memcached/">memcached deamon</a> to store your content in memory and serve it. Serving content from memory will be faster than serving it from disk. memcached's default listening port is 11211. You can find instructions on Danga Software's website how to compile and run memcached.<br /><br />Now, we can look our Nginx configuration for memcache implementation. Let's suppose we have two Apache webservers running on two different physical servers. IP addresses of the Apache webservers are 192.168.2.3 and 192.168.2.4. We'll use those Apache webservers as back-end servers. We have a Nginx server as front-end to them on 192.168.2.1 ip adress. First of all, we have to tell Nginx about those back-end servers. We use <a href="http://wiki.nginx.org/NginxHttpUpstreamModule">Nginx upstream module</a> for this purpose. As you can see below, we defined a upstream named "backend". The configuration has our two Apache webservers ip addresses. Upstream module let's you also give weight to each server in configuration. Our first server's hardware configuration is better than the second one, so we gave the first one weight value 2. This configuration should be in http section of Nginx configration file (nginx.conf).<br /><br /><br /><pre style="border: 1px solid gray; padding: 10px;"><br />upstream backend {<br /> server 192.168.2.3 weight=2;<br /> server 192.168.2.4;<br />}<br /></pre><br />We have created our upstream configuration. Now, we have to tell Nginx, which files will be server by memcache module. I have decided to only serve some image types by memcache. The following configuration part should be in server section of Nginx configuration. The "location" directive tell's the nginx to handle every file which ends with given extensions like .jpg,.png and .gif in url. As first step, Nginx will check the url in memcached. Memcached is simple key value memory database. Every row has a unique key.In our case the key is our url. If Nginx, finds the key (url) in memcached, it will get contents of the key from mecached and send it back to client. This operation is running completely from memory. In case that the key (url) not found, it will fallback to 404 and as you can see, we catch 404 error and send request to our back-end Apache servers. Nginx will then send Apache's response to client.<br /><br /><pre style="border: 1px solid gray; padding: 10px;"><br />location ~* \.(jpg|png|gif)$ {<br />access_log off;<br />expires max;<br />add_header Last-Modified "Thu, 26 Mar 2000 17:35:45 GMT";<br /><span style="font-weight: bold;">set $memcached_key $uri;</span><br /><span style="font-weight: bold;">memcached_pass 127.0.0.1:11211;</span><br /><span style="font-weight: bold;">error_page 404 = /fetch;</span><br />}<br /><br /><span style="font-weight: bold;">location /fetch {</span><br />internal;<br />access_log off;<br />expires max;<br />add_header Last-Modified "Thu, 26 Mar 2000 17:35:45 GMT";<br /><span style="font-weight: bold;">proxy_pass http://backend;</span><br />break;<br /><span style="font-weight: bold;">}</span><br /></pre><br />Of course, we have a drawback here. Nginx's memcache module never put anything automatically in memcached. You have to store your information in it manually by using something like a script. Considering our example, if we forget to store information about a file in memcached, it will be always served by back-end Apache servers. Here is a simple php script, which finds given image types and deploy it into memcached for Nginx.<br /><br /><pre style="border: 1px solid gray; padding: 10px;"><br /><?php<br /><br />function rscandir($base='', &$data=array()) {<br />$array = array_diff(scandir($base), array('.', '..'));<br /><br />foreach($array as $value) :<br /> if (is_dir($base.$value)) {<br /> $data = rscandir($base.$value.'/', $data);<br /><br /> }<br /> elseif (is_file($base.$value)) {<br /> $rest = substr($value, -4);<br /> if ((!strcmp($rest,'.jpg')) || (!strcmp($rest,'.png'))<br /> || (!strcmp($rest,'.gif')) ){<br /> $data[] = $base.$value;<br /> }<br /> }<br /><br />endforeach;<br />return $data;<br />}<br /><br />$mylist=rscandir("/var/www/mysite");<br /><br />$srch = array('/var/www/mysite');<br />$newval = array('');<br /><br />$memcache_obj = memcache_connect("192.168.2.1", 11211);<br /><br />while (list($key, $val) = each($mylist)) {<br /> $url=str_replace($srch,$newval,$val);<br /> echo "$key => $val -> ".filesize($val)."\n";<br /> $value = file_get_contents($val);<br /> memcache_add($memcache_obj, $url, $value, false, 0);<br />}<br />?><br /><br /></pre><br />You need to run this script one time, it will find all given image types and store them into memcached. I run this on one of the Apache back-end servers. It will store data into memcached. This memcached is located on Nginx server which ip address is 192.168.2.1 .Anonymoushttp://www.blogger.com/profile/06161379979453710759noreply@blogger.com14tag:blogger.com,1999:blog-1610041778732887994.post-90886673226039102272009-03-22T03:31:00.000-07:002009-03-22T05:28:14.387-07:00Hammer FilesystemOn Feb 17 2009, new <a href="http://www.dragonflybsd.org/release22/">release 2.2 of DragonFly BSD</a> released which includes a new filesystem called <a href="http://www.dragonflybsd.org/hammer/">hammer</a>. Hammer is a new filesystem which intented to replace ffs on DragonFly systems. Hammer filesystem has many advanced futures compared to ffs filesystem like<br /><ul><li>instant crash recovery</li><li>large file systems and multivolume support</li><li>data integrity checking</li><li>history and snapshots</li><li>reblocking (on the fly defragmentation)</li><li>mirroring<br /></li></ul>Let's start by creating a new hammer filesystem on a new disk device by using hammer's <a href="http://leaf.dragonflybsd.org/cgi/web-man?command=newfs_hammer&section=8">newfs_hammer</a> command. As you can see below, we used the '-L' parameter. Every hammer filesystem requires a label name and it's mandatory.<br /><br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghoQo4LW3gWPtkTGIaGObKepZ5N3PlbA-hUggcWztUni4PpCEWUhL4qmRktL1-fCIgvnZMcEPLJrthadgyo0oZ-e4zW6pG6faqk6Nl4UfM2sn9eDEEGu1imNNajV6WRZLkgnKFLZ1KqPRl/s1600-h/hammer_1.JPG"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer; width: 320px; height: 178px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghoQo4LW3gWPtkTGIaGObKepZ5N3PlbA-hUggcWztUni4PpCEWUhL4qmRktL1-fCIgvnZMcEPLJrthadgyo0oZ-e4zW6pG6faqk6Nl4UfM2sn9eDEEGu1imNNajV6WRZLkgnKFLZ1KqPRl/s320/hammer_1.JPG" alt="" id="BLOGGER_PHOTO_ID_5315971441683108082" border="0" /></a><br /><br />Let's mount in under /mnt directory by using "noatime" option. <a href="http://leaf.dragonflybsd.org/cgi/web-man?command=mount_hammer&section=8">mount_hammer</a> command must be used for this operation.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJDQe-hpEgt_0Bru33gCUrlbqDBJZ1dIt0FOUChub7gGBkkMwWBHpqGewTchZuvDj1Ako2y_JxYXSZPCg_VFhQI38Gj5IyvsVWzT9uAAO4a-w2KY_PDXRlOE_TrFJSXGFBRiKhqyUya2bN/s1600-h/hammer_2.JPG"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer; width: 320px; height: 178px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJDQe-hpEgt_0Bru33gCUrlbqDBJZ1dIt0FOUChub7gGBkkMwWBHpqGewTchZuvDj1Ako2y_JxYXSZPCg_VFhQI38Gj5IyvsVWzT9uAAO4a-w2KY_PDXRlOE_TrFJSXGFBRiKhqyUya2bN/s320/hammer_2.JPG" alt="" id="BLOGGER_PHOTO_ID_5315972054140891314" border="0" /></a><br />Looking to df output you can see our new hammer filesystem labeled as "mydata". Now, let's see where hammer filesystem differs from ffs.<br /><br /><span style="font-weight: bold;font-size:130%;" >History feature</span><br /><br />History metadata of a hammer filesystem is written on every 30 seconds. You can use <a href="http://leaf.dragonflybsd.org/cgi/web-man?command=hammer&section=8">hammer</a> utility to access your files history data. Usage is simple as, calling hammer utility with history command and given your file name which you want to see whole history.<br /><br /># hammer history /mnt/test.txt<br /><br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUfIM6RNOac3Zkf3jJs4QTfj2Fu_9CDr3dImsCp8PEVksTRaf60tWVkUO_fzjti83Oy2bbkT6wBJCSdzf1Kbg3wDpevmlAmTqpej8iZqI9ZYjf1aFdOdu-l0GaDGA0cq8DmTybSYCKnZVg/s1600-h/history_1.JPG"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer; width: 320px; height: 178px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUfIM6RNOac3Zkf3jJs4QTfj2Fu_9CDr3dImsCp8PEVksTRaf60tWVkUO_fzjti83Oy2bbkT6wBJCSdzf1Kbg3wDpevmlAmTqpej8iZqI9ZYjf1aFdOdu-l0GaDGA0cq8DmTybSYCKnZVg/s320/history_1.JPG" alt="" id="BLOGGER_PHOTO_ID_5315973744639844258" border="0" /></a><br />You may be noticed that I have used "sync" command, because I didn't want to wait the the kernel sync operation occur (30 seconds) and forced it by using sync comand to get history result fast.<br /><span style="font-weight: bold;"><br /></span><br /><span style="font-weight: bold;font-size:130%;" >Snapshot feature</span><br /><br />Now let's see hammer's snapshot feature. I have created a directory called "/snapshots" where I want to keep all of my hammer filesystem snaphosts. We'll use hammer utility for taking snapshots.<br /><br /># hammer snapshot /mnt /snapshots/mnt_snapshot1<br /><br />I have created a file on /mnt filesystem and took a snapshot. After a sucsessful snapshot, I deleted the original file from /mnt and accessed it by using the snapshot I took before deleting the file (/snapshots/mnt_snapshot1/test.txt).<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhNZdqddh3TKw0LzRN1EhM3g-FPg7eKEtpg1_ffu2BEEt99e9IXNRhbvOn0e1td4o7JDLi_i7Kme53LCEeFE7hLUm2srJ8hklD-_ngOyvv7AOW998hsZ209ybz1yD_8Qmdu5XpsPQxn5g49/s1600-h/snapshot_1.JPG"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer; width: 320px; height: 178px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhNZdqddh3TKw0LzRN1EhM3g-FPg7eKEtpg1_ffu2BEEt99e9IXNRhbvOn0e1td4o7JDLi_i7Kme53LCEeFE7hLUm2srJ8hklD-_ngOyvv7AOW998hsZ209ybz1yD_8Qmdu5XpsPQxn5g49/s320/snapshot_1.JPG" alt="" id="BLOGGER_PHOTO_ID_5315976941846085202" border="0" /></a><br /><span style="font-weight: bold;font-size:130%;" >Undo Feature</span><br /><br />You can see and rollback every change between the taken snapshots and original files by using hammer's <a href="http://leaf.dragonflybsd.org/cgi/web-man?command=undo&section=1">undo</a> command. You can use "-d" option to get diff output to see every changes on the given file line by line between every snapshot and original one. Also, "-a" option gives you whole change history of given file. Here, I created a test file called "myfile.txt" under my new hammer filesystem (/mnt) and then took a snapshot. After taking snapshot, I added one more line to my test file and used "undo -d /mnt/myfile.txt" to get diff output and see what changes made the file between snapshots.<br /><br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjfqASrVO0J1XC1EdSAdPk-Eix2vbhHlcVVn7NkEyvdphZ4D64HeXa3rPB-OmK3madlgmxOFqO1paDXUwzftMRURgFZPvr0aCaK3mDJudodwcaeOfjZUTh4mNwmU8K6WzZkhBIWApUWvoXQ/s1600-h/undo_1.JPG"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer; width: 320px; height: 178px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjfqASrVO0J1XC1EdSAdPk-Eix2vbhHlcVVn7NkEyvdphZ4D64HeXa3rPB-OmK3madlgmxOFqO1paDXUwzftMRURgFZPvr0aCaK3mDJudodwcaeOfjZUTh4mNwmU8K6WzZkhBIWApUWvoXQ/s320/undo_1.JPG" alt="" id="BLOGGER_PHOTO_ID_5315983733299304386" border="0" /></a><br /><br /><br /><span style="font-weight: bold;font-size:130%;" >Mutivolume Feature </span><br /><br />As I mentioned above, you can use multi volumes and create a big disk volume with hammer filesystem. Now, we'll create a single hammer volume by using two disks (ad1 and ad3).<br />You just call newfs_hammer and mount_hammer commands by only giving two or more disk device names instead of one.Commands are straight forward. First, we need to create new hammer filesystem on disks.<br /><br /># newfs_hammer -L big /dev/ad1 /dev/ad3<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3zY6s6xPYCirvxv4gheuitH5Ha5KYZg_s-chthHO4iXWMriEALeeNdrcxTIa5i_dm0S3w8JRdG-EZTx90PQSBJqLbX2Xd9Mx86BBuhiUq8l2EfZGLymX1N2d6feyeUE6VSXXcGlY4Aupe/s1600-h/multi_1.JPG"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer; width: 320px; height: 178px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3zY6s6xPYCirvxv4gheuitH5Ha5KYZg_s-chthHO4iXWMriEALeeNdrcxTIa5i_dm0S3w8JRdG-EZTx90PQSBJqLbX2Xd9Mx86BBuhiUq8l2EfZGLymX1N2d6feyeUE6VSXXcGlY4Aupe/s320/multi_1.JPG" alt="" id="BLOGGER_PHOTO_ID_5315975341627319666" border="0" /></a><br /><br />I have used to disks and each size is 8GB. After creating new hammer filesystems on disks , we can mount them as one big volume. (I know 8GB is not big :) ).<br /><br /># mount_hammer /dev/ad1 /dev/ad3 /mnt<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3wLvfZ5T5q0XCTWoQxpvLlvLliH7ZcCsMWlq8g12wHLQfQ1wmVWNOOcCZ2xApwJE2bNQCRMEUMgqIU2LaySQUcMbjRUKpYYqzYKas_TC7ZcmwkfgdpPOW9s5mMg0MM_7H-DbOdza9p4wN/s1600-h/multi_2.JPG"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer; width: 320px; height: 178px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3wLvfZ5T5q0XCTWoQxpvLlvLliH7ZcCsMWlq8g12wHLQfQ1wmVWNOOcCZ2xApwJE2bNQCRMEUMgqIU2LaySQUcMbjRUKpYYqzYKas_TC7ZcmwkfgdpPOW9s5mMg0MM_7H-DbOdza9p4wN/s320/multi_2.JPG" alt="" id="BLOGGER_PHOTO_ID_5315975746312699698" border="0" /></a>You can see on /mnt mount point we have a disk in size 16GB, total of two 8GB disks.<br /><br /><span style="font-weight: bold;font-size:130%;" >Mirror Feature</span><br /><br />Hammer uses Pseudo File Systems (PFS) to duplicate inode numbers to slaves. Therefore, mount_null command required for this operation. This duplication is triggered by using hammer utlity with mirror-copy parameter. First we need to create two pfs, called for master and slave. I created them as /mymaster and /myslave. Master volume must be created with pfs-master and slave with pfs-slave parameter by using hammer utility. One thing to take in accounting here is that slave pfs must use master's shared-uuid number.<br /><br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgEYQFHI8baGaM3Xl4Q5qWOjoBwcTL6FyrwSsGe0fthnCUPGNFhcicqX2maFQQLdGaBbGnXs6OkPABnW1SMoSrgSrZz0RrKRrXC19seSM0aqQR-ePDgxk-CpX_Znm_S_uvrpKX_fa9_jAuV/s1600-h/mirror_1.JPG"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer; width: 320px; height: 178px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgEYQFHI8baGaM3Xl4Q5qWOjoBwcTL6FyrwSsGe0fthnCUPGNFhcicqX2maFQQLdGaBbGnXs6OkPABnW1SMoSrgSrZz0RrKRrXC19seSM0aqQR-ePDgxk-CpX_Znm_S_uvrpKX_fa9_jAuV/s320/mirror_1.JPG" alt="" id="BLOGGER_PHOTO_ID_5315980002811089282" border="0" /></a><br /><br />After creating mymaster and myslave pfs we have to use mirror-copy to start initial mirroring operation by using these pfs links , because mount_null can't access them without this copy operation.<br /><br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgO9-MqkRlOp5xW0uEWOcbB9B08JI5uC2yfP5H-IdwJXrzjZSzGNkVF5_ZI6MhVw8MSUEEOi9x1Hr9-n-DPUvlp4zLvWBfHWP2jHGw_htEyOtia-DR7Va9uXjdun8Qd6nvsoN45JezS_1Og/s1600-h/mirror_2.JPG"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer; width: 320px; height: 178px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgO9-MqkRlOp5xW0uEWOcbB9B08JI5uC2yfP5H-IdwJXrzjZSzGNkVF5_ZI6MhVw8MSUEEOi9x1Hr9-n-DPUvlp4zLvWBfHWP2jHGw_htEyOtia-DR7Va9uXjdun8Qd6nvsoN45JezS_1Og/s320/mirror_2.JPG" alt="" id="BLOGGER_PHOTO_ID_5315980728346879682" border="0" /></a><br />After, running first mirror copy operation on our pfs links, we can mount our original hamme volumes where we want and associate them with the pfs links we created and sync (mirror-copy) our mounted hammer volumes.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjC_8-pOPfaE9t44zEcMSBKUd_-1-WgyizmJiRpfqWImSq59qZNmIKT1uOiECK1HOXZeR09mZm9hHf-gHAEhGXTjrrTdo0RCtqfFVbKM6B95jSdronCv5LP__PjcChd5E7U4IyoIiuBg8FE/s1600-h/mirror_3.JPG"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer; width: 320px; height: 178px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjC_8-pOPfaE9t44zEcMSBKUd_-1-WgyizmJiRpfqWImSq59qZNmIKT1uOiECK1HOXZeR09mZm9hHf-gHAEhGXTjrrTdo0RCtqfFVbKM6B95jSdronCv5LP__PjcChd5E7U4IyoIiuBg8FE/s320/mirror_3.JPG" alt="" id="BLOGGER_PHOTO_ID_5315981279588876562" border="0" /></a><br />You can use mount_null mount points to do your filesystems operations. You can see that slave volume (myslave) is read-only and you don't have write access to it. You can only use master volume (mymaster) to make your write operations.If you're looking an alternative filesystem to ffs on BSD systems, I suggest you to evaluate both ZFS on FreeBSD and Hammer on DragonFly BSD.Anonymoushttp://www.blogger.com/profile/06161379979453710759noreply@blogger.com1