In one of the first posts of this blog I covered some basic SystemTap functionality from an email that I sent to members of my team, but I have always felt that I haven't given SystemTap as thorough of an article as this incredible tool deserves. Today I want to correct that.
For today's article I will show how to compile SystemTap scripts on one server while running the compiled module on a production server without installing debug-info or devel packages in production.
Linux has many tools available for troubleshooting some are easy to use, some are more advanced.
I/O Wait is an issue that requires use of some of the more advanced tools as well as an advanced usage of some of the basic tools. The reason I/O Wait is difficult to troubleshoot is due to the fact that by default there are plenty of tools to tell you that your system is I/O bound, but not as many that can narrow the problem to a specific process or processes.
Whenever I perform any type of activity that requires me to look at historical system statistics such as load average, CPU utilization, I/O wait state, or even memory usage; I usually skip the System Monitoring Applications like Nagios or Zenoss and start running the sar command. While I'm not saying that sar completely replaces those tools, I am saying that sar is quick and dirty and if all you want is some raw numbers from a certain time frame; sar is a great tool.
Sar is a utility that will grab cpu, disk i/o, memory, and network statistics on a 10 minute interval. Sar is great for grabbing system statistics quickly; I use sar whenever I'm doing any kind of performance tuning or troubleshooting.
To use sar all you need to do is install the sysstat package. The package will install a cron in /etc/cron.d that will kick off a sar capture every 10 minutes.