troubleshooting

Adding and Troubleshooting Static Routes on Red Hat based Linux Distributions

Adding static routes in Linux can be troublesome, but also absolutely necessary depending on your network configuration. I call static routes troublesome because they can often be the cause of long troubleshooting sessions wondering why one server can't connect to another. This is especially true when dealing with teams that may not fully understand or know the remote servers IP configuration. The Default Route Linux, like any other OS has a routing table that determines what is the next hop for every packet.

Advanced Linux System Statistics and Diagnostics with SystemTap

In one of the first posts of this blog I covered some basic SystemTap functionality from an email that I sent to members of my team, but I have always felt that I haven't given SystemTap as thorough of an article as this incredible tool deserves. Today I want to correct that. For today's article I will show how to compile SystemTap scripts on one server while running the compiled module on a production server without installing debug-info or devel packages in production.

Troubleshooting High I/O Wait in Linux

Linux has many tools available for troubleshooting some are easy to use, some are more advanced. I/O Wait is an issue that requires use of some of the more advanced tools as well as an advanced usage of some of the basic tools. The reason I/O Wait is difficult to troubleshoot is due to the fact that by default there are plenty of tools to tell you that your system is I/O bound, but not as many that can narrow the problem to a specific process or processes.

When Zombies Invade Linux: What are Zombie Processes and What to do about them

Zombies don't just appear in scary movies anymore, sometimes they also appear on your Linux systems; but don't fret they are mostly harmless. What is a Zombie Process? Before we get started I wanted to first cover what exactly a Zombie process is. Linux and Unix both have the ability for a process to create a sub process otherwise known as a “Child Process”. Once a process creates a new sub process the first process then becomes a “Parent Process” as it has spawned a child process during its execution.

Linux Troubleshooting with strace

Today I want to cover one of the best troubleshooting tools in any sysadmins arsenal; strace. Strace is a command that will trace the system calls and signals from a specified command. What does that mean in layman's terms? Strace will output all of the inner workings of a process you run it against. If a process opens a file or binds a port, strace will print that action; it is a great utility for troubleshooting when a process is not behaving as expected and you can't find any reason in the commands output or log files.

kill: Creating a core dump

Sometimes when dealing with application problems you run into a point where logs and environmental data just don't seem to provide you with the exact issue. For this reason there are many times where a developer may ask you to create a core file for their application. Core files contain a lot of very good information around the application such as process information and the data stored in memory. Sometimes this can be the difference in finding the issue and not finding the issue.

ulimit: Checking the user limits of running processes

The /proc filesystem is a wealth of useful information. Found this one today looking for something unrelated. You can read the limits file of a running process to find its currently defined limits, this is helpful for things like Max open files. You can use this to verify that the settings in /etc/security/limits.conf matches the running process. Example: [[email protected] ~]$ cat /proc/5610/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 3874 3874 processes Max open files 1024 1024 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 3874 3874 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us