Checking the number of LWP (threads) in Linux

Linux has these nice little processes called LWP (Light Weight Process) or otherwise known as threads. Generally these are spawned by 1 master process that will show up in your normal ps output.

# ps -elf | wc -l  
145

So does this mean your system only has 145 processes running? No. If you run ps with a -T you will see all of the threads as well.

# ps -elfT | wc -l  
275

As you can see the process count jumped significantly due to threads. Normally this is never any type of problem, However sometimes a process (usually java based) will have a thread leak. This can cause your system to run into system limits.

Specifically one limit I have seen systems hit is the kernel.pid_max limit.

# sysctl -a | grep kernel.pid_max  
kernel.pid_max = 32768

As you can see on my system the pid_max is 32768, this means the system can only give out 32768 PID's at one time (it will rollover if pid #‘s are available).

The reason threads come into play here is because each thread has a PID, and SPID number. The SPID's also take from the pid_max number.

To see the number of threads one specific process is using you can do the following.

# ps -p 2089 -lfT  
F S UID PID SPID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD  
0 S bcane 2089 2089 1 0 80 0 - 29457 poll_s 09:17 ? 00:00:09 gnome-terminal  
1 S bcane 2089 2091 1 0 80 0 - 29457 poll_s 09:17 ? 00:00:00 gnome-terminal  
1 S bcane 2089 2094 1 0 80 0 - 29457 pipe_w 09:17 ? 00:00:00 gnome-terminal

The lesson for today, is watch your threads!

Avatar
Benjamin Cane
Principal Engineer, Vice President

Benjamin Cane is Principal Engineer at American Express. He has more than 16 years of experience with roles in both systems and software engineering. He leverages both his systems and software skills to build end-to-end platforms. Platforms, purpose built for performance and resiliency. Benjamin is also the author of Red Hat Enterprise Linux - Troubleshooting Guide (2015, Packt Publishing), and he has published many popular articles on topics such as Linux, Docker, Python, Go and Performance Tuning. Thoughts and Opinions expressed in my articles are my own.

Next