Signal TRAPS

What Are Signals and how to use them

To run production applications best, we must know how apps run.

What Are Signals?

In Unix, Linux, and other POSIX-Compliant operating systems, signals are a process control mechanism that sends shut down notifications to running processes.

 

If you've stopped a process on a POSIX-Compliant system, then you've used signals.

Why are they IMportant

Signals are essential for both ops and developers to understand. 

 

For Ops, whether it is Docker, Kubernetes, or a Stand-Alone process on Linux, to shut down a service a signal is involved.

 

For Developers, signals tell our software to stop. We can use this to gracefully shutdown our applications.

How signals work

When stopping a process, it is typical for users to use the kill command. This command will send the specified signal to the system's Kernel. The Kernel will then either kill the process itself or forward the signal to the target process.

Common Signals

There are many different types of signals, and each has a default behavior. There are also some with undocumented conventions such as SIGHUP.

  • SIGHUP [1] - Generally used to signal configuration refresh or in multi-process services indicate the control process should restart without restarting workers.
  • SIGINT [2] - Sent to a process from a CRTL+C press and used to stop the process.
  • SIGQUIT [3] - Sent to a process from a CTRL+\ press and used to stop the process and create a core dump.
  • SIGKILL [9] - This signal is never sent to the process as it triggers the Kernel to force stop the process.
  • SIGTERM [15] - The Default signal sent when executing the kill command with no options. This signal is the primary signal used to stop processes.

Note: The signals referenced with the associated [value] represent the Ubuntu implementation. A reasonably common application. To determine the Signal to Value mapping on your operating system, reference the manpages via man 7 signal.

Trapping Signals

We can bypass the default actions for a signal by using signal traps.

 

A signal trap, as shown, will allow the application to capture the SIGTERM signal. The goal is to avoid an immediate shutdown, allowing for any pre-shutdown activity.

 

When transactional safety is vital to an application, this concept of a graceful shutdown is a crucial resiliency feature.

// Kick off a Graceful Shutdown Goroutine
go func() {
	// Make the Trap
	trap := make(chan os.Signal, 1)
	signal.Notify(trap, syscall.SIGTERM)

	// Wait for a signal then action
	s := <-trap
	log.Infof("Received shutdown signal %s", s)

	// Shutdown the HTTP Server
	err := srv.httpServer.Shutdown(context.Background())
	if err != nil {
		log.Errorf("Error shutting down %s", err)
	}

	// Shutdown the app
	os.Exit(0)
}()

For a full service example check out: github.com/madflojo/healthchecks-example

Signals in the Wild

In today's environments, there are many ways to run applications. The most common being the following.

 

  • Stand-Alone Process - This is the more traditional approach to run a service. Generally, non-container processes will run via systemd, a task scheduler, or some other process manager. But not always, some applications have their little startup quirks.
  • Docker Containers - Docker, has changed how applications run in production. While it's a little less common to run Docker alone, it's by no means rare.
  • Kubernetes - Many applications run as containers within Kubernetes clusters. Kubernetes has its way of managing the application lifecycle.

Stand-Alone Processes

A stand-alone process is a process that runs without any process manager or container. These processes start with simple start scripts or a series of shell commands.
 

To stop these processes, we leverage the kill command. When executed with no options, the kill command sends a SIGTERM. If another signal is required, there are two ways to do that. We can send the signal using the signal number (i.e., 1 for SIGHUP, 9 for SIGKILL). Or, we can specify the signal by name with the -s option.

# Send a SIGTERM to PID 7158
$ kill 7158

# Send a SIGHUP to PID 7158
$ kill -1 7158
# Or
$ kill -s SIGHUP 7158

# Send the kernal a SIGKILL for PID 7158
$ kill -9 7158
# Or
$ kill -s SIGKILL 7158

Signals in the Wild

Docker

A word of caution, Docker, will not send signals to all processes within the container. Docker will only forward the signal to the primary running process. This information is vital for containers that use entry-point scripts. Check out this article to learn more about making sure entry-point scripts pickup shutdown signals.

# Stop a container with a SIGTERM
$ docker stop some_container

# Stop a container with a SIGTERM, and
# a max stop time (30 seconds).
$ docker stop -t 30 some_container

# Stop a container with a SIGKILL
$ docker kill some_container

# Stop a container with a SIGINT
$ docker kill -s SIGINT some_container

Signals in the Wild

Many users of Docker never realize that Docker uses signals to stop processes. By default, when you use the stop option to stop a container, Docker will send the process a SIGTERM. To send alternative signals, we can use the kill option, which defaults to SIGKILL. As a general practice, avoid using SIGKILL (more about this later).

Kubernetes

Like Docker, Kubernetes will send containerized applications a SIGTERM. If the container fails to finish in time, Kubernetes will then use a SIGKILL to kill the app immediately.

 

By default, Kubernetes will wait 30 seconds for a container to stop. We can change this default by using the --grace-period option. This setting is known as terminationGracePeriodSeconds within the Pod Spec.

Signals in the Wild

A Small Rant about kill -9

# This is a last resort
$ kill -9 1234

# It's the same as this
$ kill -s SIGKILL 1234

SIGKILL is sent to the kernel and never sent to the target process. Meaning the process is never able to cleanly shutdown.
 

While it's essential to know how to kill a service that doesn't seem to want to stop on its own, it's equally important to understand the ramifications of using SIGKILL. When executing SIGKILL, the application is not cleanly closing connections; it's not closing file handlers, it's only stopped.


Use the SIGKILL or -9 option only as a last resort, never during the first attempt.

Summary

  • Signals control how processes shut down, they have defaults, but you can also trap them to override the default action.
  • While SIGTERM is a less impactful shutdown signal unless you write traps, the process will still not shut down cleanly.
  • Docker and Kubernetes both have a time limit to how long they will wait for a container to stop. Make sure you give applications enough time.
  • Knowing signals and how they notify processes helps you understand how the overall system and application work together.
  • It also helps you understand why using kill -9 is a bad thing, seriously people stop running it unnecessarily.

EOF

Benjamin Cane

- Twitter: @madflojo 
- LinkedIn: Benjamin Cane 
- WebSite: BenCane.com

Like this? Check out some of my other stuff:

You can also share this on Twitter, LinkedIn, or Reddit!

Principal Engineer - American Express