Yesterday while re-purposing a server I was removing packages with apt-get and stumbled upon an interesting problem. After I removed the package and all of it's configurations, the subsequent installation did not re-deploy the configuration files.
After a bit of digging I found out that there are two methods for removing packages with apt-get. One of those method should be used if you want to remove binaries, and the other should be used if you want to remove both binaries and configuration files.
Recently I have been looking for ways to allow external tools and services to perform corrective actions across my infrastructure automatically. As an example, I want to allow a monitoring tool to monitor my nginx availability and if for whatever reason nginx is down I want that monitoring tool/service to do something to fix it.
While I was looking at how to implement this, I remembered that SaltStack has an API and that API can provide exactly the functionality I wanted.
When I was first got started with administrating Linux and Unix servers I was working in an environment where there were tons of adhoc scripts that other admins wrote. From time to time I would find myself troubleshooting why one of those scripts suddenly stopped working. Sometimes the scripts would be well written and easy to understand, other times they were clunky and confusing.
While troubleshooting the poorly written scripts was always a hassle at the time, it taught me an important lesson.
Configuration management and automation tools like SaltStack are great, they allow us to deploy a configuration change to thousands of servers with out much effort. However, while these tools are powerful and give us greater control of our environment they can also be dangerous. Since you can roll out a configuration change to all of your servers at once, it is easy for that change to break all of your servers at once.
If you haven't seen it yet there is a new troubleshooting tool out called sysdig. It's been touted as strace meets tcpdump and well, it seems like it is living up to the hype. I would actually rather compare sysdig to SystemTap meets tcpdump, as it has the command line syntax of tcpdump but the power of SystemTap.
In this article I am going to cover some basic and cool examples for sysdig, for a more complete list you can look over the sysdig wiki.
One of my biggest pet peeves as a Linux sysadmin is when I see users, or even other sysadmins using kill -9 on the first attempt to terminate a process. The reason this bugs me so much is because it shows either a lack of understanding of the kill command or just plain laziness. Rather than going on a long rant about why this is bad, I wanted to write an article about the kill command and how signal works in Linux.
Today is very much a “back to the basics” kind of day. In this article I am going to cover one of the most basic commands in Linux; the cd command. While today's article might be basic; it is always good even for experienced sysadmins, to look back at some of the basics and see if there are ways to improve your command line skills and Linux knowledge.
The Linux/Unix directory structure Before getting into how to change to another directory, let's take a minute to cover how Linux's directory structure is laid out.
Lately if you have been paying attention to tech or even mainstream media you might have seen a few stories about data breaches. Sometimes these data breaches have allowed attackers to gather unencrypted passwords or credit card numbers. In the past these types of attacks still happened, but there was not as many attacks as today and when they happened they were kept secret. With more and more internet based services becoming part of peoples lives, there is even more targets for attackers who are looking to get sensitive data.
Ever run a shell script and get the following error?
# ./dosfile.sh : bad interpreter: No such file or directory The error may look like there is a problem with your scripts SHEBANG where you specify the interpreter, so you go look and the line contains #!/bin/bash which is correct. So then you start wondering if there is a problem with the /bin/bash binary, and all sorts of thoughts of what would happen if /bin/bash was missing or broken start racing through your head.
Today's article is an item I covered briefly during my presentation at SaltConf 2014 (which was a pretty awesome conference by the way). One of the lesser known features of SaltStack is the ability to configure multiple master servers. Having an additional master server allows for some extra redundancy as well as capacity for large implementations. While I covered the benefits of having an additional master server in my presentation I didn't cover in full detail how to set this up, today I will cover the details of configuring multiple salt masters.