Yesterday while re-purposing a server I was removing packages with
apt-get and stumbled upon an interesting problem. After I removed the package and all of it's configurations, the subsequent installation did not re-deploy the configuration files.
After a bit of digging I found out that there are two methods for removing packages with
apt-get. One of those method should be used if you want to remove binaries, and the other should be used if you want to remove both binaries and configuration files.
What I did
Since the method I originally used caused at least 10 minutes of head scratching; I thought it would be useful to share what I did and how to resolve it.
On my system the package I wanted to remove was
supervisor which is pretty awesome btw. To remove the package I simply removed it with
apt-get remove just like I've done many times before.
# apt-get remove supervisor Reading package lists... Done Building dependency tree Reading state information... Done The following packages will be REMOVED: supervisor 0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded. After this operation, 1,521 kB disk space will be freed. Do you want to continue [Y/n]? y (Reading database ... 14158 files and directories currently installed.) Removing supervisor ... Stopping supervisor: supervisord. Processing triggers for ureadahead ...
No issues so far, the package was removed according to
apt without any issues. However, after looking around a bit I noticed that the
/etc/supervisor directory still existed. As well as the
# ls -la /etc/supervisor total 12 drwxr-xr-x 2 root root 4096 Aug 17 19:44 . drwxr-xr-x 68 root root 4096 Aug 17 19:43 .. -rw-r--r-- 1 root root 1178 Jul 30 2013 supervisord.conf
Considering I was planning on re-installing
supervisor and I didn't want to cause any weird configuration issues as I moved from one server role to another I did what any other reasonable Sysadmin would do. I removed the directory...
# rm -Rf /etc/supervisor
I knew the supervisor package was removed, and I assumed that the package didn't remove the config files to avoid losing custom configurations. In my case I wanted to start over from scratch, so deleting the directory sounded like a reasonable thing.
# apt-get install supervisor Reading package lists... Done Building dependency tree Reading state information... Done The following NEW packages will be installed: supervisor 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded. Need to get 0 B/314 kB of archives. After this operation, 1,521 kB of additional disk space will be used. Selecting previously unselected package supervisor. (Reading database ... 13838 files and directories currently installed.) Unpacking supervisor (from .../supervisor_3.0b2-1_all.deb) ... Processing triggers for ureadahead ... Setting up supervisor (3.0b2-1) ... Starting supervisor: Error: could not find config file /etc/supervisor/supervisord.conf For help, use /usr/bin/supervisord -h invoke-rc.d: initscript supervisor, action "start" failed. dpkg: error processing supervisor (--configure): subprocess installed post-installation script returned error exit status 2 Errors were encountered while processing: supervisor E: Sub-process /usr/bin/dpkg returned an error code (1)
However, it seems supervisor could not start after re-installing.
# ls -la /etc/supervisor ls: cannot access /etc/supervisor: No such file or directory
There is good reason why
supervisor wouldn't restart; because the
/etc/supervisor/supervisord.conf file was missing. Shouldn't the package installation deploy the
supervisord.conf file? Well, technically no. Not with the way I removed the
Why it didn't work
How remove works
If we look at
apt-get's man page a little closer we can see why the configuration files are still there.
remove remove is identical to install except that packages are removed instead of installed. Note that removing a package leaves its configuration files on the system.
As the manpage clearly says, remove will remove the package but leaves configuration files in place. This explains why the
/etc/supervisor directory was lingering after removing the package; but it doesn't explain why a subsequent installation doesn't re-deploy the configuration files.
If we use
dpkg to look at the
supervisor package, we will start to see the issue.
# dpkg --list supervisor Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-================================-=====================-=====================-======================================== rc supervisor 3.0b2-1 all A system for controlling process state
dpkg package manager a package can have more states than just being installed or not-installed. In fact there are several package states with
- not-installed - The package is not installed on this system
- config-files - Only the configuration files are deployed to this system
- half-installed - The installation of the package has been started, but not completed
- unpacked - The package is unpacked, but not configured
- half-configured - The package is unpacked and configuration has started but not completed
- triggers-awaited - The package awaits trigger processing by another package
- triggers-pending - The package has been triggered
- installed - The packaged is unpacked and configured OK
If you look at the first column of the
dpkg --list it shows
r in this column means the package is
remove, which as we saw above means the configuration files are left on the system. The
c in this column shows that the package is in the state of
config-files. Meaning, only the configuration files are deployed on this system.
apt-get install the
apt package manager will lookup the current state of the package, when it sees that the package is already in the
config-files state it simply skips the configuration file portion of the package installation. Since I manually removed the configuration files outside of the
dpkg process the configuration files are gone and will not be deployed with a simple
How to resolve it and remove configurations properly
Purging the package from my system
At this point, I found myself with a broken installation of
supervisor. Luckily, we can fix the issue by using the
purge option of
# apt-get purge supervisor Reading package lists... Done Building dependency tree Reading state information... Done The following packages will be REMOVED: supervisor* 0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded. 1 not fully installed or removed. After this operation, 1,521 kB disk space will be freed. Do you want to continue [Y/n]? y (Reading database ... 14158 files and directories currently installed.) Removing supervisor ... Stopping supervisor: supervisord. Purging configuration files for supervisor ... dpkg: warning: while removing supervisor, directory '/var/log/supervisor' not empty so not removed Processing triggers for ureadahead ...
Purge vs Remove
purge option of
apt-get is similar to the
remove function however with one difference. The
purge option will remove both the package and configurations. After running
apt-get purge we can see that the package was fully removed by running
dpkg --list again.
# dpkg --list supervisor dpkg-query: no packages found matching supervisor
Re-installation without error
Now that the package has been fully purged, and the state of it is now
not-installed; we can re-install without errors.
# apt-get install supervisor Reading package lists... Done Building dependency tree Reading state information... Done The following NEW packages will be installed: supervisor 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded. Need to get 0 B/314 kB of archives. After this operation, 1,521 kB of additional disk space will be used. Selecting previously unselected package supervisor. (Reading database ... 13833 files and directories currently installed.) Unpacking supervisor (from .../supervisor_3.0b2-1_all.deb) ... Processing triggers for ureadahead ... Setting up supervisor (3.0b2-1) ... Starting supervisor: supervisord. Processing triggers for ureadahead ...
As you can see from the output above, the
supervisor package has been installed and started. If we check the
/etc/supervisor directory again we can also see the necessary configuration files.
# ls -la /etc/supervisor/ total 16 drwxr-xr-x 3 root root 4096 Aug 17 19:46 . drwxr-xr-x 68 root root 4096 Aug 17 19:46 .. drwxr-xr-x 2 root root 4096 Jul 30 2013 conf.d -rw-r--r-- 1 root root 1178 Jul 30 2013 supervisord.conf
You should probably just use purge in most cases
After running into this issue I realized, most of the times I ran
apt-get remove I really wanted the functionality of
apt-get purge. While it is nice to keep configurations handy in case we need them after re-installation, using
remove all the time also leaves random config files to clutter your system. Free to cause configuration issues when packages are removed then re-installed.
In the future I will most likely default to
Recently Benjamin published his first book; Red Hat Enterprise Linux Troubleshooting Guide. In addition to writing, he has several Open Source projects focused on making Ops easier. These projects include Automatron, a project enabling auto-healing infrastructure for the masses.
Identify, capture and resolve common issues faced by Red Hat Enterprise Linux administrators using best practices and advanced troubleshooting techniques
What people are saying:
Excellent, excellent resource for practical guidance on how to troubleshoot a wide variety of problems on Red Hat Linux. I particularly enjoyed how the author made sure to provide solid background and practical examples. I have a lot of experience on Red Hat but still came away with some great practical tools to add to my toolkit. - Amazon Review