In recent articles I covered how I've built a Continuous Delivery pipeline for my blog. These articles talk about using Docker to build a container for my blog, using Travis CI to test and build that container, and finally using a Masterless SaltStack configuration to deploy the blog. Once setup, this pipeline enables me to publish new posts by simply managing them within a GitHub repository.
The nice thing about this setup is that not only are blog posts managed hands-free. All of the servers that host my blog are managed hands-free. That means required packages, services and configurations are all deployed with the same Masterless SaltStack configuration used for the blog application deployment.
The only thing that isn't hands-free about this setup, is installing and configuring the initial SaltStack Minion agent. That is, until today. In this article I am going to cover how to use salt-ssh, SaltStack's SSH functionality to install and configure the salt-minion package on a new server.
How salt-ssh works
A typical SaltStack deployment consists of a Master server running the salt-master process, and one or more Minion servers running the salt-minion process. The salt-minion service will initiate communication with the salt-master service over a ZeroMQ connection and the Master distributes desired states to the Minion.
With this typical setup, you must first have a salt-master service installed and configured, and every server that you wish to manage with Salt must have the salt-minion service installed and configured. The salt-ssh package however, changes that.
salt-ssh is designed to provide SaltStack with the ability to manage servers in an agent-less fashion. What that means is, salt-ssh gives you the ability to manage a server, without having to install and configure the salt-minion package.
Why use salt-ssh instead of the salt-minion agent?
Why would anyone not want to install the salt-minion package? Well, there are a lot of possible reasons. One reason that comes to mind is that some systems are so performance oriented that there may be a desire to avoid performance degradation by running an agent. While salt-minion doesn't normally use a lot of resources, it uses some and that some may be too much for certain environments.
Another reason may be due to network restrictions, for example if the Minions are in a DMZ segment of a network you may want to use salt-ssh from the master so that connections are only going from the master to the minion and never the minion to the master like the traditional setup.
For today's article, the reasoning behind using salt-ssh is that I wish to automate the installation and configuration of the salt-minion service on new servers. These servers will not have the salt-minion package installed by default and I wanted to automate the initial installation of Salt.
Getting started with salt-ssh
Before we can start using salt-ssh to setup our new server we will first need to setup a Master server where we can call salt-ssh from. For my environment I will be using a virtual machine running on my local laptop as the Salt Master. Since we are using salt-ssh there is no need for the Minions to connect to this Master which makes running it from a laptop a simple solution.
Installing SaltStack
On this Salt Master, we will need to install both the salt-master and the salt-ssh packages. Like previous articles we will be following SaltStack's official guide for installing Salt on Ubuntu systems.
Setup Apt
The official install guide for Ubuntu uses the Apt package manager to install SaltStack. In order to install these packages with Apt we will need to setup the SaltStack repository. We will do so using the add-apt-repository command.
$ sudo add-apt-repository ppa:saltstack/salt
Salt, the remote execution and configuration management tool.
More info: https://launchpad.net/~saltstack/+archive/ubuntu/salt
Press [ENTER] to continue or ctrl-c to cancel adding it
gpg: keyring `/tmp/tmpvi1b21hk/secring.gpg' created
gpg: keyring `/tmp/tmpvi1b21hk/pubring.gpg' created
gpg: requesting key 0E27C0A6 from hkp server keyserver.ubuntu.com
gpg: /tmp/tmpvi1b21hk/trustdb.gpg: trustdb created
gpg: key 0E27C0A6: public key "Launchpad PPA for Salt Stack" imported
gpg: Total number processed: 1
gpg: imported: 1 (RSA: 1)
OK
Once the Apt repository has been added we will need to refresh Apt's package cache with the apt-get update command.
$ sudo apt-get update
Get:1 http://security.ubuntu.com trusty-security InRelease [65.9 kB]
Ign http://ppa.launchpad.net trusty InRelease
Ign http://archive.ubuntu.com trusty InRelease
Get:2 http://archive.ubuntu.com trusty-updates InRelease [65.9 kB]
Get:3 http://ppa.launchpad.net trusty Release.gpg [316 B]
Get:4 http://ppa.launchpad.net trusty Release [15.1 kB]
Get:5 http://security.ubuntu.com trusty-security/main Sources [118 kB]
Fetched 10.9 MB in 7s (1,528 kB/s)
Reading package lists... Done
This update command causes Apt to look through it's known package repositories and refresh a local inventory of available packages.
Installing the salt-master and salt-ssh packages
Now that we have SaltStack's Apt repository configured we can proceed with installing the required Salt packages. We will do this with the apt-get install command specifying the two packages we wish to install.
$ sudo apt-get install salt-master salt-ssh
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
git git-man liberror-perl libpgm-5.1-0 libzmq3 python-async python-croniter
python-dateutil python-git python-gitdb python-jinja2 python-m2crypto
python-markupsafe python-msgpack python-smmap python-zmq salt-common
Suggested packages:
git-daemon-run git-daemon-sysvinit git-doc git-el git-email git-gui gitk
gitweb git-arch git-bzr git-cvs git-mediawiki git-svn python-jinja2-doc
salt-doc python-mako
The following NEW packages will be installed:
git git-man liberror-perl libpgm-5.1-0 libzmq3 python-async python-croniter
python-dateutil python-git python-gitdb python-jinja2 python-m2crypto
python-markupsafe python-msgpack python-smmap python-zmq salt-common
salt-master salt-ssh
0 upgraded, 19 newly installed, 0 to remove and 189 not upgraded.
Need to get 7,069 kB of archives.
After this operation, 38.7 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
In addition to the salt-ssh and salt-master packages Apt will install any dependencies that these packages require. With these packages installed, we can now move on to configuring our Salt master.
Configuring salt-ssh to connect to our target minion
Before we can start using salt-ssh to manage our new minion server we will first need to tell salt-ssh how to connect to that server. We will do this by editing the /etc/salt/roster file.
$ sudo vi /etc/salt/roster
With a traditional SaltStack setup the minion agents would initiate the first connection to the Salt master. This first connection is the way the master service identifies new minion servers. With salt-ssh there is no process for Salt to automatically identify new minion servers. This is where the /etc/salt/roster file comes into the picture, as this file used as an inventory of minions for salt-ssh.
As with other configuration files in Salt, the roster file is a YAML formatted file which makes it fairly straight forwarder to understand. The information required to specify a new minion is also pretty straight forward.
blr1-001:
host: 10.0.0.2
user: root
The above information is fairly minimal, the basic definition is a Target name blr1-001, a Hostname or IP address specified by host and then the Username specified by user.
The target name is used when running the salt-ssh command to specify what minion we wish to target. The host key is used by salt-ssh to define where to connect to, and the user key is used to define who to connect as.
In the example above I specified to use the root user. It is possible to use salt-ssh with a non-root user by simply adding sudo: True to the minion entry.
Testing connectivity to our minion
With the minion now defined within the /etc/salt/roster file we should now be able to connect to our minion with salt-ssh. We can test this out by executing a test.ping task against this target with salt-ssh.
$ sudo salt-ssh 'blr1-001' --priv=/home/vagrant/.ssh/id_rsa test.ping
blr1-001:
----------
retcode:
254
stderr:
stdout:
The host key needs to be accepted, to auto accept run salt-ssh with the -i flag:
The authenticity of host '10.0.0.2 (10.0.0.2)' can't be established.
ECDSA key fingerprint is 2c:34:0a:51:a2:bb:88:cc:3b:86:25:bc:b8:d0:b3:d0.
Are you sure you want to continue connecting (yes/no)?
The salt-ssh command above has a similar syntax to the standard salt command called with a typical Master/Minion setup; the format is salt-ssh <target> <task>. In this case our target was blr1-001 the same name we defined earlier and our task was test.ping.
You may also notice that I passed the --priv flag followed by a path to my SSH private key. This flag is used to specify an SSH key to use when connecting to the minion server. By default, salt-ssh will use SaltStack's internal SSH key, which means if you wish to use an alternative key you will need to specify the key with the --priv flag.
In many cases it's perfectly fine to use SaltStack's internal SSH key, in my case the SSH public key has already been distributed which means I do not want to use Salt's internal SSH key.
Bypassing Host Key Validation
If we look at the output of the salt-ssh command executed earlier, we can see that the command was not successful. The reason for this is because this master server has not accepted the host key from the new minion server. We can get around this issue by specifying the -i flag when running salt-ssh.
$ sudo salt-ssh 'blr1-001' --priv /home/vagrant/.ssh/id_rsa -i test.ping
blr1-001:
True
The -i flag tells salt-ssh to ignore host key checks from SSH. We can see from the above salt-ssh execution, when the -i flag is used, everything works as expected.
Specifying a password
In the salt-ssh commands above we used an SSH key for authentication with the Minion server. This worked because prior to setting up Salt, I deployed the public SSH key to the minion server we are connecting with. If we didn't want to use SSH keys for authentication with the Salt Minion for whatever reason, we could also use password based authentication by specifying the password within the roster file.
$ sudo vi /etc/salt/roster
To add a password for authentication, simply add the passwd key within the target servers specification.
blr1-001:
host: 10.0.0.2
user: root
passwd: example
With the above definition salt-ssh will now connect to our minion by establishing an SSH connection to 10.0.0.2 and login to this system as the root user with the password of example. With a password defined, we can rerun our test.ping this time with the --priv flag omitted.
$ sudo salt-ssh 'blr1-001' -i test.ping
blr1-001:
True
Now that a test.ping has returned correctly we can move on to our next step of defining the Salt states to install and configure the salt-minion package.
Using Salt to install Salt
In the Master-less Salt Minions article I had two GitHub repositories setup with various Salt state files. One repository contains salt state files that can be used to setup a generic base system, and the other contains custom state files used to setup the environment running this blog.
Breaking down the minion state
Within this second repository is a salt state file that installs and configures the salt-minion service. Let's take a quick look at this state to understand how the salt-minion package is being installed.
salt-minion:
pkgrepo:
- managed
- humanname: SaltStack Repo
- name: deb http://repo.saltstack.com/apt/ubuntu/14.04/amd64/latest {{ grains['lsb_distrib_codename'] }} main
- dist: {{ grains['lsb_distrib_codename'] }}
- key_url: https://repo.saltstack.com/apt/ubuntu/14.04/amd64/latest/SALTSTACK-GPG-KEY.pub
pkg:
- latest
service:
- dead
- enable: False
/etc/salt/minion.d/masterless.conf:
file.managed:
- source: salt://salt/config/etc/salt/minion.d/masterless.conf
/etc/cron.d/salt-standalone:
file.managed:
- source: salt://salt/config/etc/cron.d/salt-standalone
In the above state we can see that the Salt Apt repository is being added with the pkgrep module. The salt-minion package is being installed with the pkg module and the salt-minion service is being disabled by the service module.
We can also see that two files /etc/salt/minion.d/masterless.conf and /etc/cron.d/salt-standalone are being deployed. For this article, we will use this state as is to perform the initial salt-minion installation and configuration.
Setting up the master
As with the previous article we will use both of these repositories to setup our minion server. We will get started by first using git to clone these repositories into a directory within /srv/salt.
For the first repository, we will clone the contents into a base directory.
$ sudo git clone https://github.com/madflojo/salt-base.git /srv/salt/base
Cloning into '/srv/salt/base'...
remote: Counting objects: 54, done.
remote: Total 54 (delta 0), reused 0 (delta 0), pack-reused 54
Unpacking objects: 100% (54/54), done.
Checking connectivity... done.
The second repository we will clone into /srv/salt/bencane.
$ sudo git clone https://github.com/madflojo/blog-salt.git /srv/salt/bencane
Cloning into '/srv/salt/bencane'...
remote: Counting objects: 46, done.
remote: Total 46 (delta 0), reused 0 (delta 0), pack-reused 46
Unpacking objects: 100% (46/46), done.
Checking connectivity... done.
With all of the salt states now on our local system we can configure Salt to use these state files. To do this we will need to edit the /etc/salt/master configuration file.
$ sudo vi /etc/salt/master
Within the master file the file_roots configuration parameter is used to define where Salt's state files are located on the master. Since we have two different locations for the two sets of state files we will specify them individually as their own item underneath file_roots.
file_roots:
base:
- /srv/salt/base
bencane:
- /srv/salt/bencane
With the above defined, we can now use our Salt states to setup a new minion server. To do this we will once again run salt-ssh but this time specifying the state.highstate task.
$ sudo salt-ssh 'blr1-001' -i state.highstate
----------
ID: salt-minion
Function: pkgrepo.managed
Name: deb http://repo.saltstack.com/apt/ubuntu/14.04/amd64/latest trusty main
Result: True
Comment: Configured package repo 'deb http://repo.saltstack.com/apt/ubuntu/14.04/amd64/latest trusty main'
Started: 21:18:54.462571
Duration: 20440.965 ms
Changes:
----------
repo:
deb http://repo.saltstack.com/apt/ubuntu/14.04/amd64/latest trusty main
----------
ID: salt-minion
Function: pkg.latest
Result: True
Comment: The following packages were successfully installed/upgraded: salt-minion
Started: 21:19:14.903859
Duration: 16889.713 ms
Changes:
----------
dctrl-tools:
----------
new:
2.23ubuntu1
old:
salt-minion:
----------
new:
2016.3.1+ds-1
old:
----------
ID: salt-minion
Function: service.dead
Result: True
Comment: Service salt-minion has been disabled, and is dead
Started: 21:19:32.488449
Duration: 133.722 ms
Changes:
----------
salt-minion:
True
----------
ID: /etc/salt/minion.d/masterless.conf
Function: file.managed
Result: True
Comment: File /etc/salt/minion.d/masterless.conf updated
Started: 21:19:32.626328
Duration: 11.762 ms
Changes:
----------
diff:
New file
mode:
0644
----------
ID: /etc/cron.d/salt-standalone
Function: file.managed
Result: True
Comment: File /etc/cron.d/salt-standalone updated
Started: 21:19:32.638297
Duration: 4.049 ms
Changes:
----------
diff:
New file
mode:
0644
Summary
-------------
Succeeded: 37 (changed=23)
Failed: 0
-------------
Total states run: 37
From the results of the state.highstate task, we can see that 37 Salt states were verified and 23 of those resulted in changes being made. We can also see from the output of the command above that our salt-ssh execution just resulted in the installation and configuration of the salt-minion package.
If we wanted to verify that this is true even further we can use the cmd.run Salt module to execute the dpkg --list command.
$ sudo salt-ssh 'blr1-001' -i cmd.run "dpkg --list | grep salt"
blr1-001:
ii salt-common 2016.3.1+ds-1 all shared libraries that salt requires for all packages
ii salt-minion 2016.3.1+ds-1 all client package for salt, the distributed remote execution system
Summary
With the above, we can see that we were successfully able to install the salt-minion agent to a remote system via salt-ssh. While this may seem like quite a bit of work to setup for a single minion. The ability to install Salt with salt-ssh can be very useful when you are setting up multiple minions, as this same methodology works whether you're installing Salt on 1 or 1,000 minions.