Salt Guide

Salt can be used as a configuration management tool similar to Ansible or Puppet.

This guide describes orcharhino’s Salt support. It describes how to install the Salt plugin on orcharhino, how to integrate it with your Salt master, as well as its usage and supported features.

Salt supports two distinct modes of operation: Clientless via SSH or using the Salt minion client software. Currently, the orcharhino Salt plugin supports exclusively the Salt minion approach.

Summary of supported features:

Importing Salt states, assigning them to Salt environments, and finally associating those Salt environments with hosts or host groups.
Using orcharhino as an external node classifier (ENC) for Salt. Salt variables can be set for Salt states similar to Puppet smart class parameters. orcharhino parameters are made available as Salt pillars which can in turn be used as variables in Salt states, too.
Viewing configuration management reports and facts (aka Salt grains), from within orcharhino.
Running an arbitrary Salt function (e.g. state.highstate) from the management UI.

Installation

When installing a new orcharhino, it is possible to select Salt as a configuration management system during step five of the main orcharhino installation steps. Choosing this option will install and configure both the Salt plugin and a Salt master on your orcharhino.

To install the Salt Plugin on an existing orcharhino installation, use the following command:

foreman-installer --enable-foreman-plugin-salt --enable-foreman-proxy-plugin-salt

This will install the orcharhino Salt plugin as well as the smart proxy functionality which is always included in your orcharhino.

If you do not yet have a working Salt master, you will need to install one. The Salt master can either run on your orcharhino or on any orcharhino proxy with the Salt feature enabled. It is also possible to use an existing Salt master by installing and configuring the relevant orcharhino proxy features on the existing Salt master host.

For more information on installing a Salt master, please consult the official SaltStack documentation. Alternatively, you can use the bootstrap instructions found in the official SaltStack package repository.

It is recommended to use the same Salt version on the Salt master as you are using on your Salt minions.

Manual Configuration

The following steps describe how to integrate a Salt master with the orcharhino Salt plugin.

If you select Salt during step five of the main orcharhino installation steps, then these steps are done automatically on orcharhino itself. The following subsections can be used as a reference for manual installations on orcharhino or, if necessary, on an orcharhino proxy. This is needed when adding Salt support to an existing orcharhino installation, when adding an existing Salt master to orcharhino, or when setting up Salt on an orcharhino proxy.

Unless otherwise stated, all of the following actions are to be performed on the host that the Salt master is on. This will either be the orcharhino or an orcharhino proxy with the relevant smart proxy functionality enabled.

System Configuration

Add the following to the /etc/sudoers:

Cmnd_Alias SALT = /usr/bin/salt, /usr/bin/salt-key
foreman-proxy ALL = NOPASSWD: SALT
Defaults:foreman-proxy !requiretty

Add a user saltuser which will be used for the Salt API:

adduser --no-create-home -s /bin/false -d / saltuser
passwd saltuser
# enter a password for the saltuser twice

The command adduser saltuser -p <salt_password> won’t work. Using it will prevent you from importing any Salt states later.

Salt Master Configuration

Change the following parts in /etc/salt/master so that Salt will use orcharhino as the external node classifier (ENC):

master_tops:
  ext_nodes: /usr/bin/foreman-node

Enable Salt pillar data for use with orcharhino:

ext_pillar:
  - puppet: /usr/bin/foreman-node

The following construct adds a Salt environment called base which is associated with the /srv/salt directory:

file_roots:
  base:
    - /srv/salt

The autosign mechanism is used to allow Salt minions to connect to the Salt master. For enabling the autosign file which the orcharhino proxy will use, change the following line in /etc/salt/master:

autosign_file: /etc/salt/autosign.conf

Add the /etc/salt/autosign.conf file with appropriate ownership and permissions:

touch /etc/salt/autosign.conf
chgrp foreman-proxy /etc/salt/autosign.conf
chmod 660 /etc/salt/autosign.conf

Use the saltuser for the Salt API and specify needed connection settings in /etc/salt/master:

external_auth:
  pam:
    saltuser:
      - '@runner'

rest_cherrypy:
  port: 9191
  host: 0.0.0.0
  ssl_key: /etc/puppetlabs/puppet/ssl/private_keys/orcharhino.<domain>.pem
  ssl_crt: /etc/puppetlabs/puppet/ssl/certs/orcharhino.<domain>.pem

It’s possible to run any common command via the Salt connection on the Salt minion with Salt as remote execution provider. This enhances orcharhino to reuse the Salt connection without relying on SSH anymore. To use Salt as a remote execution provider, you need to configure the following in /etc/salt/master to allow the foreman-proxy user to run Salt commands:

publisher_acl:
  foreman-proxy:
    - state.template_str

Salt Grains Upload

To enable the upload of Salt grains (analogous to Puppet facts) to orcharhino, you will need to adjust the configuration in /etc/salt/foreman.yaml (on the host that your Salt master is on).

:proto: https
:host: orcharhino.<domain>
:port: 443
:ssl_ca: "/etc/puppetlabs/puppet/ssl/ssl_ca.pem"
:ssl_cert: "/etc/puppetlabs/puppet/ssl/client_cert.pem"
:ssl_key: "/etc/puppetlabs/puppet/ssl/client_key.pem"
:timeout: 10
:salt: /usr/bin/salt
:upload_grains: true

Make sure you adapt this example as appropriate for your installation.

Salt API Configuration

Edit /etc/foreman-proxy/settings.d/salt.yml and configure the API-related settings (make sure to use the correct password for the saltuser):

:use_api: true
:api_auth: pam
:api_url: https://orcharhino.example.com:9191
:api_username: saltuser
:api_password: <salt_password>

Activation

For the changes to take effect, we need to restart the relevant services. You will need to run the following command on the host that your Salt master is on. This is either your orcharhino or any attached orcharhino proxy:

systemctl restart salt-master salt-api

You will also need to run the following command on your orcharhino as well as on any attached orcharhino proxy based Salt master installation you might have:

foreman-maintain service restart

The first time you install (i.e. enable) a new orcharhino proxy feature, you will also need to refresh the available orcharhino proxy features from within orcharhino:

In the management UI, go to the smart proxies page (Infrastructure > Smart Proxies) and select the Refresh action from the drop down menu for the relevant orcharhino proxies.

Minion Setup

Hosts that are to be managed using orcharhino’s Salt features will require the Salt minion client software. The rest of this section provides a best practice recommendation for how to provide such hosts with the needed software, and how to deploy them using orcharhino.

Using orcharhino’s Content Management

Since it is recommended to use the same Salt version on your Salt minions as you are using on your Salt master, we recommend using orcharhino’s content management to provide your hosts with the correct version of this client software.

The following steps are opinionated. They make certain assumptions about your setup and orcharhino usage. Feel free to make adjustments if needed.

The following are step-by-step instructions for how to achieve this:

Create and sync the needed products and repositories.

Create a new product named SaltStack.

For each operating system supported by orcharhino that you want to install Salt minion client software on, create a new repository within your new SaltStack product from the previous step.

We recommend the following naming schema for these repositories: SaltStack for <os_name> (e.g. SaltStack for Ubuntu Focal (20.04)).

Find the correct SaltStack package repository from the official SaltStack package repository for the Upstream URL for each repository. Here, correct implies for your desired operating system, as well as in the correct version (the same version used by your Salt master). For example https://repo.saltstack.com/py3/ubuntu/20.04/amd64/3001/.

Creating, and publishing the needed content views.
1. Create an orcharhino content view for each repository from step 1b.
  
  We recommend using the same name for the content view, as you did for the relevant repository. Don’t forget to add this repository to the content view.
2. Publish an initial content view version for each of your content views from the previous step.

Make the new content available, using the relevant composite content views.

We recommend creating a single composite content view for each major version of each operating system you are using. That way, all content hosts using a specific major version, of a specific operating system, can all use this content view. Because it is a composite content view, it can be easily extended with additional content, as needed in the future.

Add each of your operating system specific SaltStack content views to the main composite content view for that operating system and version.
Publish a new version of the composite content view from the previous step.
Promote the new content view version from the previous step to your lifecycle environments as appropriate.
If needed, create new activation keys for your composite content view plus lifecycle environment combinations.

Deploying Minion Hosts

Once everything is correctly configured (as laid out in this guide), you can automatically deploy new hosts, that are fully provisioned and configured for Salt usage. All you need to do is to make sure that the Salt Master and Salt Environment fields are set during host creation and that the host will use the content you set up in the previous section.

The preferred way to achieve this is to set up a host group that has these fields set to the desired values, and also uses the correct content view, lifecycle environment, and activation key (as set up in the previous section).

Any new hosts deployed via such a host group, will automatically install and configure the needed Salt minion client software, and will automatically register with the Salt master.

Connection Test

Once you have deployed your first Salt minion, you can verify that the communication between minion and Salt master works using the following command (on your Salt master host):

salt "*" test.ping

Print all grains of all connected Salt minions:

salt "*" grains.items

Usage

As noted above, every orcharhino managed Salt minion host is always associated with a Salt master and a Salt environment. The name of the associated Salt environment within orcharhino must match the actual Salt environment from the file_roots option in the /etc/salt/master file. (This is generally ensured, since the orcharhino environment is chosen from a drop down menu.) Once this association stands (and the Salt minion software is installed), the orcharhino host is also a Salt minion, and by extension ready for Salt usage.

Salt States

A Salt state configures a certain part of a host - e.g. writes the configuration file of a service or installs an application. Salt States can be imported from the Salt master to orcharhino. The Salt master configuration used above uses a Salt environment named base which includes the Salt states stored in /srv/salt/ Navigate to the Salt states page to import these states. This will display a list of newly found states as follows:

Salt states

Using the Update action on each Salt state, you can change the name of the Salt state and change the Salt environment assignment. To remove a Salt State from orcharhino, use the Delete action. This will only remove the Salt state from orcharhino, not from the disk of the Salt master.

Later on, the states are assigned to a host or to a host group. Salt will apply these states to any hosts they are assigned to whenever state.highstate is run.

Assigning Salt states to a host or host group

The paths for states and pillars have to be configured in /etc/salt/master. The conventional default is /srv/salt for states and /srv/pillar for pillars.

Salt Keys

The Salt keys interface lists hosts and their Salt keys. You can manually accept, reject, or delete keys from here:

Salt key management

Salt Autosign

The Salt autosign feature can be used to automatically accept signing requests from hosts. This is not normally needed for hosts managed by orcharhino, since such hosts will be supplied with a Salt key during host provisioning. orcharhino is able to manage the Salt autosign entries:

Salt autosign management

Salt Reports

Salt reports are uploaded to orcharhino every 10 minutes via the /etc/cron.d/smart_proxy_salt cron job. This action can also be triggered manually by running the following command on the host your Salt master is on:

/usr/sbin/upload-salt-reports

Salt reports can be viewed on the config management reports page (navigate to Monitor > Config Management Reports in the management UI). The reports associated with individual hosts can also be viewed via the relevant host overview page.

Salt Variables

Similar to Puppet smart class parameters, you can configure Salt variables on the orcharhino management UI. The configured values are then available as pillar data.

Creating a new Salt variable

Refer to the Puppet smart class parameters section on how to configure the variables' data.

ENC Parameters

As mentioned already, orcharhino can be used as an external node classifier (ENC) for Salt. To view what is being made available for Salt usage with some given host, you can press the Salt ENC button on the host overview page of the relevant host. You will see a list of parameters that are made available for Salt usage as pillar data.

You can also check, what parameters are truly available on the Salt side, by running the following command on your Salt master:

salt '*' pillar.items

If you feel some parameter is missing, you may need to refresh the pillar data first:

salt '*' saltutil.refresh_pillar

Running Salt

The Salt plugin leverages orcharhino’s remote execution features to allow users to run arbitrary Salt functions.

Most commonly, users will want to run salt.highstate on one or more minions. This will apply all relevant Salt states. It can be done via the host overview page of individual minions by selecting Run Salt from the Schedule Remote Job drop down menu.

Older versions of the Salt plugin provided a Run Salt button that did not use orcharhino’s remote execution features. It is possible to fall back to this old behaviour by changing the salt_hide_run_salt_button setting in the Salt tab of the settings page.

Triggering a remote execution job will take you to a corresponding job overview page:

Running Salt using schedule remote job

If you select a host from the list in the screenshot above, you can see the output of the remote job in real time:

Salt state highstate output

Remote jobs are recorded on the jobs page and can be viewed there later.

Running the state.highstate function will also generate a Salt report as detailed above (remember that reports are only uploaded to orcharhino every 10 minutes).

As mentioned already you can use orcharhino’s remote execution features to run arbitrary Salt functions. This includes an optional test mode, that will tell you what would happen without actually changing anything. To use either feature, perform the following steps:

Navigate to Hosts > All Hosts > Your Host > Schedule Remote Job.
For the Job category field, select Salt from the drop down menu.
For the Job template field, select Salt Run function from the drop down menu.
Fill in the function you want to trigger (e.g. pillar.items or test.ping).
(optional) Click "Display advanced fields" and enable "test-run".
Run the job by clicking the Submit button.

You can also schedule remote jobs, or let them run recurrently. Refer to the remote execution guide for more information.

Alternatively, you can also achieve recurrent actions using the native Salt way. For example, you can schedule hourly state.highstate runs on individual minions by adding the following to /etc/salt/minion:

schedule:
  highstate:
    function: state.highstate
    minutes: 60

Refer to the SaltStack job management section for more information.

Final Example

This example uses a Salt state to manage the /etc/motd file on one or more Salt minions. In doing so we demonstrate the use of orcharhino as an external node classifier (ENC), as well as the use of Salt grains (which are analogous to Puppet facts).

First we will create a global parameter named vendor_name with the string orcharhino as its value.

Next, we will add a new Salt state named motd to our Salt master. To do so, we will create the /srv/salt/motd/ directory, and add two files to it.

The /srv/salt/motd/init.sls file, a Salt state file, will contain the following:

/etc/motd:
  file.managed:
    - user: root
    - group: root
    - mode: 0644
    - source: salt://motd/motd.template
    - template: jinja

The /srv/salt/motd/motd.template file, a template referenced from our Salt state file, will contain the following:

Welcome to {{ grains['fqdn'] }} Powered by {{ salt['pillar.get']('vendor_name') }}

As you can see, we are attempting to access the fqdn Salt grain from within this template. We are also attempting to retrieve the vendor_name parameter from the Salt pillar.

Having created the new motd Salt state, we need to import it into orcharhino.

Next, verify that Salt has been given access to the vendor_name parameter, by running either of the following commands on your Salt master:

salt '*' pillar.items | grep -A 1 vendor_name
salt '*' pillar.get vendor_name

If the output does not include the value of the vendor_name parameter ("orcharhino"), we may need to refresh the pillar data first. Refer to the Salt ENC parameters subsection on how to refresh the pillar data.

salt '*' saltutil.refresh_pillar

You can now add the motd Salt state to your minion host, or a relevant host group, and apply it by running state.highstate.

Finally, verify the contents of /etc/motd on any target minions, and our proof of concept example is complete!

Other Resources

The official SaltStack documentation is a good resource for all things Salt.
The official SaltStack package repository is the source for Salt packages.
Refer to the Salt logging page for more information.