Configuring and Setting Up Remote Jobs

Use this section as a guide to configuring orcharhino to execute jobs on remote hosts.

Any command that you want to apply to a remote host must be defined as a job template. After you have defined a job template you can execute it multiple times.

About Running Jobs on Hosts

You can run jobs on hosts remotely from orcharhino Proxies using shell scripts or Ansible tasks and playbooks. This is referred to as remote execution.

For custom Ansible roles that you create, or roles that you download, you must install the package containing the roles on the orcharhino Proxy base operating system. Before you can use Ansible roles, you must import the roles into orcharhino from the orcharhino Proxy where they are installed.

Communication occurs through orcharhino Proxy, which means that orcharhino Server does not require direct access to the target host, and can scale to manage many hosts. For more information, see transport modes for remote execution.

orcharhino uses ERB syntax job templates. For more information, see Template Writing Reference in Managing Hosts.

Several job templates for shell scripts and Ansible are included by default. For more information, see Setting up Job Templates in Managing Hosts.

Any orcharhino Proxy base operating system is a client of orcharhino Server’s internal orcharhino Proxy, and therefore this section applies to any type of host connected to orcharhino Server, including orcharhino Proxies.

You can run jobs on multiple hosts at once, and you can use variables in your commands for more granular control over the jobs you run. You can use host facts and parameters to populate the variable values.

In addition, you can specify custom values for templates when you run the command.

For more information, see Executing a Remote Job in Managing Hosts.

Remote Execution Workflow

When you run a remote job on hosts, for every host, orcharhino performs the following actions to find a remote execution orcharhino Proxy to use.

orcharhino searches only for orcharhino Proxies that have the remote execution feature enabled.

  1. orcharhino finds the host’s interfaces that have the Remote execution checkbox selected.

  2. orcharhino finds the subnets of these interfaces.

  3. orcharhino finds remote execution orcharhino Proxies assigned to these subnets.

  4. From this set of orcharhino Proxies, orcharhino selects the orcharhino Proxy that has the least number of running jobs. By doing this, orcharhino ensures that the jobs load is balanced between remote execution orcharhino Proxies.

If you have enabled Prefer registered through orcharhino Proxy for remote execution, orcharhino runs the REX job using the orcharhino Proxy the host is registered to.

By default, Prefer registered through orcharhino Proxy for remote execution is set to No. To enable it, in the orcharhino management UI, navigate to Administer > Settings, and on the Content tab, set Prefer registered through orcharhino Proxy for remote execution to Yes. This ensures that orcharhino performs REX jobs on hosts by the orcharhino Proxy to which they are registered to.

If orcharhino does not find a remote execution orcharhino Proxy at this stage, and if the Fallback to Any orcharhino Proxy setting is enabled, orcharhino adds another set of orcharhino Proxies to select the remote execution orcharhino Proxy from. orcharhino selects the most lightly loaded orcharhino Proxy from the following types of orcharhino Proxies that are assigned to the host:

  • DHCP, DNS and TFTP orcharhino Proxies assigned to the host’s subnets

  • DNS orcharhino Proxy assigned to the host’s domain

  • Realm orcharhino Proxy assigned to the host’s realm

  • Puppet server orcharhino Proxy

  • Puppet CA orcharhino Proxy

  • OpenSCAP orcharhino Proxy

If orcharhino does not find a remote execution orcharhino Proxy at this stage, and if the Enable Global orcharhino Proxy setting is enabled, orcharhino selects the most lightly loaded remote execution orcharhino Proxy from the set of all orcharhino Proxies in the host’s organization and location to execute a remote job.

Permissions for Remote Execution

You can control which roles can run which jobs within your infrastructure, including which hosts they can target. The remote execution feature provides two built-in roles:

  • Remote Execution Manager: Can access all remote execution features and functionality.

  • Remote Execution User: Can only run jobs.

You can clone the Remote Execution User role and customize its filter for increased granularity. If you adjust the filter with the view_job_templates permission on a customized role, you can only see and trigger jobs based on matching job templates. You can use the view_hosts and view_smart_proxies permissions to limit which hosts or orcharhino Proxies are visible to the role.

The execute_template_invocation permission is a special permission that is checked immediately before execution of a job begins. This permission defines which job template you can run on a particular host. This allows for even more granularity when specifying permissions.

You can run remote execution jobs against orcharhino and orcharhino Proxy registered as hosts to orcharhino with the execute_jobs_on_infrastructure_hosts permission. Standard Manager and Site Manager roles have this permission by default. If you use either the Manager or Site Manager role, or if you use a custom role with the execute_jobs_on_infrastructure_hosts permission, you can execute remote jobs against registered orcharhino and orcharhino Proxy hosts.

For more information on working with roles and permissions, see Creating and Managing Roles in Administering orcharhino.

The following example shows filters for the execute_template_invocation permission:

name = Reboot and host.name = staging.example.com
name = Reboot and host.name ~ *.staging.example.com
name = "Restart service" and host_group.name = webservers

Use the first line in this example to apply the Reboot template to one selected host. Use the second line to define a pool of hosts with names ending with .staging.example.com. Use the third line to bind the template with a host group.

Permissions assigned to users with these roles can change over time. If you have already scheduled some jobs to run in the future, and the permissions change, this can result in execution failure because permissions are checked immediately before job execution.

Creating a Job Template

Use this procedure to create a job template. To use the CLI instead of the orcharhino management UI, see the CLI procedure.

Procedure

  1. In the orcharhino management UI, navigate to Hosts > Job templates.

  2. Click New Job Template.

  3. Click the Template tab, and in the Name field, enter a unique name for your job template.

  4. Select Default to make the template available for all organizations and locations.

  5. Create the template directly in the template editor or upload it from a text file by clicking Import.

  6. Optional: In the Audit Comment field, add information about the change.

  7. Click the Job tab, and in the Job category field, enter your own category or select from the default categories listed in Default Job Template Categories in Managing Hosts.

  8. Optional: In the Description Format field, enter a description template. For example, Install package %{package_name}. You can also use %{template_name} and %{job_category} in your template.

  9. From the Provider Type list, select SSH for shell scripts and Ansible for Ansible tasks or playbooks.

  10. Optional: In the Timeout to kill field, enter a timeout value to terminate the job if it does not complete.

  11. Optional: Click Add Input to define an input parameter. Parameters are requested when executing the job and do not have to be defined in the template. For examples, see the Help tab.

  12. Optional: Click Foreign input set to include other templates in this job.

  13. Optional: In the Effective user area, configure a user if the command cannot use the default remote_execution_effective_user setting.

  14. Optional: If this template is a snippet to be included in other templates, click the Type tab and select Snippet.

  15. Click the Location tab and add the locations where you want to use the template.

  16. Click the Organizations tab and add the organizations where you want to use the template.

  17. Click Submit to save your changes.

You can extend and customize job templates by including other templates in the template syntax. For more information, see Template Writing Reference and Job Template Examples and Extensions in Managing Hosts.

CLI procedure

  • To create a job template using a template-definition file, enter the following command:

    # hammer job-template create \
--file "Path_to_My_Template_File" \
--job-category "My_Category_Name" \
--name "My_Template_Name" \
--provider-type SSH

Changing the Associated Template

Procedure

  1. Navigate to Administer > Remote Execution Features.

  2. Select the remote execution feature of which you want to change the associated template.

  3. In the Job Template drop down menu, select a different job template.

    Changing the default job templates might break existing orcharhino workflows. Evaluate carefully why you want to change the default job template.

    Cloned templates do not receive updates when upgrading orcharhino.

Configuring the Fallback to Any orcharhino Proxy Remote Execution Setting in orcharhino

You can enable the Fallback to Any orcharhino Proxy setting to configure orcharhino to search for remote execution orcharhino Proxies from the list of orcharhino Proxies that are assigned to hosts. This can be useful if you need to run remote jobs on hosts that have no subnets configured or if the hosts' subnets are assigned to orcharhino Proxies that do not have the remote execution feature enabled.

If the Fallback to Any orcharhino Proxy setting is enabled, orcharhino adds another set of orcharhino Proxies to select the remote execution orcharhino Proxy from. orcharhino also selects the most lightly loaded orcharhino Proxy from the set of all orcharhino Proxies assigned to the host, such as the following:

  • DHCP, DNS and TFTP orcharhino Proxies assigned to the host’s subnets

  • DNS orcharhino Proxy assigned to the host’s domain

  • Realm orcharhino Proxy assigned to the host’s realm

  • Puppet server orcharhino Proxy

  • Puppet CA orcharhino Proxy

  • OpenSCAP orcharhino Proxy

Procedure

  1. In the orcharhino management UI, navigate to Administer > Settings.

  2. Click Remote Execution.

  3. Configure the Fallback to Any orcharhino Proxy setting.

CLI procedure

  • Enter the hammer settings set command on orcharhino to configure the Fallback to Any orcharhino Proxy setting. To set the value to true, enter the following command:

    $ hammer settings set \
--name=remote_execution_fallback_proxy \
--value=true

Configuring the Global orcharhino Proxy Remote Execution Setting in orcharhino

By default, orcharhino searches for remote execution orcharhino Proxies in hosts' organizations and locations regardless of whether orcharhino Proxies are assigned to hosts' subnets or not. You can disable the Enable Global orcharhino Proxy setting if you want to limit the search to the orcharhino Proxies that are assigned to hosts' subnets.

If the Enable Global orcharhino Proxy setting is enabled, orcharhino adds another set of orcharhino Proxies to select the remote execution orcharhino Proxy from. orcharhino also selects the most lightly loaded remote execution orcharhino Proxy from the set of all orcharhino Proxies in the host’s organization and location to execute a remote job.

Procedure

  1. In the orcharhino management UI, navigate to Administer > Settings.

  2. Click Remote Execution.

  3. Configure the Enable Global orcharhino Proxy setting.

CLI procedure

  • Enter the hammer settings set command on orcharhino to configure the Enable Global orcharhino Proxy setting. To set the value to true, enter the following command:

    $ hammer settings set \
--name=remote_execution_global_proxy \
--value=true

Configuring orcharhino to Use an Alternative Directory to Execute Remote Jobs on Hosts

By default, orcharhino uses the /var/tmp directory on the client system to execute the remote execution jobs. If the client system has noexec set for the /var/ volume or file system, you must configure orcharhino to use an alternative directory because otherwise the remote execution job fails since the script cannot be run.

Procedure

  1. Create a new directory:

    $ mkdir /My_Remote_Working_Directory

  2. Copy the SELinux context from the default var directory:

    $ chcon --reference=/var /My_Remote_Working_Directory

  3. Configure the system:

    $ orcharhino-installer \
--foreman-proxy-plugin-remote-execution-script-remote-working-dir /My_Remote_Working_Directory

Distributing SSH Keys for Remote Execution

For orcharhino Proxies in ssh mode, remote execution connections are authenticated using SSH. The public SSH key from orcharhino Proxy must be distributed to its attached hosts that you want to manage.

Ensure that the SSH service is enabled and running on the hosts. Configure any network or host-based firewalls to enable access to port 22.

Use one of the following methods to distribute the public SSH key from orcharhino Proxy to target hosts:

  1. distributing ssh keys for remote execution manually.

  2. Using the orcharhino API to Obtain SSH Keys for Remote Execution.

  3. Configuring a Kickstart Template to Distribute SSH Keys During Provisioning.

  4. For new orcharhino hosts, you can deploy SSH keys to orcharhino hosts during registration using the global registration template. For more information, see Registering a Host to orcharhino Using the Global Registration Template in Managing Hosts.

orcharhino distributes SSH keys for the remote execution feature to the hosts provisioned from orcharhino by default.

If the hosts are running on Amazon Web Services, enable password authentication. For more information, see New User Accounts.

Distributing SSH Keys for Remote Execution Manually

To distribute SSH keys manually, complete the following steps:

Procedure

  • Copy the SSH pub key from your orcharhino Proxy to your target host:

    $ ssh-copy-id -i ~foreman-proxy/.ssh/id_rsa_foreman_proxy.pub root@client.example.com

    Repeat this step for each target host you want to manage.

Verification

  • To confirm that the key was successfully copied to the target host, enter the following command on orcharhino Proxy:

    $ ssh -i ~foreman-proxy/.ssh/id_rsa_foreman_proxy root@client.example.com

Using the orcharhino API to Obtain SSH Keys for Remote Execution

To use the orcharhino API to download the public key from orcharhino Proxy, complete this procedure on each target host.

Procedure

  1. On the target host, create the ~/.ssh directory to store the SSH key:

    $ mkdir ~/.ssh

  2. Download the SSH key from orcharhino Proxy:

    $ curl https://orcharhino-proxy.network2.example.com:443/ssh/pubkey >> ~/.ssh/authorized_keys

  3. Configure permissions for the ~/.ssh directory:

    $ chmod 700 ~/.ssh

  4. Configure permissions for the authorized_keys file:

    $ chmod 600 ~/.ssh/authorized_keys

Configuring a Kickstart Template to Distribute SSH Keys During Provisioning

You can add a remote_execution_ssh_keys snippet to your custom Kickstart template to deploy SSH Keys to hosts during provisioning. Kickstart templates that orcharhino ships include this snippet by default. orcharhino copies the SSH key for remote execution to the systems during provisioning.

Procedure

  • To include the public key in newly-provisioned hosts, add the following snippet to the Kickstart template that you use:

    <%= snippet 'remote_execution_ssh_keys' %>

Configuring a keytab for Kerberos Ticket Granting Tickets

Use this procedure to configure orcharhino to use a keytab to obtain Kerberos ticket granting tickets. If you do not set up a keytab, you must manually retrieve tickets.

Procedure

  1. Find the ID of the foreman-proxy user:

    $ id -u foreman-proxy

  2. Modify the umask value so that new files have the permissions 600:

    $ umask 077

  3. Create the directory for the keytab:

    $ mkdir -p "/var/kerberos/krb5/user/My_User_ID"

  4. Create a keytab or copy an existing keytab to the directory:

    $ cp My_Client.keytab /var/kerberos/krb5/user/My_User_ID/client.keytab

  5. Change the directory owner to the foreman-proxy user:

    $ chown -R foreman-proxy:foreman-proxy "/var/kerberos/krb5/user/My_User_ID"

  6. Ensure that the keytab file is read-only:

    $ chmod -wx "/var/kerberos/krb5/user/My_User_ID/client.keytab"

  7. Restore the SELinux context:

    $ restorecon -RvF /var/kerberos/krb5

Configuring Kerberos Authentication for Remote Execution

You can use Kerberos authentication to establish an SSH connection for remote execution on orcharhino hosts.

Prerequisites

  • Enroll orcharhino Server on the Kerberos server

  • Enroll the orcharhino target host on the Kerberos server

  • Configure and initialize a Kerberos user account for remote execution

  • Ensure that the foreman-proxy user on orcharhino has a valid Kerberos ticket granting ticket

Procedure

  1. To install and enable Kerberos authentication for remote execution, enter the following command:

    $ orcharhino-installer --scenario katello \
--foreman-proxy-plugin-remote-execution-script-ssh-kerberos-auth true

  2. To edit the default user for remote execution, in the orcharhino management UI, navigate to Administer > Settings and click the Remote Execution tab. In the SSH User row, edit the second column and add the user name for the Kerberos account.

  3. Navigate to remote_execution_effective_user and edit the second column to add the user name for the Kerberos account.

Verification

  • To confirm that Kerberos authentication is ready to use, run a remote job on the host. For more information, see Executing a Remote Job in Managing Hosts.

Setting up Job Templates

orcharhino provides default job templates that you can use for executing jobs. To view the list of job templates, navigate to Hosts > Job templates. If you want to use a template without making changes, proceed to Executing a Remote Job in Managing Hosts.

You can use default templates as a base for developing your own. Default job templates are locked for editing. Clone the template and edit the clone.

Procedure

  1. To clone a template, in the Actions column, select Clone.

  2. Enter a unique name for the clone and click Submit to save the changes.

Job templates use the Embedded Ruby (ERB) syntax. For more information about writing templates, see the Template Writing Reference in Managing Hosts.

Ansible Considerations

To create an Ansible job template, use the following procedure and instead of ERB syntax, use YAML syntax. Begin the template with ---. You can embed an Ansible playbook YAML file into the job template body. You can also add ERB syntax to customize your YAML Ansible template. You can also import Ansible playbooks in orcharhino. For more information, see Synchronizing Repository Templates in Managing Hosts.

Parameter Variables

At run time, job templates can accept parameter variables that you define for a host. Note that only the parameters visible on the Parameters tab at the host’s edit page can be used as input parameters for job templates. If you do not want your Ansible job template to accept parameter variables at run time, in the orcharhino management UI, navigate to Administer > Settings and click the Ansible tab. In the Top level Ansible variables row, change the Value parameter to No.

Executing a Remote Job

You can execute a job that is based on a job template against one or more hosts.

To use the CLI instead of the orcharhino management UI, see the CLI procedure.

Procedure

  1. In the orcharhino management UI, navigate to Hosts > All Hosts and select the target hosts on which you want to execute a remote job. You can use the search field to filter the host list.

  2. From the Select Action list, select Schedule a Job.

  3. On the Job invocation page, define the main job settings:

  4. Select the Job category and the Job template you want to use.

  5. Optional: Select a stored search string in the Bookmark list to specify the target hosts.

  6. Optional: Further limit the targeted hosts by entering a Search query. The Resolves to line displays the number of hosts affected by your query. Use the refresh button to recalculate the number after changing the query. The preview icon lists the targeted hosts.

  7. The remaining settings depend on the selected job template. See Creating a Job Template for information on adding custom parameters to a template.

  8. Optional: To configure advanced settings for the job, click Display advanced fields. Some of the advanced settings depend on the job template, the following settings are general:

    • Effective user defines the user for executing the job, by default it is the SSH user.

    • Concurrency level defines the maximum number of jobs executed at once, which can prevent overload of systems' resources in a case of executing the job on a large number of hosts.

    • Timeout to kill defines time interval in seconds after which the job should be killed, if it is not finished already. A task which could not be started during the defined interval, for example, if the previous task took too long to finish, is canceled.

    • Type of query defines when the search query is evaluated. This helps to keep the query up to date for scheduled tasks.

    • Execution ordering determines the order in which the job is executed on hosts: alphabetical or randomized.

      Concurrency level and Timeout to kill settings enable you to tailor job execution to fit your infrastructure hardware and needs.

  9. To run the job immediately, ensure that Schedule is set to Execute now. You can also define a one-time future job, or set up a recurring job. For recurring tasks, you can define start and end dates, number and frequency of runs. You can also use cron syntax to define repetition.

  10. Click Submit. You can view status of the jobs in the Recent Jobs section on the same page.

CLI procedure

  1. Enter the following command on orcharhino:

    $ hammer settings set \
--name=remote_execution_global_proxy \
--value=false

  2. Find the ID of the job template you want to use:

    $ hammer job-template list

  3. Show the template details to see parameters required by your template:

    $ hammer job-template info --id My_Template_ID

  4. Execute a remote job with custom parameters:

    # hammer job-invocation create \
--inputs My_Key_1="My_Value_1",My_Key_2="My_Value_2",... \
--job-template "My_Template_Name" \
--search-query "My_Search_Query"

    Replace My_Search_Query with the filter expression that defines hosts, for example "name ~ My_Pattern". For more information about executing remote commands with hammer, enter hammer job-template --help and hammer job-invocation --help.

Scheduling a Recurring Ansible Job for a Host

You can schedule a recurring job to run Ansible roles on hosts.

Prerequisite

  • Ensure you have the view_foreman_tasks, view_job_invocations, and view_recurring_logics permissions.

Procedure

  1. In the orcharhino management UI, navigate to Hosts > All Hosts and select the target host on which you want to execute a remote job.

  2. On the Ansible tab, select Jobs.

  3. Click Schedule recurring job.

  4. Define the repetition frequency, start time, and date of the first run in the Create New Recurring Ansible Run window.

  5. Click Submit.

  6. Optional: View the scheduled Ansible job in host overview or by navigating to Ansible > Jobs.

Scheduling a Recurring Ansible Job for a Host Group

You can schedule a recurring job to run Ansible roles on host groups.

Procedure

  1. In the orcharhino management UI, navigate to Configure > Host groups.

  2. In the Actions column, select Configure Ansible Job for the host group you want to schedule an Ansible roles run for.

  3. Click Schedule recurring job.

  4. Define the repetition frequency, start time, and date of the first run in the Create New Recurring Ansible Run window.

  5. Click Submit.

Monitoring Jobs

You can monitor the progress of a job while it is running. This can help in any troubleshooting that may be required.

Ansible jobs run on batches of 100 hosts, so you cannot cancel a job running on a specific host. A job completes only after the Ansible playbook runs on all hosts in the batch.

Procedure

  1. In the orcharhino management UI, navigate to Monitor > Jobs. This page is automatically displayed if you triggered the job with the Execute now setting. To monitor scheduled jobs, navigate to Monitor > Jobs and select the job run you wish to inspect.

  2. On the Job page, click the Hosts tab. This displays the list of hosts on which the job is running.

  3. In the Host column, click the name of the host that you want to inspect. This displays the Detail of Commands page where you can monitor the job execution in real time.

  4. Click Back to Job at any time to return to the Job Details page.

CLI procedure

  1. Find the ID of a job:

    # hammer job-invocation list

  2. Monitor the job output:

    # hammer job-invocation output \
--host "My_Host_Name" \
--id My_Job_ID

  3. Optional: To cancel a job, enter the following command:

    # hammer job-invocation cancel \
--id My_Job_ID

The text and illustrations on this page are licensed by ATIX AG under a Creative Commons Attribution–Share Alike 3.0 Unported ("CC-BY-SA") license. This page also contains text from the official Foreman documentation which uses the same license ("CC-BY-SA").