Backup Mechanisms

When and how to backup orcharhino is a question highly dependent on your individual usage and setup. We generally recommend performing backups before and after orcharhino upgrades.

The default use case is for orcharhino installations to run on virtual machines, most commonly within a VMware virtualization environment. We recommend using the snapshot functionality provided by the virtualization solution for backups in such a setup. We recommend making a file based backup with your enterprise backup solution if you run orcharhino on bare metal hardware.

Note that performing backups using Foreman maintain is still considered a technical preview.

Running orcharhino on a Virtual Machine

The ATIX recommended backup strategy is to make backups of system snapshots as provided by the virtualization solution orcharhino is running on (e.g. using VMware’s vSphere client).

The orcharhino VM is ideally powered down before taking a snapshot. Only if powering down your orcharhino is operationally not possible, you may take a live snapshot that also retains the VM’s memory.

Either way, it is worth checking that there are no long running tasks before taking the snapshot or stopping any services:

foreman-rake katello:upgrade_check

This will ideally result in output similar as follows:

Checking for running tasks...
[SUCCESS] - There are 0 active tasks.
         You may proceed with the upgrade.

Newer snapshots will include lots of references to older snapshots. This can be inefficient and negatively affect system performance if too many snapshots are kept at the same time. It is recommended to limit the number of snapshots to no more than three. Consider deleting older snapshots after creating new ones.


Alternatively, you can also create a backup based on the snapshot functionality of LVM, which would include the following steps:

  1. Your enterprise backup solution triggers all pre backup tasks, which include -depending on your backup strategy- stopping all orcharhino services or powering down the orcharhino instance altogether.

  2. Create an LVM snapshot via your backup application. In case the VM is still running, create a snapshot of its memory, too.

  3. Restart all orcharhino services or restart your orcharhino instance.

  4. Create a block-based backup of the snapshot according to your general backup routine.

  5. Have the backup application delete the LVM snapshot.

Running orcharhino on Bare Metal

Backing up orcharhino running on bare metal strongly depends to how you backup bare metal machines running CentOS, Oracle Linux, and Red Hat Enterprise Linux in general. This part describes a general backup strategy using a file-based approach.

Depending on the size of the data to be backup up, orcharhino will be offline for a correspondingly long time. The required offline period depends largely on the performance of your hardware and backup infrastructure as well as the number and size of synchronized repositories. Note that there are also other valid ways to backup orcharhino, but remember to always backup your orcharhino instance in a consistent manner, which is usually achieved by stopping all orcharhino services.

Using this approach results in the following four steps:

  1. Stop your orcharhino instance with a pre backup hook by your enterprise backup solution:

    foreman-maintain service stop

    Check if all orcharhino services have been stopped successfully:

    foreman-maintain service status
  2. Deactivate all cron jobs or similar which could potentially trigger orcharhino tasks with a pre backup hook by your enterprise backup solution. This is necessary as some tasks may alter databases on your orcharhino even though the orcharhino service itself isn’t running.

  3. Execute a file based backup with your enterprise backup solution.

    You must include all relevant paths of the operating system and any installed applications. These might vary from the standard paths depending on your environment.

    You must also include all relevant paths of the orcharhino. This can also vary depending on the setup, installed plugins, and running services.

    The following list will assume a basic configuration of your orcharhino.

    The .hammer directory has to be backed up as it contains configuration files for the Hammer CLI:

    /root/.hammer/

    The following directories including all subdirectories must be backed up in order to maintain consistency in case of expanding the orcharhino feature set in the future:

    /etc/
    /opt/
    /var/

    In case you don’t want to backup the above mentioned directory structure, at least the following directories and subdirectories must be backed up for a fundamentally functioning orcharhino instance regardless of the operating system:

    /opt/orcharhino/
    /opt/theforeman/
    /opt/puppetlabs/
    /usr/bin/
    /usr/lib/
    /usr/sbin/
    /usr/share/

    We always recommend a complete backup of all file systems of the orcharhino system without exceptions.

  4. Restart all orcharhino services with a post backup hook by your enterprise backup solution.

    foreman-maintain service start

    Verify if all orcharhino services have been started successfully:

    foreman-maintain service status

Restoring your orcharhino backup takes place according to your general approach regarding restoring bare metal CentOS, Oracle Linux, and Red Hat Enterprise Linux systems.

Using Foreman Maintain

foreman-maintain provides various maintenance features including backups for the upstream Foreman open source project. It is always included with orcharhino but its use for orcharhino backups is considered to have technical preview status.

The Foreman maintain utility for orcharhino backups has technical preview status and can be used at your own risk. If you are interested in doing so in a production environment, please contact ATIX Support.

Backup

Make sure that no other administrator plans to run any long running job like synchronizing a repository or creating a content view before starting the backup process. You can use foreman-maintain health check to run various health checks on your orcharhino.

Both commands, backup and restore, require explicit confirmation by the caller. However, this can be bypassed for scripts by the option --assumeyes.

There are currently two ways to perform a backup: online and offline backups. While the system is available during an online backup, all orcharhino services are turned off during the offline backup. We recommend using offline backups for production use since this is the only way to ensure data integrity of the backup.

To perform a backup, you will need to provide a path where the backup files should be stored. Make sure there is enough free disk space available on this backup path. We are using /var/backup/ as backup path in the following examples.

A full backup of all orcharhino databases may take a long time. Your orcharhino will be unavailable for the entire duration of the backup as all orcharhino services are stopped for offline backups.


The following command creates an offline backup: it stops all orcharhino services first, creates a full backup of all necessary data including Pulp content and all databases, and then starts the services again.

foreman-maintain backup offline /var/backup/

This creates a directory like /var/backup/katello-backup-2018-09-12-08-21-11 in which the backup is stored. Use the --preserve-directory (-p) option in case you don’t want to create a backup specific subdirectory.

The backup contains all of the Pulp content like RPM and Debian packages, Puppet modules and more. You can omit this step by using the --skip-pulp-content (-s) option.

Restoring a fully working orcharhino from a backup without Pulp content is not possible.


The second way to create a backup is the online mode. Only use an online backup for debugging purposes as the data integrity between the various databases cannot be fully guaranteed. You must not make any modification regarding host provisioning, repository synchronization, content view changes, etc. during an online backup as it leads to inconsistencies in the online backup.

foreman-maintain backup online /var/backup/

It is also possible to make an incremental backup. You need to specify the path to an existing full backup or to another incremental backup. You may create an incremental backup based on the previously created full backup as follows:

foreman-maintain backup offline --incremental /var/backup/katello-backup-2018-09-12-08-21-11 /var/backup/

This creates an incremental backup based on the backup /var/backup/katello-backup-2018-09-12-08-21-11 and stores it in the /var/backup/ directory like /var/backup/katello-backup-2018-09-13-08-23-38.

An incremental backup can be based either on a full backup or another incremental backup. Keep in mind that you will need this backup as well as the last full backup and all incremental backups in between those two to successfully restore your system.

Restore

Make sure to use the right orcharhino instance and that the orcharhino software version has not changed before starting the restore procedure. The host name within the backup must match the host name of the system you are restoring. In the following example, we will restore the orcharhino based on the previously taken full backup:

foreman-maintain restore /var/backup/katello-backup-2018-09-12-08-21-11

This results in output similar to the following:

Console output while restoring a full backup

The system is in the state it was in at the time of the backup and all orcharhino services are available again once the restore procedure is finished.

For restoring orcharhino from the incremental backup, you need to restore the full backup and the desired incremental backups in sequence:

foreman-maintain restore /var/backup/katello-backup-2018-09-12-08-21-11
foreman-maintain restore -i /var/backup/katello-backup-2018-09-13-08-23-38