Content Management Guide

Content management is a central feature of orcharhino. Within the context of orcharhino the term content management primarily refers to the provisioning of software beyond the basic operating system on some given host managed through orcharhino. For example, a host that is to be used as a web server would typically require access to any packages needed by the Apache server (or something equivalent).

orcharhino’s content management offers a highly flexible framework, that can be adapted to a wide variety of use cases. Many operating systems and content types are supported, as well as version control and lifecycle management.

As a result, orcharhino’s content management features several interrelated parts and a relatively high degree of conceptual complexity. This guide exists to break down that complexity, so as to provide an accessible introduction.

The content management guide contains the following subsections:

Refer to the content menu for more information on using orcharhino’s content management.

Key Terminology

To make any progress in understanding orcharhino’s content management we must first introduce the key terminology involved. This subsection of the guide provides background information, and can also serve as a quick reference.

  • Repository: The smallest storage unit for software content in orcharhino. An orcharhino software repository generally needs to be synchronized with some content source outside of orcharhino to obtain content. To be usable in orcharhino, a remote repository must be of a type supported by orcharhino. These types are deb, yum, Puppet, Docker, and file.

  • Product: A named collection of one or more repositories. A single product may contain repositories of different types. Generally, repositories can only ever be added to orcharhino as part of some product.

  • Subscription: The usage of Red Hat and SUSE products is limited by the amount of available subscriptions.

    Advanced subscriptions management is generally only needed for Red Hat and SUSE products. Other products generally have an unlimited number of subscriptions available and each subscribed content host will automatically be given one.

  • Red Hat Subscription: For Red Hat customers, orcharhino provides additional features for the management of Red Hat content.

  • SUSE Subscription: For SUSE customers, orcharhino provides additional features for the management of SUSE content.

  • Content View: A named and versioned collection of repositories. Whenever a new content view version is created (published), the current software content state of the repositories within it is frozen. Any subsequent changes to the underlying repositories will no longer affect the published content view version.

  • Composite Content View: A content view created by combining existing content views. Unless noted otherwise, we will use the term content view to refer to both content views and composite content views in this guide.

  • Lifecycle Environment: A lifecycle environment exists to restrict host access to content based on the hosts role or affiliation. For example, the classic scenario for lifecycle management is to distinguish between hosts for development, testing, and production. (In this scenario hosts would be assigned to either the development, testing, or production lifecycle environment). It would then be possible to run different versions of the same software in the different environments. This way, new versions of a software can be developed, and then tested before being used in the production environment, greatly reducing the risk of disruption by prematurely rolled out updates.

  • Lifecycle Environment Path: Lifecycle environments are organized into directional paths. Content view versions are then "promoted" through the path. Sticking with the previous example, a new content view version would be used in the development environment fist, then be promoted to testing, and only enter production once it has been sufficiently tested. It is possible to create multiple lifecycle environment paths, containing a variable number of lifecycle environments. All paths originate from the "library" environment, which is always present by default.

  • Activation Key: Part of the mechanism used for hosts to register with orcharhino’s content management. An activation key is associated to exactly one lifecycle environment and exactly one content view (though this may be a composite content view). An activation key must also be associated to one or more subscriptions, which generally correspond to products. When a host registers using a given key content is made available to the host. The content that is made available is the set theoretic intersection, between the content present in the key’s content view/lifecycle environment combination, and the content present in the key’s subscriptions. (See the below image for a visual representation).

    Specifying the content host with products
  • Subscription Manager: The client-side software is responsible for registering hosts with orcharhino’s content management using an activation key, as well as managing the hosts repository access once registered.

  • Katello Host Tools and Katello Host Tools Tracer: The client-side software is responsible for the connection between managed hosts and content from orcharhino or orcharhino Proxies. It uploads currently used repositories and installed packages to orcharhino and signals if the managed host needs to be rebooted after updating the Linux Kernel. Note that katello-host-tools-tracer is not available for hosts running SLES 12 and below.

  • Katello Agent: The client-side software allows orcharhino users to manage registered hosts (i.e. content hosts) within orcharhino’s management UI.

    The Katello agent supports package actions for Red Hat and SUSE based systems. The remote execution plugin additionally supports package actions for Debian and Ubuntu. As a result, use of the Katello agent for package actions may be deprecated in favour of the remote execution plugin in the not too distant future.

  • Content Host: A host which is managed through orcharhino’s content management. Each content host is always associated to exactly one content view and exactly one lifecycle environment.

Features

This subsection of the guide answers the question what one can do using orcharhino’s content management. The following list can also be read as a feature set:

  • Gaining access to SUSE and Red Hat Enterprise Linux content. (Only relevant to SUSE and Red Hat customers).

  • Adding content sources (repositories) to orcharhino (organized into products) and synchronizing orcharhino with those (remote) repositories.

  • Creating orcharhino content views (and/or composite content views) to create useful collections of repositories. (For example: All the repositories needed by a particular type of database server.)

  • Publishing content views for version control. This action will freeze the current state of all repositories in the content view, to create a stable software source (referred to as a version of the content view).

  • Creating lifecycle paths and lifecycle environments to restrict content access to a particular content view version based on role.

  • Promoting particular versions of content views through a lifecycle path (into the next lifecycle environment) to upgrade content in a careful and controlled way.

  • Creating activation keys to register hosts (or entire host groups) to a particular content view and lifecycle environment (further restricted using subscriptions).

  • Managing registered content hosts through the content hosts page.

Refer to the setup subsection for more information on how to do the above things.

Supported Content Types

orcharhino supports several different content types, and each of these types is given it’s own page in the content menu section of the management UI chapter:

These content types are related to the various repository types supported by orcharhino. However, a repository of a given type, might contain content of several different types. For example, repositories of type yum may contain content of type errata, as well as content of type RPM package. Repository types include deb, yum, Puppet, Docker, and file.

Setup

When confronted by the various menu items in the content menu it is not immediately clear where to begin. However, the relations between the various parts of orcharhino’s content management do imply a logical order. For example, it makes little sense to create content views before importing the relevant content (in the form of products).

This subsection of the guide introduces a natural order in which to approach orcharhino’s content management, providing a step-by-step setup guide. It is of particular interest to first time users and orcharhino administrators.

Before you actually start creating the various content management entities, you may also want to check the strategy and best practices subsection.

To avoid duplicate documentation, we will make frequent references to the relevant subsections from the content menu section of the management UI chapter. Follow these references for detailed documentation on how to perform each step described.

Adding RHEL Content

Procedure
  1. Import a Red Hat manifest file that can be obtained from Red Hat to orcharhino. You can view any associated Red Hat content on the Red Hat repositories page.

  2. Select Red Hat content for inclusion in orcharhino. You can view selected content on the products page.

  3. To synchronize Red Hat products, continue with products, repositories, and synchronization below.

Running Red Hat Enterprise Linux on managed hosts requires a subscription from Red Hat. Ensure to provide valid licenses for all used Red Hat products. Using insufficient, invalid, or otherwise inadequate licenses might violate your terms with Red Hat.

We recommend using a Manifest file from your Red Hat account to import available content to orcharhino. Navigate to Content > Subscriptions to see available and consumed Red Hat subscriptions.

Adding SLES Content

Procedure
  1. Add your SUSE Customer Center (short: SCC) accounts and synchronize the list of available SUSE products to orcharhino.

  2. Select SUSE content to be imported into orcharhino. You can view imported SUSE content on the products page.

  3. To synchronize SUSE products, continue with products, repositories, and synchronization below.

Refer to the managing SLES systems guide for more information on how to manage SUSE content using the SCC Manager plugin.

Running SUSE Linux Enterprise Server on managed hosts requires a subscription from SUSE. Ensure to provide valid licenses for all used SUSE products. Using insufficient, invalid, or otherwise inadequate licenses might violate your terms with SUSE.

We recommend using the SCC Manager plugin to import available products from SUSE to orcharhino. Navigate to Content > SUSE Subscriptions to see available and consumed SUSE subscriptions.

Products, Repositories, and Synchronization

This subsubsection includes step-by-step instructions on how to add arbitrary content to the orcharhino. If you are a SUSE or Red Hat customer and followed the steps for adding RHEL and SLES content above, you already have some products within orcharhino. In addition, you can add your own products from private or public repositories, for example using a link to a public mirror:

  1. Optional: Add GPG keys and/or SSL certificates to orcharhino.

  2. Optional: Create sync plans to periodically synchronize products.

  3. Create and synchronize products with repositories.

    1. Create a product.

    2. Add repositories to your product.

    3. Manually synchronize your new product, or wait for your sync plan to take effect.

  4. Optional: Check on the synchronization status of your products.

Depending on your download policy set for individual repositories, synchronization will create a local copy on the orcharhino of all repositories contained in each product being synchronized. Depending on the size of the repositories involved, and the speed of the internet connection to them, this can take a significant amount of time. Many hours are not uncommon and should be planned for accordingly.

Once your products are synchronized, continue with content views and lifecycle environments/paths.

Content Views and Lifecycle Environments/Paths

This subsubsection includes step-by-step instructions on how to manage content on the orcharhino once it has been successfully synchronized. These steps are necessary to provide version control and lifecycle management for this content.

  1. (Optional, best practice) Create any lifecycle environment paths as desired.

    1. Create new environment paths by appending a single new lifecycle environment to library.

    2. Complete your new environment paths by appending additional new lifecycle environments to the ends of existing paths.

  2. Create any desired content views.

    1. Create a content view.

    2. Add repositories to your content view.

    3. Publish a new version of your new content view.

    4. (In the future) Publish additional major versions of the content view once the underlying repositories have changed (2.0, 3.0, etc.) or

    5. Optional: Apply errata to a major version of the content view to create minor versions (like 1.1, 1.2, 2.1, etc.)

    6. Promote relevant versions of your content view through your lifecycle environment paths. (Different versions can be assigned to different environments in a path).

  3. Optional: Create any desired composite content views.

    1. Create a composite content view.

    2. Add any desired (existing) content views to your composite content view.

    3. Select which version of the added content views to use.

    4. Publish a new version of your new composite content view (version 1.0)

    5. (In the future) Publish additional major versions (2.0, 3.0, etc.) of the composite content view once additional content views have been added or if you want to use new versions of the underlying content views.

    6. Promote relevant versions of your content view through your lifecycle environment paths. (Different versions can be assigned to different environments in a path).

Once content views and lifecycle environment paths have been fully created, continue with the following subsection.

Activation Keys and Content Hosts

This subsubsection includes step-by-step instructions on how to register content hosts with orcharhino (using activation keys). Provisioning content hosts with content managed through orcharhino is the ultimate goal of orcharhino’s content management.

  1. Create any desired activation keys.

    1. Create the key using exactly one lifecycle environment and exactly one content view (or composite content view)

    2. Add any desired subscriptions to the key.

    3. (Optional, discouraged) override repository sets as desired.

    4. Optional: Add any desired host collections.

  2. Optional: Associate your host groups with new activation keys as desired.

  3. Create any desired content hosts. Content hosts are created by registering an existing (plain) host with orcharhino’s content management (using an activation key), or by creating new hosts belonging to a host group already associated to an activation key.

Once content hosts have been successfully created, they can be managed through orcharhino.

Usage

This subsection of the content management guide includes instructions on how to make effective use of orcharhino’s content management after it has been set up. As such, this subsection presumes the presence of fully entitled content hosts with access to fully configured content views, lifecycle environments, and subscriptions. Refer to the setup subsection if you do not yet have these things set up.

Strategy and Best Practices

These best practice guidelines describe how to use orcharhino’s content management. Note that these examples may not be suitable for all use cases and therefore should be viewed as friendly suggestions.

  • We recommend keeping products as small as possible. For example, one product each for PostgreSQL-7.0, PostgreSQL-7.1, and PostgreSQL-8.1 with one repository each is better than a single PostgreSQL product including multiple repositories for different versions. Likewise, we recommend separate products for things like CentOS-7.1, Apache 2.0, MySQL-5.1, and PostgreSQL 7.1 rather than a single WebServer product. Since products correspond to subscriptions, small products will give you greater control over what subscriptions (and by extension what content) is used (via your activation keys).

  • There are two basic approaches to content views: Either create content views that contain everything needed for a certain type of host, e.g. a WebServer content view, which includes repositories for CentOS-7.1, Apache-2.0, and PostgreSQL-7.1. Alternatively, create a content view each for CentOS-7.1, Apache-2.0, and PostgreSQL-7.1 along with a composite content view WebServer, which includes all of these content views.

  • The same two basic approaches exist for activation keys: Either create a single activation key for a specific kind of use case (e.g. corporate_webserver_prod), which uses the WebServer content view and selects all subscriptions necessary to run the service. Alternatively, create separate activation keys like basic_centos_7_prod, webserver_prod, and database_mysql_prod, all using the same content view (i.e. WebServer) along with a different set of subscriptions.

Errata Management

Email notifications may help you keeping track of available errata for specific hosts. Navigate to the users page to enable email notifications.

Information relevant to errata management is available in various locations in orcharhino’s management UI. This subsubsection exists to collect the entire workflow relevant for errata management in a single place. As such, it is meant to complement the relevant documentation elsewhere in this document, and will make frequent references.

When dealing with errata, care must be taken to distinguish between applicable and installable errata. An erratum is applicable to a content host if that host has packages affected by an issue that the erratum fixes. An erratum is installable if it is both applicable as well as present in the relevant content view version, such that the relevant content host has access to it.

The expected workflow for obtaining and applying errata goes as follows:

  1. New errata are obtained when some relevant repository/product that provides errata is synchronized. Assuming a sync plan is in use, this process requires no user input.

  2. Any content hosts with any applicable errata (see the info box above) will now display a relevant warning on both the all hosts page as well as their individual host overview page. Note that the content hosts page as well as any content host overview pages list only installable but not applicable errata.

  3. Once alerted to the presence of applicable errata, the orcharhino user can review new errata and select them for application on the errata page.

    To help choose which errata to select for application, each erratum has a detailed description as well as a list of affected packages on the relevant erratum overview page.

  4. Detailed documentation on applying any chosen errata can be found in the applying errata section of the errata page. This process generally involves publishing a new minor content view version to ensure any merely applicable errata also become installable.

  5. Once the desired errata are installable, they can be installed via remote execution, or directly on the host in question. There is a checkbox to do so as part of the previous step.

The ATIX Debian and Ubuntu Errata service provides errata for Debian and Ubuntu. When creating a repository of type deb, point the Errata URL at ATIX’s Debian and Ubuntu Errata Parser server. Use https://dep.atix.de/dep/api/v1/debian for Debian and https://dep.atix.de/dep/api/v1/ubuntu for Ubuntu.

An erratum contains the information which packages have to be updated to fix a security issue. Debian and Ubuntu errata are derived from the Debian security announcements (DSA) and the Ubuntu security notices (USN).

Ensure to only add Debian and Ubuntu errata to Debian and Ubuntu repositories that contain the packages needed to apply the errata. For Debian, you need the <debian_release>/updates repository, for example bullseye/updates. For Ubuntu, you need the <ubuntu_release>-security repository, for example focal-security.

Best Practices to Manage Content with orcharhino

To make the most out of orcharhino and its advanced content and patch management capabilities, we recommend certain best practices on how to manage content.

Managing Products and Repositories

  • Separate products by major release, for example Oracle Linux 7 and Oracle Linux 8 or Ubuntu 18.04 and Ubuntu 20.04.

  • Use one content type per product and content view, for example deb or yum only.

  • Make file repositories available through HTTP. Using TLS is currently only working with a global debugging certificate.

  • Add sync plans to products.

  • Spread out starting sync plans over a large time span. We do not recommend to start synchronizing a lot of repositories at the same time.

  • Synchronize your content regularly to avoid overly large changes in repositories. We recommend synchronizing content rather each night than only once a month.

  • Use file repositories for installation media. File repositories require a Pulp Manifest file which you can create using pulp-manifest /path/to/files. Use local repositories with file:///path/to/files as Upstream URL in orcharhino, for example for installation media. Alternatively, place the installation media file repositories on a web server that can be reached from hosts during provisioning.

    ATIX provides file repositories for installation media. Once synchronized, you can perform offline installations using a local installation media. See ATIX Service Portal for the upstream URL.

    ATIX provides the following installation media as file repository:

    • Debian 9 and 10

    • Ubuntu 18.04

    • Ubuntu 20.04

  • Use a Hammer script or Ansible Playbook to create a lot of products and repositories when starting with a fresh orcharhino installation.

  • For SUSE and RHEL content, use the SCC Manager Plugin or import the Red Hat Manifest file to orcharhino.

Managing Content Views

  • If you require daily updated content, use the content view Default Organization View. This contains the latest synchronized content from all repositories and is available in lifecycle environment Library.

  • Use content views and composite content views to assemble and structure lists of repositories. You can match different versions of content views or update specific content views when bundling them to a composite content view.

  • We recommend using composite content views only if you require more flexibility.

  • Making content management too small scale results in more manual work.

  • When using composite content views, publish the content views first, then the composite content views second. The more content views you bundle, the more work it becomes when making changes or updating content. We recommend making it as simple as possible and as complex as necessary.

  • Setting a lifecycle environment for content views is unnecessary if they are solely bundled to a composite content view.

  • Automate creating and publishing composite content views and lifecycle environments using Hammer scripts or Ansible Playbooks. Use cron jobs or recurring logics for more visibility. Recurring logics require SSH based access to orcharhino and the remote execution plugin.

  • We recommend adding the changes and date to the description of each published content view or composite content view version. The date shown in the management UI shows the latest action, for example promoting content to a different lifecycle environment, which is independent from the latest change to the actual content.

  • Publishing a new content view or composite content view creates a new major version. Incremental errata updates increment the minor version. Note that you cannot change or reset this counter.

Using Lifecycle Environments

  • Use multiple lifecycle environment paths to implement multiple sequential stages of content consumption. Each stage contains a defined set of content, for example in lifecycle environment Production.

  • Default use case: Fixed stages in each lifecycle environment paths, for example Development, Test, and Production.

    • Promote content views to lifecycle environments.

    • To make content from Test available in Production, promote the content view version from Test to Production. All content hosts consuming this content view or composite content view are able to install packages from the Production lifecycle environment. Note that these packages are not installed or updated automatically.

    • If you encounter errors during patching content hosts, attach the host to a previous version of the content view. This only affects the availability of packages but does not downgrade installed packages.

  • Alternative use case: Using stages in lifecycle environments for fixed content, for example quarterly updates.

    • Promote the content view to a specific stage, for example 2021-Q1 and only ever publish new minor versions when adding incremental updates from errata.

    • When patching content hosts, change the lifecycle environment from 2020-Q4 to 2021-Q1 using the management UI, the API, a Hammer script, or an activation key.

    • Advantage: You can directly see which software packages a hosts receives by looking at its lifecycle environment.

    • Disadvantage: Promoting content is less dynamic without clearly defined stages such as Development, Test, and Production.

  • Use multiple lifecycle environment paths to define multiple stages for different environments, for example to decouple web server and database hosts.

  • orcharhino Proxies running Pulp use lifecycle environments to synchronize content. They synchronize content more efficiently if you split content into multiple lifecycle environment paths. If a specific orcharhino Proxy only serves content for one operating system in a single lifecycle environment path, it only synchronizes actually required content.

Using Activation Keys

  • Register hosts to orcharhino with an activation key. You can add additional content using content views, lifecycle environments, subscriptions, and repository sets using the management UI, the API, Hammer scripts, or Ansible.

  • You can attach multiple activation keys to a host as long as all activation keys belong to the same lifecycle environment and content view. This allows you to separately activate additional repositories at a later stage.

  • We recommend using meaningful names for activation keys to indicate the content and lifecycle environment, for example ubuntu_20_webserver.

Patching Content Hosts

  • Attaching hosts to orcharhino requires the orcharhino clients which contain the subscription-manager package, katello-host-tools, and their dependencies. Use the bootstrap.py script to attach existing hosts. When attaching hosts manually, ensure to provide the necessary configuration files and certificates.

  • Use the management UI to install, upgrade, and remove packages from managed hosts. You can update content hosts using job templates using SSH and Ansible.

  • Apply errata on content hosts using the management UI.

  • Modify or replace job templates to add custom steps. This allows you to run commands or execute scripts on managed hosts.

  • When running bulk actions on managed hosts, we recommend bundling them by the same major operating system version, especially when upgrading packages.

  • Select via remote execution - customize first to define the time when patches are applied on managed hosts when performing bulk actions.

  • When patching packages on managed hosts using the default package manager, orcharhino receives a list of packages and repositories to recalculate applicable errata and available updates.

  • You cannot apply errata to packages that are not part of the repositories on orcharhino and the attached content view.

  • Modifications to installed packages using rpm or dpkg are sent to orcharhino with the next run of apt, yum, or zypper.

Errata

  • Errata are classified in several distinct types and contain information about what is fixed in which version. Applying errata updates the described packages.

  • orcharhino shows which errata are applicable for each content host by comparing package lists with installed and available packages.

  • View errata on the content hosts page and choose between errata from the current content view and lifecycle environment and the Library lifecycle environment which contains the latest synchronized packages.

    You can only apply errata included in the content view version of the lifecycle of your managed host. See this as a recommendation to promote content views.

  • You can subsequently add synchronized errata which are applicable on at least one content hosts to content views.

    • Find errata on Content > Errata filtering for a specific repository or name of an erratum.

    • Applicable errata indicate that a host contains packages that have errata available.

    • Installable errata indicate that a host has updated packages available in its content view ready to be installed.

    • If an erratum is applicable but not installable, select the erratum and click Apply. This displays information which hosts are affected by the erratum. Trying to install the erratum prompts you the option to create an incremental content view version and install the erratum afterwards.

    • An incremental update publishes a new minor version of the content view and promotes it to the necessary lifecycle environment. Use bulk actions to apply errata to multiple content hosts at once.

    • This allows you to add patches for security issues to a frozen set of content without unnecessarily updating other unaffected packages.

    • You cannot manually increment a content view without adding errata.

  • Your orcharhino subscriptions contains access to security errata for Debian and Ubuntu provided by ATIX.

Sync Plans

  • Use sync plans to periodically synchronize content to orcharhino.

  • Sync plans are associated with products.

  • Use cron line to specify recurring synchronization using cron expressions.

  • We recommend distributing synchronization tasks over several hours if possible to reduce the task load by creating multiple sync plans with cron line. Using a single sync plan would start all sync tasks at the same time resulting in a high load significantly lowering the performance of your orcharhino. This might also lead to occasional difficulties in the correct assigning of the Pulp tasks.

    Table 1. Cron Expression Examples
    Cron Expression Explanation

    0 22 * * 1-5

    every night at 22:00 from Monday through Friday

    30 3 * * 6,0

    at 03:30 on each Saturday and Sunday

    15 1-9/2 * * *

    15 minutes past every second hour from 1 through 9

    30 2 8-14 * *

    at 02:30 every day of the month from 8 through 14