Synchronizing Content Between orcharhino Servers

orcharhino uses Inter-Server Synchronization (ISS) to synchronize content between two orcharhino Servers including those that are airgapped.

Exporting from a Connected orcharhino Server

Connected orcharhino Server as a Content Store

In this scenario, you have one connected orcharhino Server that is receiving content from an external source like the CDN. The rest of your infrastructure is completely isolated, including another disconnected orcharhino Server. The connected orcharhino Server is mainly used as a content store for updates. The disconnected orcharhino Server serves as the main orcharhino Server for managing content for all infrastructure behind the isolated network.

On the connected orcharhino Server

  1. Ensure that repositories are using the immediate download policy, either by:

    1. For existing repos using On Demand, change their download policy on the repository details page to Immediate.

    2. For new repositories, ensure that the Default Red Hat Repository download policy setting is set to Immediate before enabling Red Hat repositories, and that the Default download policy is set to Immediate for custom repositories For more information, see Download Policies Overview.

  2. Enable the content that you want to use.

  3. Synchronize the enabled content.

  4. For the first export, perform a complete Library export so that all the synchronized content are exported. This generates content archives that you can later import into one or more disconnected orcharhino Servers. For more information on performing a complete Library export, see Exporting the Library Environment.

  5. Export all future updates in the connected orcharhino Servers incrementally. This generates leaner content archives that contain changes only from the recent set of updates. For example, if you enable and synchronize a new repository, the next exported content archive contains content only from the newly enabled repository. For more information on performing an incremental Library export, see Exporting Library Incrementally.

On the disconnected orcharhino Server

  1. Copy the exported content archive that you want from the connected orcharhino Server.

  2. Import content to an organization using the procedure outlined in Importing into the Library Environment. You can then manage content using Content Views or Lifecycle Environments as you require on the disconnected orcharhino Server.

Connected orcharhino Server for Managing Content View Versions

In this scenario, your connected orcharhino Server is receiving content from an external source like the CDN. The rest of your infrastructure is completely isolated, including a disconnected orcharhino Server. However, the connected orcharhino Server is not only used as a content store, but also is used to manage content. Updates coming from the CDN are curated into Content Views and Lifecycle Environments. Once the content has been promoted to a designated Lifecycle Environment, the content can be exported and imported into the disconnected orcharhino Server.

On the connected orcharhino Server

  1. Ensure that repositories are using the immediate download policy:

    1. For existing repos using On Demand, change their download policy on the repository details page to Immediate.

    2. For new repositories, ensure that the Default Red Hat Repository download policy setting is set to Immediate before enabling Red Hat repositories, and Default download policy is set to Immediate for custom repositories For more information, see Download Policies Overview.

  2. Enable any content that you want to use. For more information, see Enabling Red Hat Repositories.

  3. Synchronize the enabled content.

  4. When you have new content, republish the Content Views that includes this content. For more information, see Managing Content Views. This creates a new Content View Version with the appropriate content to export.

  5. For the first export, perform a complete version export on the Content View Version that you want to export. For more information see, Exporting a Content View Version. This generates content archives that you can import into one or more disconnected orcharhino Servers.

  6. Export all future updates in the connected orcharhino Servers incrementally. This generates leaner content archives that contain changes only from the recent set of updates. For example, if your Content View has a new repository, this exported content archive contains only the latest changes. For more information on performing an incremental export on Content View Versions, see Exporting Version Incrementally.

On the disconnected orcharhino Server

  1. Copy the exported content archive from the connected orcharhino Server.

  2. Import the content to the organization that you want. For more information, see Importing a Content View Version. This will create a Content View Version from the exported content archives and then import content appropriately.

Keeping Track of Your Exports

If you are exporting content to several disconnected orcharhino Servers, then using a --destination-server option provides a way to organize or maintain a record of what versions got exported to a given destination. For more information, see Examining the Exports.

This option is available for all content-export operations. You can use the destination-server to

  • Query what was previously exported to a given destination.

  • Generate incremental exports automatically to the given destination server.

Exporting a Content View Version

You can export a version of a Content View to an archive file from orcharhino Server and use this archive file to create the same Content View version on another orcharhino Server or on another orcharhino Server organization. orcharhino exports composite Content Views as normal Content Views. The composite nature is not retained. On importing the exported archive, a regular Content View is created or updated on your disconnected orcharhino Server. The exported archive file contains the following data:

  • A JSON file containing Content View version metadata

  • An archive file containing all the repositories included into the Content View version

orcharhino Server exports only deb and kickstart files added to a version of a Content View. orcharhino does not export the following content:

  • Docker content

  • Content View definitions and metadata, such as package filters.

Prerequisites

To export a Content View, ensure that orcharhino Server where you want to export meets the following conditions:

  • Ensure that the export directory has free storage space to accommodate the export.

  • Ensure that the /var/lib/pulp/exports directory has free storage space equivalent to the size of the repositories being exported for temporary files created during the export process.

  • Ensure that you set download policy to Immediate for all repositories within the Content View you export. For more information, see Download Policies Overview.

  • Ensure that you synchronize Products that you export to the required date.

  • Ensure that the user exporting the content has the Content Exporter role.

To Export a Content View Version

  1. List versions of the Content View that are available for export:

    # hammer content-view version list \
--content-view="My_Content_View" \
--organization="My_Organization"

---|----------|---------|-------------|-----------------------
ID | NAME     | VERSION | DESCRIPTION | LIFECYCLE ENVIRONMENTS
---|----------|---------|-------------|-----------------------
5  | view 3.0 | 3.0     |             | Library
4  | view 2.0 | 2.0     |             |
3  | view 1.0 | 1.0     |             |
---|----------|---------|-------------|----------------------
Export a Content View version

  1. Get the version number of desired version. The following example targets version 1.0 for export.

    # hammer content-export complete version \
--content-view="Content_View_Name" \
--version=1.0 \
--organization="My_Organization"

  2. Verify that the archive containing the exported version of a Content View is located in the export directory:

    # ls -lh /var/lib/pulp/exports/My_Organization/Content_View_Name/1.0/2021-02-25T18-59-26-00-00/

You require all three files, for example, the tar.gz archive file, the toc.json and metadata.json to import the content successfully.

Export with chunking

In many cases, the exported archive content can be several gigabytes in size. You might want to split it smaller sizes or chunks. You can use the --chunk-size-gb option with in the hammer content-export command to handle this. The following example uses the --chunk-size-gb=2 to split the archives into 2 GB chunks.

# hammer content-export complete version \
--chunk-size-gb=2 \
--content-view="Content_View_Name" \
--organization="My_Organization" \
--version=1.0
# ls -lh /var/lib/pulp/exports/My_Organization/view/1.0/2021-02-25T21-15-22-00-00/

Examining the Exports

When importing content to several orcharhino Servers, the --destination-server option is especially useful for keeping track of which content was exported and to where.

You can use this flag to let the exporting orcharhino Server keep track of content in specific servers. The --destination-server option functions to indicate the destination server that your content is imported to. The following example uses --destination-server=mirror1 to export content to mirror1. The archive is created on the exporting orcharhino Server. However, a record of each export is also maintained. This can be very useful when incrementally exporting.

# hammer content-export complete version \
--content-view="Content_View_Name" \
--destination-server=mirror1 \
--organization="My_Organization" \
--version=1.0

Exporting a Content View Version Incrementally

Exporting complete versions can be a very expensive operation in terms of system resources. The size of the exported Content View versions depends on the number of products. Content View versions that have multiple Debian trees can occupy several gigabytes of space on orcharhino Server.

In such cases, you can use Incremental Export to export only pieces of content that changed since the previous export. Incremental exports typically result in smaller archive files than the full exports.

The example below targets the version 2.0 for export, because the version 1.0 was exported previously.

Procedure

  1. Create an incremental export:

    # hammer content-export incremental version \
--content-view="Content_View_Name" \
--organization="My_Organization" \
--version=2.0

  2. Optional: View the exported Content View:

    # ls -lh /var/lib/pulp/exports/My_Organization/view/2.0/2021-02-25T21-45-34-00-00/

Viewing Content Export Reports

You can query on the exports that you previously have created via the hammer content-export list command.

hammer content-export list --organization="My_Organization"

---|--------------------|------------------------------------------------------------------------------|-------------|----------------------|-------------------------|-------------------------|------------------------
ID | DESTINATION SERVER | PATH                                                                         | TYPE        | CONTENT VIEW VERSION | CONTENT VIEW VERSION ID | CREATED AT              | UPDATED AT
---|--------------------|------------------------------------------------------------------------------|-------------|----------------------|-------------------------|-------------------------|------------------------
1  |                    | /var/lib/pulp/exports/My_Organization/view/1.0/2021-02-25T18-59-26-00-00 | complete    | view 1.0             | 3                       | 2021-02-25 18:59:30 UTC | 2021-02-25 18:59:30 UTC
2  |                    | /var/lib/pulp/exports/My_Organization/view/1.0/2021-02-25T21-15-22-00-00 | complete    | view 1.0             | 3                       | 2021-02-25 21:15:26 UTC | 2021-02-25 21:15:26 UTC
3  |                    | /var/lib/pulp/exports/My_Organization/view/2.0/2021-02-25T21-45-34-00-00 | incremental | view 2.0             | 4                       | 2021-02-25 21:45:37 UTC | 2021-02-25 21:45:37 UTC
---|--------------------|------------------------------------------------------------------------------|-------------|----------------------|-------------------------|-------------------------|------------------------

Importing a Content View Version

You can use the archive that the hammer content-export command outputs to create a version of a Content View with the same content as the exported Content View version. For more information about exporting a Content View version, see Exporting a Content View Version.

When you import a Content View version, it has the same major and minor version numbers and contains the same repositories with the same packages and errata. The Custom Repositories, Products and Content Views are automatically created if they do not exist in the importing organization.

Prerequisites

  1. The exported archive must be in a directory under /var/lib/pulp/imports.

  2. The directory must have pulp:pulp permissions so that Pulp can read and write the .json files in that directory.

  3. If there are any Red Hat repositories in the export archive, the importing organization’s manifest must contain subscriptions for the products contained within the export.

  4. The user importing the Content View Version must have the 'Content Importer' Role.

Procedure

  1. Copy the archived file with the exported Content View version to the /var/lib/pulp/imports directory on orcharhino Server where you want to import.

  2. Set the user:group permission of the archive files to pulp:pulp.

    # chown -R pulp:pulp /var/lib/pulp/imports/2021-02-25T21-15-22-00-00/

  3. Verify that the permission change occurs:

    # ls -lh /var/lib/pulp/imports/2021-02-25T21-15-22-00-00/

  4. To import the Content View version to orcharhino Server, enter the following command:

    # hammer content-import version \
--organization-id=My_Organization_ID \
--path=/var/lib/pulp/imports/2021-02-25T21-15-22-00-00/

    Note that you must enter the full path /var/lib/pulp/imports/<path>. Relative paths do not work.

  5. To verify that you import the Content View version successfully, list Content Views for your organization:

    # hammer content-view version list \
--organization-id=My_Organization_ID

Exporting the Library Environment

You can export contents of all Yum repositories in the Library environment of an organization to an archive file from orcharhino Server and use this archive file to create the same repositories in another orcharhino Server or in another orcharhino Server organization. The exported archive file contains the following data:

  • A JSON file containing Content View version metadata

  • An archive file containing all the repositories from the Library environment of the organization.

orcharhino Server exports only RPM and kickstart files included in a Content View version. orcharhino does not export the following content:

  • Docker content

Prerequisites

To export the contents of the Library lifecycle environment of the organization, ensure that orcharhino Server where you want to export meets the following conditions:

  • Ensure that the export directory has free storage space to accommodate the export.

  • Ensure that the /var/lib/pulp/exports directory has free storage space equivalent to the size of the repositories being exported for temporary files created during the export process.

  • Ensure that you set download policy to Immediate for all repositories within the Library lifecycle environment you export. For more information, see Download Policies Overview.

  • Ensure that you synchronize Products that you export to the required date.

To Export the Library Content of an Organization:

Use the organization name or ID to export.

# hammer content-export complete library --organization="My_Organization"

  1. Verify that the archive containing the exported version of a Content View is located in the export directory:

    # ls -lh /var/lib/pulp/exports/My_Organization/Export-Library/1.0/2021-03-02T03-35-24-00-00
total 68M
-rw-r--r--. 1 pulp pulp 68M Mar  2 03:35 export-1e25417c-6d09-49d4-b9a5-23df4db3d52a-20210302_0335.tar.gz
-rw-r--r--. 1 pulp pulp 333 Mar  2 03:35 export-1e25417c-6d09-49d4-b9a5-23df4db3d52a-20210302_0335-toc.json
-rw-r--r--. 1 root root 443 Mar  2 03:35 metadata.json

  2. You require all three files, for example, the tar.gz, the toc.json and the metadata.json file to be able to import.

  3. A new Content View Export-Library is created in the organization. This Content View contains all the repositories belonging to this organization. A new version of this Content View is published and exported automatically.

Export with chunking

In many cases the exported archive content may be several gigabytes in size. If you want to split it into smaller sizes or chunks. You can use the --chunk-size-gb flag directly in the export command to handle this. In the following example, you can see how to specify --chunk-size-gb=2 to split the archives in 2 GB chunks.

# hammer content-export complete library \
--chunk-size-gb=2 \
--organization="My_Organization"

Generated /var/lib/pulp/exports/My_Organization/Export-Library/2.0/2021-03-02T04-01-25-00-00/metadata.json

# ls -lh /var/lib/pulp/exports/My_Organization/Export-Library/2.0/2021-03-02T04-01-25-00-00/

Exporting the Library Environment Incrementally

Exporting Library content can be a very expensive operation in terms of system resources. The size of the exported Library depends on the number of products. Organizations that have multiple Debian trees can occupy several gigabytes of space on orcharhino Server.

In such cases, you can use Incremental Export to export only pieces of content that changed since the previous export. Incremental exports typically result in smaller archive files than the full exports.

The example below shows incremental export of all repositories in the organization’s Library.

Procedure

  1. Create an incremental export:

    # hammer content-export incremental library --organization="My_Organization"

Generated /var/lib/pulp/exports/My_Organization/Export-Library/3.0/2021-03-02T04-22-14-00-00/metadata.json

  2. Optional: View the exported data:

    # ls -lh /var/lib/pulp/exports/My_Organization/Export-Library/3.0/2021-03-02T04-22-14-00-00/
total 172K
-rw-r--r--. 1 pulp pulp 161K Mar  2 04:22 export-436882d8-de5a-48e9-a30a-17169318f908-20210302_0422.tar.gz
-rw-r--r--. 1 pulp pulp  333 Mar  2 04:22 export-436882d8-de5a-48e9-a30a-17169318f908-20210302_0422-toc.json
-rw-r--r--. 1 root root  492 Mar  2 04:22 metadata.json

Importing into the Library Environment

You can use the archive that the hammer content-export command outputs to import into the Library lifecycle environment of another organization For more information about exporting contents from the Library environment, see Exporting the Library Environment.

Prerequisites

To import in to an Organization’s library lifecycle environment ensure that orcharhino Server where you want to import meets the following conditions:

  1. The exported archive must be in a directory under /var/lib/pulp/imports.

  2. The directory must have pulp:pulp permissions so that Pulp can read and write the .json files in that directory.

  3. If there are any Red Hat repositories in the export archive, the importing organization’s manifest must contain subscriptions for the products contained within the export.

  4. The user importing the content must have the 'Content Importer' Role.

Procedure

  1. Copy the archived file with the exported Content View version to the /var/lib/pulp/imports directory on orcharhino Server where you want to import.

  2. Set the permission of the archive files to pulp:pulp.

    # chown -R pulp:pulp /var/lib/pulp/imports/2021-03-02T03-35-24-00-00
# ls -lh /var/lib/pulp/imports/2021-03-02T03-35-24-00-00
total 68M
-rw-r--r--. 1 pulp pulp 68M Mar  2 04:29 export-1e25417c-6d09-49d4-b9a5-23df4db3d52a-20210302_0335.tar.gz
-rw-r--r--. 1 pulp pulp 333 Mar  2 04:29 export-1e25417c-6d09-49d4-b9a5-23df4db3d52a-20210302_0335-toc.json
-rw-r--r--. 1 pulp pulp 443 Mar  2 04:29 metadata.json

  3. On orcharhino Server where you want to import, create/enable repositories the same name and label as the exported content.

  4. In the orcharhino management UI, navigate to Content > Products, click the Yum content tab and add the same Yum content that the exported Content View version includes.

  5. Identify the Organization that you wish to import into.

  6. To import the Library content to orcharhino Server, enter the following command:

    # hammer content-import library \
--organization="My_Organization" \
--path=/var/lib/pulp/imports/2021-03-02T03-35-24-00-00

    Note you must enter the full path /var/lib/pulp/imports/<path>. Relative paths do not work.

  7. To verify that you imported the Library content, check the contents of the Product and Repositories. A new Content View called Import-Library is created in the target organization. This Content View is used to facilitate the library content import.

Import/Export Cheat Sheet

Table 1. Export
Intent Command

Fully Export a Content View version

hammer content-export complete version --content-view="My_Content_View" --version=1.0 --organization="My_Organization"

Incrementally Export a Content View version (assuming you have already exported something)

hammer content-export incremental version --content-view="My_Content_View" --version=2.0 --organization="My_Organization"

Fully Export a Repository

hammer content-export complete repository --product="My_Product" --name="My_Repository" --organization="My_Organization"

Incrementally Export a Repository (assuming you have already exported something)

hammer content-export incremental repository --product="My_Product" --name="My_Repository" --organization="My_Organization"

Fully Export an Organization’s Library

hammer content-export complete library --organization="My_Organization"

Incrementally Export an Organization’s Library (assuming you have already exported something)

hammer content-export incremental library --organization="My_Organization"

Export a Content View version promoted to the Dev Environment

hammer content-export complete version --content-view="My_Content_View" --organization="My_Organization" --lifecycle-environment="Dev"

Export a Content View in smaller chunks (2-GB slabs)

hammer content-export complete version --content-view="My_Content_View" --version=1.0 --organization="My_Organization" --chunk-size-gb=2

Get a list of exports

hammer content-export list --content-view="My_Content_View" --organization="My_Organization"
Table 2. Import
Intent Command

Import to a Content View version

hammer content-import version --organization="My_Organization" --path="/var/lib/pulp/imports/My_Exported_Repo_Dir"

Import a Repository

hammer content-import repository --organization="My_Organization" --path="/var/lib/pulp/imports/My_Exported_Repo_Dir"

Import to an Organization’s Library

hammer content-import library --organization="My_Organization" --path="/var/lib/pulp/imports/My_Exported_Repo_Dir"

The text and illustrations on this page are licensed by ATIX AG under a Creative Commons Attribution–Share Alike 3.0 Unported ("CC-BY-SA") license. This page also contains text from the official Foreman documentation which uses the same license ("CC-BY-SA").