Synchronizing Content Between orcharhino Servers
In a orcharhino setup with multiple orcharhino Servers, you can use Inter-Server Synchronization (ISS) to synchronize content from one upstream server to one or more downstream servers.
There are two possible ISS configurations of orcharhino, depending on how you deployed your infrastructure:
- ISS Network Sync
-
If your upstream server can communicate with the downstream server over a network, you can synchronize content over HTTPS.
Configure your orcharhino to synchronize content over a network. For more information, see configuring server to synchronize content over a network.
- ISS Export Sync
-
If your upstream and downstream servers are air-gapped, you can synchronize content using export and import.
Configure your orcharhino to synchronize content using export and import. For more information, see configuring server to sync content using exports.
-
For more information on ISS use cases and scenarios, see how to configure inter server sync.
How to Synchronize Content Using Export and Import
There are multiple approaches for synchronizing content using the export and import workflow:
-
You employ the upstream orcharhino Server as a content store, which means that you sync the whole Library rather than Content View versions. This approach offers the simplest export/import workflow. In such case, you can manage the versions downstream. For more information, see Using Upstream Server as a Content Store.
-
You use the upstream orcharhino Server to sync Content View versions. This approach offers more control over what content is synced between orcharhino Servers. For more information, see Using Upstream Server to Sync Content View Versions.
-
You sync a single repository. This can be useful if you use the Content-View syncing approach, but you want to sync an additional repository without adding it to an existing Content View. For more information, see synchronizing a single repository.
Using an Upstream orcharhino Server as a Content Store
In this scenario, you use the upstream orcharhino Server as a content store for updates rather than to manage content. You use the downstream orcharhino Server to manage content for all infrastructure behind the isolated network. You export the Library content from the upstream orcharhino Server and import it into the downstream orcharhino Server.
-
Ensure that repositories are using the Immediate download policy in one of the following ways:
-
For existing repositories using On Demand, change their download policy on the repository details page to Immediate.
-
For new repositories, ensure that the Default Red Hat Repository download policy setting is set to Immediate before enabling Red Hat repositories, and that the Default download policy is set to Immediate for custom repositories.
For more information, see Download Policies Overview.
-
-
Enable the content that you want to synchronize. For more information, see Enabling Red Hat Repositories.
If you want to sync custom content, first create a custom Product and synchronize Product repositories.
-
Synchronize the enabled content:
-
On the first export, perform a
complete
Library export so that all the synchronized content is exported. This generates content archives that you can later import into one or more downstream orcharhino Servers. For more information on performing a complete Library export, see Exporting the Library Environment. -
Export all future updates on the upstream orcharhino Server incrementally. This generates leaner content archives that contain only a recent set of updates. For example, if you enable and synchronize a new repository, the next exported content archive contains content only from the newly enabled repository. For more information on performing an incremental Library export, see Exporting Library Incrementally.
-
-
Bring the content exported from the upstream orcharhino Server over to the hard disk.
-
Place it inside a directory under
/var/lib/pulp/imports
. -
Import the content to an organization using the procedure outlined in Importing into the Library Environment.
You can then manage content using Content Views or Lifecycle Environments as you require.
Using an Upstream orcharhino Server to Sync Content View Versions
In this scenario, you use the upstream orcharhino Server not only as a content store, but also to synchronize content for all infrastructure behind the isolated network. You curate updates coming from the CDN into Content Views and Lifecycle Environments. Once you promote content to a designated Lifecycle Environment, you can export the content from the upstream orcharhino Server and import it into the downstream orcharhino Server.
-
Ensure that repositories are using the Immediate download policy in one of the following ways:
-
For existing repositories using On Demand, change their download policy on the repository details page to Immediate.
-
For new repositories, ensure that the Default Red Hat Repository download policy setting is set to Immediate before enabling Red Hat repositories, and that the Default download policy is set to Immediate for custom repositories.
For more information, see Download Policies Overview.
-
-
Enable the content that you want to synchronize. For more information, see Enabling Red Hat Repositories.
If you want to sync custom content, first create a custom Product and synchronize Product repositories.
-
Synchronize the enabled content:
-
For the first export, perform a
complete
Version export on the Content View Version that you want to export. For more information see, Exporting a Content View Version. This generates content archives that you can import into one or more downstream orcharhino Servers. -
Export all future updates in the connected orcharhino Servers incrementally. This generates leaner content archives that contain changes only from the recent set of updates. For example, if your Content View has a new repository, this exported content archive contains only the latest changes. For more information, see Exporting Version Incrementally.
-
When you have new content, republish the Content Views that include this content before exporting the increment. For more information, see Managing Content Views. This creates a new Content View Version with the appropriate content to export.
-
-
Bring the content exported from the upstream orcharhino Server over to the hard disk.
-
Place it inside a directory under
/var/lib/pulp/imports
. -
Import the content to the organization that you want. For more information, see Importing a Content View Version. This will create a Content View Version from the exported content archives and then import content appropriately.
Synchronizing a Single Repository
In this scenario, you export and import a single repository.
-
Ensure that the repository is using the Immediate download policy in one of the following ways:
-
For existing repositories using On Demand, change their download policy on the repository details page to Immediate.
-
For new repositories, ensure that the Default Red Hat Repository download policy setting is set to Immediate before enabling Red Hat repositories, and that the Default download policy is set to Immediate for custom repositories.
For more information, see Download Policies Overview.
-
-
Enable the content that you want to synchronize. For more information, see Enabling Red Hat Repositories.
If you want to sync custom content, first create a custom Product and synchronize Product repositories.
-
Synchronize the enabled content:
-
On the first export, perform a
complete
repository export so that all the synchronized content is exported. This generates content archives that you can later import into one or more downstream orcharhino Servers. For more information on performing a complete repository export, see Exporting a Repository. -
Export all future updates on the upstream orcharhino Server incrementally. This generates leaner content archives that contain only a recent set of updates. For more information on performing an incremental repository export, see Exporting Repository Incrementally.
-
-
Bring the content exported from the upstream orcharhino Server over to the hard disk.
-
Place it inside a directory under
/var/lib/pulp/imports
. -
Import the content to an organization. See Importing a Repository.
You can then manage content using Content Views or Lifecycle Environments as you require.
Exporting the Library Environment
You can export contents of all Yum repositories in the Library environment of an organization to an archive file from orcharhino Server and use this archive file to create the same repositories in another orcharhino Server or in another orcharhino Server organization. The exported archive file contains the following data:
-
A JSON file containing Content View version metadata.
-
An archive file containing all the repositories from the Library environment of the organization.
orcharhino Server exports only RPM and kickstart files included in a Content View version. orcharhino does not export the following content:
-
Docker content
To export the contents of the Library lifecycle environment of the organization, ensure that orcharhino Server where you want to export meets the following conditions:
-
Ensure that the export directory has free storage space to accommodate the export.
-
Ensure that the
/var/lib/pulp/exports
directory has free storage space equivalent to the size of the repositories being exported for temporary files created during the export process. -
Ensure that you set download policy to Immediate for all repositories within the Library lifecycle environment you export. For more information, see Download Policies Overview.
-
Ensure that you synchronize Products that you export to the required date.
-
Use the organization name or ID to export.
# hammer content-export complete library --organization="My_Organization"
-
Verify that the archive containing the exported version of a Content View is located in the export directory:
# ls -lh /var/lib/pulp/exports/My_Organization/Export-Library/1.0/2021-03-02T03-35-24-00-00 total 68M -rw-r--r--. 1 pulp pulp 68M Mar 2 03:35 export-1e25417c-6d09-49d4-b9a5-23df4db3d52a-20210302_0335.tar.gz -rw-r--r--. 1 pulp pulp 333 Mar 2 03:35 export-1e25417c-6d09-49d4-b9a5-23df4db3d52a-20210302_0335-toc.json -rw-r--r--. 1 pulp pulp 443 Mar 2 03:35 metadata.json
You need all three files, the
tar.gz
, thetoc.json
, and themetadata.json
file to be able to import. -
A new Content View Export-Library is created in the organization. This Content View contains all the repositories belonging to this organization. A new version of this Content View is published and exported automatically.
In many cases the exported archive content may be several gigabytes in size.
If you want to split it into smaller sizes or chunks.
You can use the --chunk-size-gb
flag directly in the export command to handle this.
In the following example, you can see how to specify --chunk-size-gb=2
to split the archives in 2 GB
chunks.
# hammer content-export complete library \ --chunk-size-gb=2 \ --organization="My_Organization" Generated /var/lib/pulp/exports/My_Organization/Export-Library/2.0/2021-03-02T04-01-25-00-00/metadata.json # ls -lh /var/lib/pulp/exports/My_Organization/Export-Library/2.0/2021-03-02T04-01-25-00-00/
Exporting the Library Environment Incrementally
Exporting Library content can be a very expensive operation in terms of system resources. The size of the exported Library depends on the number of products. Organizations that have multiple Amazon Linux trees can occupy several gigabytes of space on orcharhino Server.
In such cases, you can use Incremental Export to export only pieces of content that changed since the previous export. Incremental exports typically result in smaller archive files than the full exports.
The example below shows incremental export of all repositories in the organization’s Library.
-
Create an incremental export:
# hammer content-export incremental library --organization="My_Organization" Generated /var/lib/pulp/exports/My_Organization/Export-Library/3.0/2021-03-02T04-22-14-00-00/metadata.json
-
Optional: View the exported data:
# ls -lh /var/lib/pulp/exports/My_Organization/Export-Library/3.0/2021-03-02T04-22-14-00-00/ total 172K -rw-r--r--. 1 pulp pulp 161K Mar 2 04:22 export-436882d8-de5a-48e9-a30a-17169318f908-20210302_0422.tar.gz -rw-r--r--. 1 pulp pulp 333 Mar 2 04:22 export-436882d8-de5a-48e9-a30a-17169318f908-20210302_0422-toc.json -rw-r--r--. 1 pulp pulp 492 Mar 2 04:22 metadata.json
Exporting a Content View Version
You can export a version of a Content View to an archive file from orcharhino Server and use this archive file to create the same Content View version on another orcharhino Server or on another orcharhino Server organization. orcharhino exports composite Content Views as normal Content Views. The composite nature is not retained. On importing the exported archive, a regular Content View is created or updated on your downstream orcharhino Server. The exported archive file contains the following data:
-
A JSON file containing Content View version metadata
-
An archive file containing all the repositories included into the Content View version
orcharhino Server exports only RPM and kickstart files added to a version of a Content View. orcharhino does not export the following content:
-
Docker content
-
Content View definitions and metadata, such as package filters.
To export a Content View, ensure that orcharhino Server where you want to export meets the following conditions:
-
Ensure that the export directory has free storage space to accommodate the export.
-
Ensure that the
/var/lib/pulp/exports
directory has free storage space equivalent to the size of the repositories being exported for temporary files created during the export process. -
Ensure that you set download policy to Immediate for all repositories within the Content View you export. For more information, see Download Policies Overview.
-
Ensure that you synchronize Products that you export to the required date.
-
Ensure that the user exporting the content has the
Content Exporter
role.
-
List versions of the Content View that are available for export:
# hammer content-view version list \ --content-view="My_Content_View" \ --organization="My_Organization" ---|----------|---------|-------------|----------------------- ID | NAME | VERSION | DESCRIPTION | LIFECYCLE ENVIRONMENTS ---|----------|---------|-------------|----------------------- 5 | view 3.0 | 3.0 | | Library 4 | view 2.0 | 2.0 | | 3 | view 1.0 | 1.0 | | ---|----------|---------|-------------|----------------------
-
Get the version number of desired version. The following example targets version
1.0
for export.# hammer content-export complete version \ --content-view="Content_View_Name" \ --version=1.0 \ --organization="My_Organization"
-
Verify that the archive containing the exported version of a Content View is located in the export directory:
# ls -lh /var/lib/pulp/exports/My_Organization/Content_View_Name/1.0/2021-02-25T18-59-26-00-00/
You require all three files, for example, the tar.gz
archive file, the toc.json
and metadata.json
to import the content successfully.
In many cases, the exported archive content can be several gigabytes in size.
You might want to split it smaller sizes or chunks.
You can use the --chunk-size-gb
option with in the hammer content-export
command to handle this.
The following example uses the --chunk-size-gb=2
to split the archives into 2 GB
chunks.
# hammer content-export complete version \ --chunk-size-gb=2 \ --content-view="Content_View_Name" \ --organization="My_Organization" \ --version=1.0 # ls -lh /var/lib/pulp/exports/My_Organization/view/1.0/2021-02-25T21-15-22-00-00/
Exporting a Content View Version Incrementally
Exporting complete versions can be a very expensive operation in terms of system resources. The size of the exported Content View versions depends on the number of products. Content View versions that have multiple Amazon Linux trees can occupy several gigabytes of space on orcharhino Server.
In such cases, you can use Incremental Export to export only pieces of content that changed since the previous export. Incremental exports typically result in smaller archive files than the full exports.
The example below targets the version 2.0
for export, because the version 1.0
was exported previously.
-
Create an incremental export:
# hammer content-export incremental version \ --content-view="Content_View_Name" \ --organization="My_Organization" \ --version=2.0
-
Optional: View the exported Content View:
# ls -lh /var/lib/pulp/exports/My_Organization/view/2.0/2021-02-25T21-45-34-00-00/
Exporting a Repository
You can export the content of a repository in the Library environment of an organization from orcharhino Server. You can use this archive file to create the same repository in another orcharhino Server or in another orcharhino Server organization.
You can export the following content from orcharhino Server:
-
Ansible repositories
-
Kickstart repositories
-
Yum repositories
-
File repositories
-
Docker content
You cannot export DEB repositories from orcharhino Server.
The export contains the following data:
-
Two JSON files containing repository metadata.
-
One or more archive files containing the contents of the repository from the Library environment of the organization.
You need all the files, tar.gz
, toc.json
and metadata.json
, to be able to import.
-
Ensure that the export directory has enough free storage space to accommodate the export.
-
Ensure that the
/var/lib/pulp/exports
directory has enough free storage space equivalent to the size of all repositories that you want to export. -
Ensure that you set download policy to Immediate for the repository within the Library lifecycle environment you export. For more information, see Download Policies Overview.
-
Ensure that you synchronize products that you export to the required date.
-
Export a repository using the product name and repository name:
# hammer content-export complete repository \ --name="My_Repository" \ --product="My_Product"
The size of the exported archive depends on the number and size of the packages within the repository. If you want to split the exported archive into chunks, export your repository using the
--chunk-size-gb
argument to limit the size by an integer value in gigabytes, for example---chunk-size-gb=2
. -
Optional: Verify that the exported archive is located in the export directory:
# ls -lh /var/lib/pulp/exports/My_Organization/Export-My_Repository/1.0/2022-09-02T03-35-24-00-00/
Exporting a Repository Incrementally
Exporting a repository can be a very expensive operation in terms of system resources. A typical Amazon Linux tree may occupy several gigabytes of space on orcharhino Server.
In such cases, you can use Incremental Export to export only pieces of content that changed since the previous export. Incremental exports typically result in smaller archive files than the full exports.
The example below shows incremental export of a repository in the Library lifecycle environment.
-
Create an incremental export:
# hammer content-export incremental repository \ --organization="My_Organization" \ --product="My_Product" \ --name="My_Repository" Generated /var/lib/pulp/exports/My_Organization/Export-My_Repository/3.0/2021-03-02T03-35-24-00-00/metadata.json
-
Optional: View the exported data:
# ls -lh /var/lib/pulp/exports/My_Organization/Export-My_Repository/3.0/2021-03-02T03-35-24-00-00/ total 172K -rw-r--r--. 1 pulp pulp 20M Mar 2 04:22 export-436882d8-de5a-48e9-a30a-17169318f908-20210302_0422.tar.gz -rw-r--r--. 1 pulp pulp 333 Mar 2 04:22 export-436882d8-de5a-48e9-a30a-17169318f908-20210302_0422-toc.json -rw-r--r--. 1 root root 492 Mar 2 04:22 metadata.json
Keeping Track of Your Exports
orcharhino keeps records of all exports. Each time you export content on the upstream orcharhino Server, the export is recorded and maintained for future querying. You can use the records to organize and manage your exports, which is useful especially when exporting incrementally.
When exporting content from the upstream orcharhino Server for several downstream orcharhino Servers, you can also keep track of content exported for specific servers. This helps you track which content was exported and to where.
Use the --destination-server
argument during export to indicate the target server.
This option is available for all content-export
operations.
-
Specify the destination server when exporting the Library:
# hammer content-export complete library \ --destination-server=My_Downstream_Server_1 \ --organization="My_Organization" \ --version=1.0
-
Specify the destination server when exporting a Content View version:
# hammer content-export complete version \ --content-view="Content_View_Name" \ --destination-server=My_Downstream_Server_1 \ --organization="My_Organization" \ --version=1.0
-
List content exports using the following command:
# hammer content-export list --organization="My_Organization"
Importing into the Library Environment
You can import exported Library content into the Library lifecycle environment of an organization on another orcharhino Server. For more information about exporting contents from the Library environment, see Exporting the Library Environment.
-
The exported files must be in a directory under
/var/lib/pulp/imports
. -
If there are any Red Hat repositories in the exported content, the importing organization’s manifest must contain subscriptions for the products contained within the export.
-
The user importing the content must have the Content Importer Role.
-
Copy the exported files to a subdirectory of
/var/lib/pulp/imports
on orcharhino Server where you want to import. -
Set the ownership of the import directory and its contents to
pulp:pulp
.# chown -R pulp:pulp /var/lib/pulp/imports/2021-03-02T03-35-24-00-00 # ls -lh /var/lib/pulp/imports/2021-03-02T03-35-24-00-00 total 68M -rw-r--r--. 1 pulp pulp 68M Mar 2 04:29 export-1e25417c-6d09-49d4-b9a5-23df4db3d52a-20210302_0335.tar.gz -rw-r--r--. 1 pulp pulp 333 Mar 2 04:29 export-1e25417c-6d09-49d4-b9a5-23df4db3d52a-20210302_0335-toc.json -rw-r--r--. 1 pulp pulp 443 Mar 2 04:29 metadata.json
-
Identify the Organization that you wish to import into.
-
To import the Library content to orcharhino Server, enter the following command:
# hammer content-import library \ --organization="My_Organization" \ --path=/var/lib/pulp/imports/2021-03-02T03-35-24-00-00
Note you must enter the full path
/var/lib/pulp/imports/My_Exported_Library_Dir
. Relative paths do not work. -
To verify that you imported the Library content, check the contents of the Product and Repositories. A new Content View called
Import-Library
is created in the target organization. This Content View is used to facilitate the Library content import.By default, this Content View is not shown in the orcharhino management UI.
Import-Library
is not meant to be assigned directly to hosts. Instead, assign your hosts toDefault Organization View
or another Content View as you would normally.
Importing a Content View Version
You can import an exported Content View version to create a version with the same content in an organization on another orcharhino Server. For more information about exporting a Content View version, see Exporting a Content View Version.
When you import a Content View version, it has the same major and minor version numbers and contains the same repositories with the same packages and errata. Custom Repositories, Products and Content Views are automatically created if they do not exist in the importing organization.
-
The exported files must be in a directory under
/var/lib/pulp/imports
. -
If there are any Red Hat repositories in the exported content, the importing organization’s manifest must contain subscriptions for the products contained within the export.
-
The user importing the Content View version must have the Content Importer Role.
-
Copy the exported files to a subdirectory of
/var/lib/pulp/imports
on orcharhino Server where you want to import. -
Set the ownership of the import directory and its contents to
pulp:pulp
.# chown -R pulp:pulp /var/lib/pulp/imports/2021-02-25T21-15-22-00-00/
-
Verify that the ownership is set correctly:
# ls -lh /var/lib/pulp/imports/2021-02-25T21-15-22-00-00/
-
To import the Content View version to orcharhino Server, enter the following command:
# hammer content-import version \ --organization-id=My_Organization_ID \ --path=/var/lib/pulp/imports/2021-02-25T21-15-22-00-00/
Note that you must enter the full path
/var/lib/pulp/imports/My_Exported_Version_Dir
. Relative paths do not work. -
To verify that you imported the Content View version successfully, list Content View versions for your organization:
# hammer content-view version list \ --organization-id=My_Organization_ID
Importing a Repository
You can import an exported repository into an organization on another orcharhino Server. For more information about exporting content of a repository, see Exporting a Repository.
-
The export files must be in a directory under
/var/lib/pulp/imports
. -
If the export contains any Red Hat repositories, the manifest of the importing organization must contain subscriptions for the products contained within the export.
-
The user importing the content must have the Content Importer Role.
-
Copy the exported files to a subdirectory of
/var/lib/pulp/imports
on orcharhino Server where you want to import. -
Set the ownership of the import directory and its contents to
pulp:pulp
.# chown -R pulp:pulp /var/lib/pulp/imports/2021-03-02T03-35-24-00-00 # ls -lh /var/lib/pulp/imports/2021-03-02T03-35-24-00-00 total 68M -rw-r--r--. 1 pulp pulp 68M Mar 2 04:29 export-1e25417c-6d09-49d4-b9a5-23df4db3d52a-20210302_0335.tar.gz -rw-r--r--. 1 pulp pulp 333 Mar 2 04:29 export-1e25417c-6d09-49d4-b9a5-23df4db3d52a-20210302_0335-toc.json -rw-r--r--. 1 pulp pulp 443 Mar 2 04:29 metadata.json
-
Identify the Organization that you wish to import into.
-
To import the Library content to orcharhino Server, enter the following command:
# hammer content-import repository \ --organization="My_Organization" \ --path=/var/lib/pulp/imports/2021-03-02T03-35-24-00-00
Note that you must enter the full path
/var/lib/pulp/imports/My_Exported_Repo_Dir
. Relative paths do not work. -
To verify that you imported the repository, check the contents of the Product and Repository.
Exporting and Importing Content using Hammer CLI Cheat Sheet
Intent | Command |
---|---|
Fully export an Organization’s Library |
|
Incrementally export an Organization’s Library (assuming you have exported something previously) |
|
Fully export a Content View version |
|
Export a Content View version promoted to the Dev Environment |
|
Export a Content View in smaller chunks (2-GB slabs) |
|
Incrementally export a Content View version (assuming you have exported something previously) |
|
Fully export a Repository |
|
Incrementally export a Repository (assuming you have exported something previously) |
|
List exports |
|
Intent | Command |
---|---|
Import into an Organization’s Library |
|
Import to a Content View version |
|
Import a Repository |
|
How to Configure Inter-Server Synchronization
orcharhino uses Inter-Server Synchronization (ISS) to synchronize content between two orcharhino Servers including those that are air-gapped.
You can use ISS in cases such as:
-
If you want to copy some but not all content from your orcharhino Server to other orcharhino Servers. For example, you have Content Views that your IT department consumes from orcharhino Server, and you want to copy content from those Content Views to other orcharhino Servers.
-
If you want to copy all Library content from your orcharhino Server to other orcharhino Servers. For example, you have Products and repositories that your IT department consumes from orcharhino Server in the Library, and you want to copy all Products and repositories in that organization to other orcharhino Servers.
You cannot use ISS to synchronize content from orcharhino Server to orcharhino Proxy. orcharhino Proxy supports synchronization natively. For more information, see orcharhino Proxy Overview in Installing and Maintaining orcharhino. |
There are different ways of using ISS. The way you can use depends on your multi-server setup that can fall to one of the following scenarios.
ISS Network Sync in a Disconnected Scenario
In a disconnected scenario, there is the following setup:
-
The upstream orcharhino Server is connected to the Internet. This server consumes content from the Red Hat Content Delivery Network (CDN) or custom sources.
-
The downstream orcharhino Server is completely isolated from all external networks.
-
The downstream orcharhino Server can communicate with a connected upstream orcharhino Server over an internal network.
You can configure your downstream orcharhino Server to synchronize content from the upstream orcharhino Server over the network. See configuring server to synchronize content over a network.
ISS Export Sync in an Air-Gapped Scenario
In an air-gapped scenario, there is the following setup:
-
The upstream orcharhino Server is connected to the Internet. This server consumes content from the Red Hat CDN or custom sources.
-
The downstream orcharhino Server is completely isolated from all external networks.
-
The downstream orcharhino Server does not have a network connection to a connected upstream orcharhino Server.
The only way for an air-gapped downstream orcharhino Server to receive content updates is by exporting payload from the upstream orcharhino Server, bringing it physically to the downstream orcharhino Server, and importing the payload. For more information, see Synchronizing Content Between orcharhino Servers in Managing Content.
Configure your downstream orcharhino Server to synchronize content using exports. See configuring server to sync content using exports.
The text and illustrations on this page are licensed by ATIX AG under a Creative Commons Attribution–Share Alike 3.0 Unported ("CC-BY-SA") license. This page also contains text from the official Foreman documentation which uses the same license ("CC-BY-SA"). |