Optimizing content synchronization and storage

Use download and mirroring policies, sync concurrency limits, and package retention to balance local storage, bandwidth, and upstream alignment on orcharhino Server and orcharhino Proxy Servers. Configure defaults for new repositories or adjust individual repositories as needed.

Download policies overview

Download policies control when orcharhino stores Deb, Yum, and container image content during synchronization versus fetching it on request from orcharhino Proxies or hosts.

orcharhino Server has the following policies:

Immediate: orcharhino Server downloads all metadata and packages during synchronization.
On Demand: orcharhino Server downloads only the metadata during synchronization. orcharhino Server only fetches and stores packages on the file system when orcharhino Proxies or directly connected clients request them. This setting has no effect if you set a corresponding repository on a orcharhino Proxy to Immediate because orcharhino Server is forced to download all the packages.

The On Demand policy acts as a Lazy Synchronization feature because they save time synchronizing content. The lazy synchronization feature must be used only for Deb and Yum repositories. You can add the packages to content views and promote to lifecycle environments as normal.

orcharhino Proxy Server has the following policies:

Immediate: orcharhino Proxy Server downloads all metadata and packages during synchronization. Do not use this setting if the corresponding repository on orcharhino Server is set to On Demand as orcharhino Server is forced to download all the packages.
On Demand: orcharhino Proxy Server only downloads the metadata during synchronization. orcharhino Proxy Server fetches and stores packages only on the file system when directly connected clients request them. When you use an On Demand download policy, content is downloaded from orcharhino Server if it is not available on orcharhino Proxy Server.
Inherit: orcharhino Proxy Server inherits the download policy for the repository from the corresponding repository on orcharhino Server.
Streamed: Streamed download policy for orcharhino Proxies permits orcharhino Proxies to avoid caching any content. When content is requested from the orcharhino Proxy, it functions as a proxy and requests the content directly from the orcharhino.

You can configure the global default download policies in Content settings.

Updating the default download policy by using orcharhino management UI

Set the default download policy so new repositories inherit immediate or on-demand behavior without updating existing repositories.

Depending on whether it is a Red Hat, SUSE, or repository, orcharhino uses separate settings. Changing the default value does not change the setting in existing repositories, but only the default setting for new repositories.

Procedure

In the orcharhino management UI, navigate to Administer > Settings.
Click the Content tab.
Change the default download policy depending on your requirements:
- To change the default download policy for a Red Hat repository, change the value of the Default Red Hat Repository download policy setting.
- To change the default download policy for a non-Red Hat repository, change the value of the Default Custom Repository download policy setting.

Updating the default download policy by using Hammer CLI

Set the default download policy so new repositories inherit immediate or on-demand behavior without updating existing repositories.

Procedure

To change the default download policy for Red Hat repositories to one of immediate or on_demand, enter the following command:
```
$ hammer settings set \
--name default_redhat_download_policy \
--value immediate
```
To change the default download policy for a repository to one of immediate or on_demand, enter the following command:
```
$ hammer settings set \
--name default_download_policy \
--value immediate
```

Updating the download policy for a repository by using orcharhino management UI

Change the download policy for an existing repository when you need immediate full downloads or on-demand storage.

Procedure

In the orcharhino management UI, navigate to Content > Products.
Select the required product name.
On the Repositories tab, click the required repository name, locate the Download Policy field, and click the edit icon.
From the list, select the required download policy and then click Save.

Updating the download policy for a repository by using Hammer CLI

Update the download policy for a specific repository when you need immediate full downloads or on-demand lazy synchronization for that repository.

Procedure

List the repositories for an organization:

$ hammer repository list \
--organization-label My_Organization_Label

Change the download policy for a repository to immediate or on_demand:

$ hammer repository update \
--download-policy immediate \
--name "My_Repository" \
--organization-label My_Organization_Label \
--product "My_Product"

Mirroring policies overview

Mirroring policies control whether synchronized repositories add only new content, mirror repodata, or fully mirror upstream changes including deletions. Use them to match upstream behavior and storage needs for each repository.

orcharhino Server has the following mirroring policies:

Additive

Neither the content nor the repodata is mirrored. Thus, only new content added since the last synchronization is added to the local repository and nothing is removed.

Content Only

Mirrors only content and not the repodata. Some repositories do not support metadata mirroring, in such cases you can set the mirroring policy to content only to only mirror the content.

Complete Mirroring

Mirrors content as well as repodata. This is the fastest method. This mirroring policy is only available for Yum content.

Avoid republishing metadata for repositories with Complete Mirror mirroring policy. This also applies to content views containing repositories with the Complete Mirror mirroring policy.

Updating the default mirroring policy by using orcharhino management UI

Set default mirroring policies for new repositories so they inherit additive, content-only, or complete mirroring without affecting existing repositories.

Depending on whether it is a Yum or non-Yum repository, orcharhino uses separate settings. Only Yum repositories support the mirror_complete mirroring policy. Red Hat repositories use the mirror_complete policy by default and are not affected by these settings. Changing the default value does not change existing mirroring policy settings per repository. For more information on mirroring policies, see Mirroring Policies Overview content-management.

Procedure

In the orcharhino management UI, navigate to Administer > Settings.
Click the Content tab.
Change the default mirroring policy depending on your requirements:
- To change the default mirroring policy for a Yum repository, change the value of the Default custom yum repository mirroring policy setting.
- To change the default mirroring policy for a non-Yum repository, change the value of the Default custom non-yum repository mirroring policy setting.

Updating the default mirroring policy by using Hammer CLI

Set default mirroring policies for new repositories so they inherit additive, content-only, or complete mirroring without affecting existing repositories.

Procedure

Change the default mirroring policy for a Yum repository to additive, mirror_content_only, or mirror_complete:
```
$ hammer settings set \
--name default_yum_mirroring_policy \
--value mirror_complete
```
Change the default mirroring policy for a non-Yum repository to additive or mirror_content_only:
```
$ hammer settings set \
--name default_non_yum_mirroring_policy \
--value additive
```

Updating the mirroring policy for a repository by using orcharhino management UI

Update the mirroring policy for an existing repository to control whether synchronization adds content only, mirrors repodata, or removes content deleted upstream.

Procedure

In the orcharhino management UI, navigate to Content > Products.
Select the product name.
On the Repositories tab, click the repository name, locate the Mirroring Policy field, and click the edit icon.
From the list, select a mirroring policy and click Save.

Updating the mirroring policy for a repository by using Hammer CLI

Update the mirroring policy for a specific repository to control whether synchronization adds content only, mirrors repodata, or removes content deleted upstream.

Procedure

List the repositories for an organization:

$ hammer repository list \
--organization-label My_Organization_Label

Change the mirroring policy for a repository to additive, mirror_complete, or mirror_content_only:
```
$ hammer repository update \
--id My_Repository_ID \
--mirroring-policy mirror_complete
```

Limiting synchronization concurrency

Adjust per-repository download concurrency when sync jobs fail because upstream servers reject parallel requests, or raise the limit cautiously to improve sync performance.

Increasing the limit may improve performance, but can cause the upstream server to be overloaded or start rejecting requests. If you are seeing repository syncs fail due to the upstream servers rejecting requests, you may want to try lowering the limit.

Procedure

Define a value for synchronization concurrency:

$ hammer repository update \
--download-concurrency 5 \
--id Repository_ID \
--organization "My_Organization"

Limiting growth of synchronized content by using orcharhino management UI

Limit how many package versions orcharhino keeps in the Library by setting an additive mirroring policy and a retain count, reducing disk growth from repeated synchronizations.

Procedure

In the orcharhino management UI, navigate to Content > Products.
Select your product.
On the Repository tab, select your repository.
Set the Mirroring Policy to Additive.
Define the number of retained package versions:
- Set Retain package versions to 1 to only keep the latest synchronized package version on orcharhino.
- Set Retain package versions to 2 or higher to keep multiple package versions on orcharhino.

Additional resources

Mirroring Policies Overview content-management

Limiting growth of synchronized content by using Hammer CLI

Limit how many package versions orcharhino keeps in the Library by setting an additive mirroring policy and a retain count, reducing disk growth from repeated synchronizations.

Procedure

Configure your repository to retain a limited number of package versions in orcharhino:
```
$ hammer repository update \
--id My_Repository_ID \
--mirroring-policy additive \
--retain-package-versions-count My_Package_Versions_Count
```
- Set My_Package_Versions_Count to 1 to only keep the latest synchronized package version on orcharhino.
- Set My_Package_Versions_Count to 2 or higher to keep multiple package versions on orcharhino.

Additional resources

Mirroring Policies Overview content-management

The text and illustrations on this page are licensed by ATIX AG under a Creative Commons Attribution Share Alike 4.0 International ("CC BY-SA 4.0") license. This page also contains text from the official Foreman documentation which uses the same license ("CC BY-SA 4.0").