Upgrading orcharhino Server

This guide describes how you can upgrade orcharhino to a newer version. See the orcharhino Release Notes for a high level overview of available versions and their features.

ATIX provides guided upgrades performed by our consultants as part of our support subscriptions. Please contact us if you would like to make use of this service.

Ensure you carefully read all of the instructions, warnings, and recommendations presented in this guide and the appropriate version specific orcharhino Upgrade Notes in the ATIX Service Portal. ATIX does not offer support for recovery from a failed upgrade if you did not follow our upgrade guide.

ATIX does not support using third party repositories on your orcharhino Server or orcharhino Proxies. Resolving package conflicts or other issues due to third party or custom repositories is not part of your orcharhino support subscription. Please contact us if you have any questions.

Upgrading to orcharhino <= 5.12

Procedure

Check the version specific upgrade steps for the version you want to upgrade to first.
Backup your orcharhino instance:

We generally recommend running orcharhino as a virtual server and performing a system snapshot at this point. For more information, see backing up orcharhino.

Update and list the versions known to your orcharhino Server:

# yum update orcharhino-maintain-definitions
# foreman-maintain upgrade list-versions

Perform the upgrade itself:

# foreman-maintain upgrade run --target-version My_Target_Version

My_Target_Version must be one of the versions obtained from step 3.

If you are running an offline installation, use EXTERNAL_OR_SKIP_REPO_SETUP=1 foreman-maintain upgrade run --target-version My_Target_Version instead. In this case, you must provide all repositories for the version being upgraded to through a local mirror.

Navigate to Administer > About and check the Backend System Status. Ensure the box lists each component with status OK.

You now have a new orcharhino version. Enjoy!

Once you have upgraded successfully, it may be useful to create another backup/snapshot.

The Content Migration on orcharhino 5.12

orcharhino 5.12 is the last orcharhino Y-release with Pulp2. orcharhino 6.0 exclusively uses Pulp3, which replaces MongoDB with PostgreSQL.

You must perform a one time content migration that creates Pulp3 content in PostgreSQL based on your existing Pulp2 content within MongoDB, before upgrading to orcharhino 6.0. The following subsections contain all the information you need to plan and perform the content migration.

Quick Facts

You must run the content migration on orcharhino 5.12 before you can upgrade to orcharhino 6.0. It can take many hours or even days to complete.
You can run the content migration multiple times and each new run will continue where the last one stopped.
You can reset the content migration progress and start over from scratch.
orcharhino only starts using the migrated content after the upgrade to orcharhino 6.0.
You can continue with normal orcharhino 5.12 usage in parallel to any content migration runs.

You should expect high load on your orcharhino Server while running the content migration.
Unless you ignore the disk space requirements, there is no way for a failed content migration to negatively impact your orcharhino 5.12 installation.
The upgrade to orcharhino 6.0 includes a final content migration run in order to migrate any content you may have added to orcharhino after your final manual content migration run.

Hints and Best Practices

It is highly recommended to use the latest available version of orcharhino 5.12 for your content migration. To upgrade to the latest available version of orcharhino 5.12, run the following commands:
```
# yum update
# foreman-installer
```
You can delete old content view versions or unused repositories prior to starting your content migration. By reducing the amount of content that needs to be migrated, you will speed up the content migration process and reduce its disk usage. After you have removed content views, ensure to remove orphaned content:
```
# foreman-rake katello:delete_orphaned_content
```
If your operating environment permits it, we recommend that you temporarily disable all sync plans before starting the content migration in order to avoid generating new content that has not yet been migrated. Since there is a final content migration run during the upgrade to orcharhino 6.0, this is not strictly necessary, but it reduces the scope for performance bottlenecks during the content migration, as well as the scope for errors during the upgrade to orcharhino 6.0.
If you want to see the output from the content migration, we recommend using a tmux or screen session because of how long a content migration run can take to complete. Each content migration run launches an orcharhino background task and is therefor independent of the shell used to launch it. If the launching shell loses its connection to the background task, you can still look at the task itself within the management UI.

If you see an error or warning during a content migration run, open an ATIX support ticket. This is true even if subsequent content migration runs finish successfully.

A failed content migration may propose placing certain error conditions in a whitelist to continue. This should never be done as a first resort! Whitelisting content migration errors can create the appearance of a successful content migration, while actually resulting in an inconsistent database state after the upgrade to orcharhino 6.0. Always open an ATIX support ticket if you encounter errors during a content migration run.

Disk Space Requirements

The content migration does not duplicate your actual content, like .rpm or .deb packages. Instead, it uses hard links for migrated files. However, some meta data needs to be present in both Pulp2 and Pulp3 until you upgrade to orcharhino 6.0. Ensure your orcharhino Server meets the following disk space requirements before starting your first content migration run:

80% of the space consumed by /var/lib/mongodb/ must be available in /var/opt/rh/rh-postgresql12/lib/pgsql/. For example, if /var/lib/mongodb/ uses 100 GiB, another 80 GiB must be available for PostgreSQL.
100% of the space consumed by /var/lib/pulp/published/ must be available in /var/lib/pulp/. For example, if /var/lib/pulp/published/ uses 100 GiB, another 100 GiB must be available.

Running the Content Migration on orcharhino 5.12

Prerequisite

Ensure to carefully read the Quick Facts and Hints and Best Practices
Ensure your orcharhino Server meets the Disk Space Requirements.
Plan for the time needed for the content migration, as appropriate for your operating environment.

Procedure

Optional: Check the content migration stats:
```
# foreman-maintain content migration-stats
```
Prepare existing content in /var/lib/pulp for hardlinking to prevent duplicating all Pulp2 content during the content migration:
```
# foreman-maintain prep-6-upgrade
```

Run content-migrations:

# foreman-maintain content prepare

The foreman-maintain content prepare command starts and regularly polls an orcharhino task. If this polling connection is interrupted, the command appears to be frozen. In such cases, you can continue to check the actual task on your orcharhino, which may well complete successfully.

For more information, see Migration Task Appears Stuck.

Optional: Check the content migration stats again:
```
# foreman-maintain content migration-stats
```

Pausing or Resetting the Content Migration

We recommend that you allow any content migration runs to run to completion. However, because this can take many hours or even days, you can abort a content migration run:

# foreman-maintain content prepare-abort

You can continue the content migration later:

# foreman-maintain content prepare

It is also possible to completely drop all migration progress and start over with an empty Pulp3 database. If you think you need to do a content migration reset, please contact ATIX support!

Upgrading to orcharhino 6.0

orcharhino 6.0 is based on Foreman 2.5, Katello 4.1 and exclusively uses Pulp3:

Table 1. orcharhino component versions
orcharhino	Foreman	Katello	Pulp2	Pulp3
5.12	2.3	3.18.4	2.21.5	3.7
6.0	2.5	4.1.1	-	3.14

Prerequisites

Ensure that all content has been successfully migrated on orcharhino 5.12. For more information, see Running the Content Migration on orcharhino 5.12.
Ensure that you have upgraded all orcharhino Proxies to at least 5.12.1 because older orcharhino Proxies will not work with your orcharhino Server 6.0.

Procedure

Check the version specific upgrade steps for orcharhino 6.0.
Backup your orcharhino instance:

We generally recommend running orcharhino as a virtual server and performing a system snapshot at this point. For more information, see backing up orcharhino.
Update and list the versions known to your orcharhino Server:
```
# yum update orcharhino-maintain-definitions
```

Upgrade your orcharhino Server to 6.0:

# foreman-maintain upgrade run --target-version 6.0

Navigate to Administer > About and check the Backend System Status. Ensure the box lists each component with status OK.
Remove all Pulp2 content within MongoDB to free up disk space:
```
# foreman-maintain content remove-pulp2
```

Upgrading orcharhino Proxies to 6.0

You can upgrade your orcharhino Proxies after you have successfully upgraded your orcharhino Server. For more information, see Upgrading orcharhino Proxies to 6.0.

Troubleshooting

Setting the Log Level for the Migration

You can find log messages using journalctl.

Procedure

Enable DEBUG level logging in /etc/pulp/settings.py:

LOGGING = {"dynaconf_merge": True, "loggers": {'': {'handlers': ['console'], 'level': 'DEBUG'}}}

View all relevant content migration logs:

# journalctl -u 'pulpcore-worker*' -f | grep -v 'pulpcore.tasking.services.worker_watcher'

Accessing the Pulp3 API

We deliver a script to access the Pulp3 API.

Procedure

List all tasks in Pulp3:
```
pulp_api.sh GET /tasks/
```
Note that this output is paginated.
List details of a specific task. Replace the href with the output from an action, for example when synchronizing a repository:
```
pulp_api.sh GET /pulp/api/v3/tasks/href/
```

Synchronize a repository:

pulp_api.sh POST /pulp/api/v3/repositories/deb/apt/REPO_href/sync/ remote=/pulp/api/v3/remotes/deb/apt/REMOTE_href/

The Pulp3 REST API only works if the path ends with a /. If you omit the last slash, the API request will not work:

OK: pulp_api.sh GET /tasks/
FAILURE: pulp_api.sh GET /tasks

Starting the Content Migration From the Beginning

On orcharhino 5.12, you can drop all progress of the content migration:

# foreman-maintain content migration-reset

Migration Task Appears Stuck

While running foreman-maintain content prepare, the current state of the content migration is displayed in the console. This information is provided by the Content Migration task, which periodically polls the Pulp3 task executing the content migration. It may happen that this task stops to poll the Pulp3 task. This results in the state no longer being updated and the content migration task appearing to be stuck while Pulp3 still runs the content migration.

Indication

The output of foreman-maintain content prepare no longer changes:

Starting task.
2022-03-08 11:42:12 +0100: Processing Pulp 2 repositories, importers, distributors 5/77
2022-03-08 15:25:19 +0100: Migrating deb content to Pulp 3 11582/62105
2022-03-08 16:32:16 +0100: Migrating deb content to Pulp 3 11582/62105
2022-03-08 17:43:40 +0100: Migrating deb content to Pulp 3 11582/62105
2022-03-09 11:14:45 +0100: Migrating deb content to Pulp 3 11582/62105

Dynflow console shows the Actions::Pulp3::ContentMigration action running, but never updates the state of the Pulp3 task.

Logs of pulpcore-api will show no more GET requests to /pulp/api/v3/tasks/<task UUID>/ and /pulp/api/v3/task-groups/<task-group UUID>/ from OpenAPI-Generator/3.7.1/ruby.

Run journalctl -u pulpcore-api:

[08/Dec/2021:11:17:16 +0000] "GET /pulp/api/v3/tasks/TASK_ID/ HTTP/1.1" 200 1746 "-" "OpenAPI-Generator/3.7.1/ruby"
[08/Dec/2021:11:17:16 +0000] "GET /pulp/api/v3/task-groups/TASK_ID/ HTTP/1.1" 200 440 "-" "OpenAPI-Generator/3.7.1/ruby"
[08/Dec/2021:11:17:32 +0000] "GET /pulp/api/v3/tasks/TASK_ID/ HTTP/1.1" 200 1746 "-" "OpenAPI-Generator/3.7.1/ruby"
[08/Dec/2021:11:17:33 +0000] "GET /pulp/api/v3/task-groups/TASK_ID/ HTTP/1.1" 200 440 "-" "OpenAPI-Generator/3.7.1/ruby

Restarting dynflow daemons

Ensure that no other tasks apart from Content Migration are currently running.
Restart the dynflow services to update the hanging task and to poll the state of the Pulp3 task again.
```
# systemctl restart 'dynflow-sidekiq@*'
```

Upgrading Puppet

You can use foreman-maintain to upgrade Puppet. This check is also part of the orcharhino pre-upgrade checks.

Procedure

Connect to your orcharhino Server using SSH:
```
# ssh root@orcharhino.example.com
```
Run the Puppet upgrade check:
```
# foreman-maintain health check --label orcharhino-puppet-support
```
foreman-maintain checks if your current Puppet version is supported on your orcharhino instance. If an upgrade is available, you can follow the instructions to either abort or start the Puppet upgrade process.