Tuning orcharhino

Number of Puma threads (per Puma worker) is configured using two values: threads_min and threads_max.

Value of threads_min determines how many threads each worker spawns at a worker start. Then, as concurrent requests are coming and more threads is needed, worker will be spawning more and more workers up to threads_max limit.

We recommend setting threads_min to same value as threads_max as having fewer Puma threads lead to higher memory usage on your orcharhino Server.

For example, we have compared these two setups using concurrent registrations test:

Setting the minimum Puma threads to 16 results in about 12% less memory usage as compared to threads_min=0.

If you do not provide any Puma workers and thread values with orcharhino-installer or they are not present in your orcharhino configuration, the orcharhino-installer configures a balanced number of workers. It follows this formula:

This should be fine for most cases, but with some usage patterns tuning is needed to either limit the amount of resources dedicated to Puma (so other orcharhino components can use these) or for any other reason. Each Puma worker consumes around 1 GiB of RAM.

If you decide not to rely on Puma Workers and Threads Auto Tuning, you can apply custom numbers for these tunables. In the example below we are using 2 workers, 5 and 5 threads:

Apply your changes to orcharhino Server. For more information, see Applying configurations.

The effective value of $db_pool is automatically set to equal $foreman::foreman_service_puma_threads_max. It is the maximum of $foreman::db_pool and $foreman::foreman_service_puma_threads_max but both have default value 5, so any increase to the max threads above 5 automatically increases the database connection pool by the same amount.

If you encounter ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.006 seconds); all pooled connections were in use error in /var/log/foreman/production.log, you might want to increase this value.

If you decide not to rely on automatically configured value, you can apply custom number like this:

Apply your changes to orcharhino Server. For more information, see Applying configurations.

With the tuning in place, Apache httpd can easily open a lot of file descriptors on the server which may exceed the default limit of most of the Linux systems in place. To avoid any kind of issues that may arise as a result of exceeding max open files limit on the system, please create the following file and directory and set the contents of the file as specified in the below given example:

By default, httpd uses event request handling mechanism. When the number of requests to httpd exceeds the maximum number of child processes that can be launched to handle the incoming connections, httpd raises an HTTP 503 Service Unavailable error. Amidst httpd running out of processes to handle, the incoming connections can also result in multiple component failures on your orcharhino services side due to the dependency of some components on the availability of httpd processes.

You can adapt the configuration of httpd event to handle more concurrent requests based on your expected peak load.

In deployments using katello-agent infrastructure with a large number of content hosts, it may be necessary to increase the maximum open files for qdrouterd.

Calculate the limit for open files in qdrouterd using this formula: (N x 3) + 100, where N is the number of content hosts. Each content host may consume up to three file descriptors in the router and 100 file descriptors are required to run the router itself.

The following settings permit orcharhino to scale up to 10.000 content hosts.

In deployments using katello-agent infrastructure with a large number of content hosts, it may be necessary to increase the maximum open files for qpidd.

Calculate the limit for open files in qpidd using this formula: (N x 4) + 500, where N is the number of content hosts. A single content host can consume up to four file descriptors and 500 file descriptors are required for the operations of Broker (a component of qpidd).

In deployments using katello-agent infrastructure with a large number of content hosts, it may be necessary to increase the maximum allowable concurrent AIO requests. You can increase the maximum number of allowed concurrent AIO requests by increasing the kernel parameter fs.aio-max-nr.

Ensure you provide enough storage space for /var/lib/qpidd in advance when you are planning an installation that will use katello-agent extensively. On orcharhino Server, /var/lib/qpidd requires 2MiB disk space per content host.

You might see the following error in journal (use journalctl command to access it) in Enterprise Linux 7:

This error message appears because qpid maintains management objects for queues, sessions, and connections and recycles them every ten seconds by default. The same object with the same ID is created, deleted, and created again. The old management object is not yet purged, which is why qpid throws this error.

PGbench utility (note you may need to resize PostgreSQL data directory /var/lib/pgsql directory to 100 GiB or what does benchmark take to run) might be used to measure PostgreSQL performance on your system. Use dnf install postgresql-contrib to install it.

Choice of filesystem for PostgreSQL data directory might matter as well.

We have measured multiple test cases on multiple orcharhino Proxy configurations:

In a download test where we concurrently downloaded a 40MB repo of 2000 packages on 100, 200, .. 1000 hosts, we saw roughly 50% improvement in average download duration every time when we doubled orcharhino Proxy resources. For more precise numbers, see the table below.

When we compared download performance from orcharhino Server vs. from orcharhino Proxy, we have seen only about 5% speedup, but that is expected as orcharhino Proxy’s main benefit is in getting content closer to geographically distributed clients (or clients in different networks) and in handling part of the load orcharhino Server would have to handle itself. In some smaller hardware configurations (8 CPUs and 24 GiB), orcharhino Server was not able to handle downloads from more than 500 concurrent clients, while a orcharhino Proxy with the same hardware configuration was able to service more than 1000 and possibly even more.

For concurrent registrations, a bottleneck is usually CPU speed, but all configs were able to handle even high concurrency without swapping. Hardware resources used for orcharhino Proxy have only minimal impact on registration performance. For example, orcharhino Proxy with 16 CPUs and 46 GiB RAM have at most a 9% registration speed improvement when compared to a orcharhino Proxy with 4 CPUs and 12 GiB RAM. During periods of very high concurrency, you might experience timeouts in the orcharhino Proxy to orcharhino Server communication. You can alleviate this by increasing the default timeout using the following tunable in /etc/foreman-installer/custom-hiera.yaml:

+

We have tested executing Remote Execution jobs via both SSH and Ansible backend on 500, 2000 and 4000 hosts. All configurations were able to handle all of the tests without errors, except for the smallest configuration (4 CPUs and 12 GiB memory) which failed to finish on all 4000 hosts.

In a sync test where we synced Enterprise Linux 6, 7, 8 BaseOS and 8 AppStream we have not seen significant differences among orcharhino Proxy configurations. This will be different for syncing a higher number of content views in parallel.