Tuning orcharhino

Number of Puma threads (per Puma worker) is configured using two values: threads_min and threads_max.

Value of threads_min determines how many threads each worker spawns at a worker start. Then, as concurrent requests are coming and more threads is needed, worker will be spawning more and more workers up to threads_max limit.

We recommend setting threads_min to same value as threads_max as having fewer Puma threads lead to higher memory usage on your orcharhino Server.

For example, we have compared these two setups using concurrent registrations test:

Setting the minimum Puma threads to 16 results in about 12% less memory usage as compared to threads_min=0.

If you do not provide any Puma workers and thread values with orcharhino-installer or they are not present in your orcharhino configuration, the orcharhino-installer configures a balanced number of workers. It follows this formula:

This should be fine for most cases, but with some usage patterns tuning is needed to either limit the amount of resources dedicated to Puma (so other orcharhino components can use these) or for any other reason. Each Puma worker consumes around 1 GiB of RAM.

If you decide not to rely on Puma Workers and Threads Auto Tuning, you can apply custom numbers for these tunables. In the example below we are using 2 workers, 5 and 5 threads:

Apply your changes to orcharhino Server. For more information, see Applying configurations.

In order to recommend thread and worker configurations for the different tuning profiles, we conducted Puma tuning testing on orcharhino with different tuning profiles. The main test used in this testing was concurrent registration with the following combinations along with different number of workers and threads. Our recommendation is based purely on concurrent registration performance, so it might not reflect your exact use-case. For example, if your setup is very content oriented with lots of publishes and promotes, you might want to limit resources consumed by Puma in favor of Pulp and PostgreSQL.

Tuning number of workers is the more important aspect here and in some case we have seen up to 52% performance increase. Although installer uses 5 min/max threads by default, we recommend 16 threads with all the tuning profiles in the table above. That is because we have seen up to 23% performance increase with 16 threads (14% for 8 and 10% for 32) when compared to setup with 4 threads.

To figure out these recommendations we used concurrent registrations test case which is a very specific use-case. It can be different on your orcharhino which might have more balanced use-case (not only registrations). Keeping default 5 min/max threads is a good choice as well.

These are some of our measurements that lead us to these recommendations:

Use 4 – 6 workers on a default setup (4 CPUs) – we have seen about 25% higher performance with 5 workers when compared to 2 workers, but 8% lower performance with 8 workers when compared to 2 workers – see table below:

Use 8 – 12 workers on a medium setup (8 CPUs) – see table below:

Use 16 – 24 workers on a 32 CPUs setup (this was tested on a 90 GiB RAM machine and memory turned out to be a factor here as system started swapping – proper extra-large should have 128 GiB), higher number of workers was problematic for higher registration concurrency levels we tested, so we cannot recommend it.

If you have enough CPUs, adding more workers adds more performance. For example, we have compared orcharhino setups with 8 and 16 CPUs:

In 8 CPUs setup, changing the number of workers from 2 to 16, improved concurrent registration time by 36%. In 16 CPUs setup, the same change caused 55% improvement.

Adding more workers can also help with total registration concurrency orcharhino can handle. In our measurements, setup with 2 workers were able to handle up to 480 concurrent registrations, but adding more workers improved the situation.

More threads allow for lower time to register hosts in parallel. For example, we have compared these two setups:

Using more workers and the same total number of threads results in about 11% of speedup in highly concurrent registrations scenario. Moreover, adding more workers did not consume more CPU and RAM but gets more performance.

The effective value of $db_pool is automatically set to equal $foreman::foreman_service_puma_threads_max. It is the maximum of $foreman::db_pool and $foreman::foreman_service_puma_threads_max but both have default value 5, so any increase to the max threads above 5 automatically increases the database connection pool by the same amount.

If you encounter ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.006 seconds); all pooled connections were in use error in /var/log/foreman/production.log, you might want to increase this value.

If you decide not to rely on automatically configured value, you can apply custom number like this:

Apply your changes to orcharhino Server. For more information, see Applying configurations.

With the tuning in place, Apache httpd can easily open a lot of file descriptors on the server which may exceed the default limit of most of the Linux systems in place. To avoid any kind of issues that may arise as a result of exceeding max open files limit on the system, please create the following file and directory and set the contents of the file as specified in the below given example:

By default, httpd uses event request handling mechanism. When the number of requests to httpd exceeds the maximum number of child processes that can be launched to handle the incoming connections, httpd raises an HTTP 503 Service Unavailable error. Amidst httpd running out of processes to handle, the incoming connections can also result in multiple component failures on your orcharhino services side due to the dependency of some components on the availability of httpd processes.

You can adapt the configuration of httpd event to handle more concurrent requests based on your expected peak load.

You can tune the mosquitto MQTT server and increase the number of hosts connected to it.

When orcharhino Server is configured with the pull-based transport mode for remote execution jobs using the Script provider, orcharhino Proxy sends notifications about new jobs to clients through MQTT. This notification does not include the actual workload that the client is supposed to execute. After a client receives a notification about a new remote execution job, it queries orcharhino Proxy for its actual workload. During the job, the client periodically sends outputs of the job to orcharhino Proxy, further increasing the number of requests to orcharhino Proxy.

These requests to orcharhino Proxy together with high concurrency allowed by the MQTT protocol can cause exhaustion of available connections on orcharhino Proxy. Some requests might fail, making some child tasks of remote execution jobs unresponsive. This also depends on actual job workload, as some jobs are causing additional load to orcharhino Server, making it compete for resources if clients are registered to orcharhino Server.

To avoid this, configure your orcharhino Server and orcharhino Proxy with the following parameters:

orcharhino Proxy logs are in /var/log/foreman-proxy/proxy.log. orcharhino Proxy uses Webrick HTTP server (no httpd or Puma involved), so there is no simple way to increase its capacity.

PGbench utility (note you may need to resize PostgreSQL data directory /var/lib/pgsql directory to 100 GiB or what does benchmark take to run) might be used to measure PostgreSQL performance on your system. Use dnf install postgresql-contrib to install it.

Choice of filesystem for PostgreSQL data directory might matter as well.

We have measured multiple test cases on multiple orcharhino Proxy configurations:

In a download test where we concurrently downloaded a 40MB repo of 2000 packages on 100, 200, .. 1000 hosts, we saw roughly 50% improvement in average download duration every time when we doubled orcharhino Proxy resources. For more precise numbers, see the table below.

When we compared download performance from orcharhino Server vs. from orcharhino Proxy, we have seen only about 5% speedup, but that is expected as orcharhino Proxy’s main benefit is in getting content closer to geographically distributed clients (or clients in different networks) and in handling part of the load orcharhino Server would have to handle itself. In some smaller hardware configurations (8 CPUs and 24 GiB), orcharhino Server was not able to handle downloads from more than 500 concurrent clients, while a orcharhino Proxy with the same hardware configuration was able to service more than 1000 and possibly even more.

For concurrent registrations, a bottleneck is usually CPU speed, but all configs were able to handle even high concurrency without swapping. Hardware resources used for orcharhino Proxy have only minimal impact on registration performance. For example, orcharhino Proxy with 16 CPUs and 46 GiB RAM have at most a 9% registration speed improvement when compared to a orcharhino Proxy with 4 CPUs and 12 GiB RAM. During periods of very high concurrency, you might experience timeouts in the orcharhino Proxy to orcharhino Server communication. You can alleviate this by increasing the default timeout using the following tunable in /etc/foreman-installer/custom-hiera.yaml:

We have tested executing Remote Execution jobs via both SSH and Ansible backend on 500, 2000 and 4000 hosts. All configurations were able to handle all of the tests without errors, except for the smallest configuration (4 CPUs and 12 GiB memory) which failed to finish on all 4000 hosts.

In a sync test where we synced Enterprise Linux 6, 7, 8 BaseOS and 8 AppStream we have not seen significant differences among orcharhino Proxy configurations. This will be different for syncing a higher number of content views in parallel.