Log Files and Debug Levels

Various orcharhino components make use of various log files. These log files form an essential resource when troubleshooting orcharhino. Generally, orcharhino’s logging capabilities support various debug levels.

This section of the troubleshooting chapter describes where log files are located as well as how to enable the relevant debug levels. You may be asked to do this by ATIX Support to help us troubleshoot.

The easiest way to change the log level and enable debugging is by using the Hammer CLI tool. However, you can still manually enable debugging by changing the files as described in the sections below.

We require access to the host on which the orcharhino or the relevant orcharhino proxy is running on in order to access any log files and to change log levels. Unless stated otherwise, any commands mentioned in this section are intended to be executed on the orcharhino.

Any manual changes to orcharhino configuration files (as are performed throughout this section) will be lost when running the foreman-installer for any reason, e.g. during an orcharhino upgrade.

Using the Hammer CLI to Set Log Levels

Use the Hammer admin tool to change the log level of certain orcharhino services.

Restarting of rsyslog and the underlying services needs to be done manually. See sections below for the appropriate service restart commands.

To view the Hammer admin logging usage instructions:

# hammer admin logging --help

To list the components and the corresponding log files:

# hammer admin logging --list

To enable debug logging for all components:

# hammer admin logging --all --level-debug

To enable debug logging for the Pulp component:

# hammer admin logging --components pulp --level-debug

After changing the logging, corresponding services need to be restarted:

# systemctl restart rsyslog
# foreman-maintain service restart --only pulp_celerybeat,pulp_resource_manager,pulp_streamer,pulp_workers

To disable debug logging for the Pulp component:

# hammer admin logging --components pulp --level-production

Core Components

To enable debug level logging for core orcharhino components we perform the following steps:

  1. Amend the relevant configuration file.

    1. Ensure the following line exists in the settings.yaml file:

      /etc/foreman/settings.yaml

:logging:
  :level: debug

    2. Optional: Ensure additional loggers are enabled (true) in the settings.yaml file:

      /etc/foreman/settings.yaml

:loggers:
  :ldap:
    :enabled: true

      Possible loggers are :ldap:, :app:, :audit:, :permissions:, :proxy:, :sql:, :templates:, :notifications:, :background:, and :dynflow:.

  2. Restart all orcharhino services (this may take a minute or two):

    # foreman-maintain service restart

  3. Check the log file at /var/log/foreman/production.log (as well as other log files in the same directory).

Smart Proxy Functionalities

Both the orcharhino as well as any orcharhino proxy run various smart proxy functionalities.

Perform the following steps to enable debug level logging for smart proxy functionalities (either on the orcharhino or any attached orcharhino proxy):

  1. Ensure the following line is present in the orcharhino proxy’s settings.yml file:

    /etc/foreman-proxy/settings.yml

:log_level: DEBUG

    Possible values are DEBUG, WARN, Error, Fatal, INFO, or UNKNOWN.

  2. Restart the relevant services:

    # foreman-maintain service restart --only foreman-proxy

  3. Check the log file at /var/log/foreman-proxy/proxy.log

Pulp

Pulp is used by orcharhino’s content management to store content repositories.

To enable debug level logging for Pulp on the orcharhino host perform the following steps:

  1. Ensure the following line is present in the [server] block of the Pulp server.conf file:

    /etc/pulp/server.conf

[server]
log_level: DEBUG

(Possible values are DEBUG, CRITICAL, ERROR, WARNING, and INFO)

  2. Restart the relevant services:

    # foreman-maintain service restart --only pulp_celerybeat,pulp_resource_manager,\
    pulp_streamer,pulp_workers

  3. Check the log file at /var/log/pulp.log.

Puppet

Puppet is the default configuration management tool in orcharhino. Even if an alternate tool (like Ansible or Salt) is used instead, Puppet is still used as part of orcharhino’s installation and upgrade processes. As such, log output for the Puppet agent during upgrades, or when installing additional plugins may be interesting for debug purposes.

To enable debug level logging for the Puppet agent on the orcharhino host perform the following steps:

  1. Ensure the following line is added to the [main] block in the puppet.conf file:

    /etc/puppetlabs/puppet/puppet.conf

[main]
    log_level = debug

  2. Restart the relevant services:

    # foreman-maintain service restart --only puppetserver

  3. Perform the action you want logged.

  4. Check the contents of the log folder at /var/log/puppetlabs/puppet/

It is also possible to adjust the log level of the Puppet server by performing the above steps for the [master] block (instead of [main]). The Puppet server’s log files are located in /vat/log/puppetlabs/puppetserver/.

Salt

You can change the log level on both Salt Master (/etc/salt/master) and Salt Minion (/etc/salt/minion) by changing the following option:

log_level: debug

Salt Master logs to /var/log/salt/master and Salt Minions log to /var/log/salt/minion. Refer to the Salt guide for more information.

Candlepin

Candlepin is used by orcharhino’s content management to manage subscriptions.

To enable debug level logging for Candlepin on the orcharhino host perform the following steps:

  1. Ensure the following line is present in the candlepin.conf file:

    /etc/candlepin/candlepin.conf

log4j.logger.org.candlepin.servlet.filter=DEBUG

  2. Restart the relevant services:

    # foreman-maintain service restart --only tomcat

  3. Check the log file at /var/log/candlepin/candlepin.log

Hammer

Hammer is a command line user interface for orcharhino.

To enable debug level logging for Hammer perform the following steps:

  1. Ensure the following line is present in Hammer’s cli_config.yml file:

    /etc/hammer/cli_config.yml

:log_level: 'error'

  2. Perform any Hammer actions that require logging.

  3. Check the log file at ~/.hammer/log/hammer.log

Apache

The Apache server is used to host orcharhino’s web interface. The following Apache related log files may be of interest:

/var/log/httpd/foreman_access.log
/var/log/httpd/foreman_error.log
/var/log/httpd/foreman-ssl_access_ssl.log
/var/log/httpd/foreman-ssl_error_ssl.log
/var/log/qdrouterd/qdrouterd.log

Foreman Maintain

The log file of foreman-maintain is located at /var/log/foreman-maintain/foreman-maintain.log.