Log Files and Debug Levels

Various orcharhino components make use of various log files. When troubleshooting orcharhino these log files form an essential resource. Generally, orcharhino’s logging capabilities support various debug levels.

This section of the Troubleshooting chapter describes where log files are located, as well as how to enable the relevant debug levels. (You may be asked to do this by ATIX support to help us troubleshoot).

The easiest way to change the log level and enable debugging is by using the tool “Hammer admin logging” (see also: Hammer CLI Guide).

However, it’s still possible to enable debugging by changing the files. See sections below.

Note

To access log files and to change log levels we require access to the host orcharhino (or the relevant smart proxy) is running on. Unless stated otherwise, any commands mentioned in this section are intended for execution on the orcharhino host.

Note

Any manual changes to orcharhino configuration files (as are performed throughout this section), will be lost when running the foreman-installer for any reason. (For example during an orcharhino upgrade, see also Upgrade Guide).

Using the Hammer CLI to Set Log Levels

For changing the log level of certain orcharhino services, use the “Hammer admin” tool.

Note

Restarting of rsyslog and the underlying services needs to be done manually! See sections below for the appropriate service restart commands.

To view the Hammer admin logging usage instructions:

hammer admin logging --help

To list the components and the corresponding log files:

hammer admin logging --list

To enable debug logging for all components:

hammer admin logging --all

To enable debug logging for the pulp component:

hammer admin logging --components pulp --level-debug

After changing the logging, corresponding services need to be restarted:

systemctl restart rsyslog
foreman-maintain service restart --only pulp_celerybeat,pulp_resource_manager,\
  pulp_streamer,pulp_workers

To disable debug logging for the pulp component:

hammer admin logging --components pulp --level-production

Core Components

To enable debug level logging for core orcharhino components we perform the following steps:

  1. Amend the relevant configuration file.

    1. Ensure the following line exists in the settings.yaml file:

      /etc/foreman/settings.yaml
      
      :logging:
        :level: debug
      
    2. (Optional) Ensure additional loggers are enabled (true) in the settings.yaml file:

      /etc/foreman/settings.yaml
      
      :loggers:
        :ldap:
          :enabled: true
      

      Possible loggers are :ldap:, :app:, :audit:, :permissions:, :proxy:, :sql:, :templates:, :notifications:, :background:, and :dynflow:.

  2. Restart all orcharhino services (this may take a minute or two):

    foreman-maintain service restart
    
  3. Check the log file at /var/log/foreman/production.log (as well as other log files in the same directory).

Smart Proxies

Smart proxies (see also Smart Proxy Guide) are autonomous orcharhino components that may run on a separate host from the main orcharhino installation.

To enable debug level logging for smart proxies (either on the orcharhino host, or on some separate smart proxy host) perform the following steps:

  1. Ensure the following line is present in the smart proxy’s settings.yml file:

    /etc/foreman-proxy/settings.yml
    
    :log_level: DEBUG
    
    (Possible values are: DEBUG, WARN, Error, Fatal, INFO, and UNKNOWN)
    
  2. Restart the relevant services:

    foreman-maintain service restart --only foreman-proxy
    
  3. Check the log file at /var/log/foreman-proxy/proxy.log

Pulp

Pulp is used by orcharhino’s content management (see also Content Management Guide) to store content repositories.

To enable debug level logging for Pulp on the orcharhino host perform the following steps:

  1. Ensure the following line is present in the [server] block of the Pulp server.conf file:

    /etc/pulp/server.conf
    
    [server]
    log_level: DEBUG
    
    (Possible values are DEBUG, CRITICAL, ERROR, WARNING, and INFO)
    
  2. Restart the relevant services:

    foreman-maintain service restart --only pulp_celerybeat,pulp_resource_manager,\
      pulp_streamer,pulp_workers
    
  3. Check the log file at /var/log/pulp.log

Puppet

Puppet is the default configuration management tool in orcharhino. Even if an alternate tool (like Ansible or Salt) is used instead, Puppet is still used as part of orcharhino’s installation and upgrade processes. As such, log output for the Puppet agent during upgrades, or when installing additional plugins may be interesting for debug purposes.

To enable debug level logging for the Puppet agent on the orcharhino host perform the following steps:

  1. Ensure the following line is added to the [main] block in the puppet.conf file:

    /etc/puppetlabs/puppet/puppet.conf
    
    [main]
        log_level = debug
    
  2. Restart the relevant services:

    foreman-maintain service restart --only puppetserver
    
  3. Perform the action you want logged.

  4. Check the contents of the log folder at /var/log/puppetlabs/puppet/

It is also possible to adjust the log level of the Puppet server by performing the above steps for the [master] block (instead of [main]). The Puppet server’s log files are located in /vat/log/puppetlabs/puppetserver/.

Salt

For more information on Salt, see the Salt Guide.

You can change the log level on both Salt master (/etc/salt/master) and Salt minion (/etc/salt/minion) by changing the following option:

log_level: debug

The log files are stored in /var/log/salt/master and /var/log/salt/minion respectively.

Candlepin

Candlepin is used by orcharhino’s content management (see also Content Management Guide) to manage subscriptions (see also Subscriptions).

To enable debug level logging for Candlepin on the orcharhino host perform the following steps:

  1. Ensure the following line is present in the candlepin.conf file:

    /etc/candlepin/candlepin.conf
    
    log4j.logger.org.candlepin.servlet.filter=DEBUG
    
  2. Restart the relevant services:

    foreman-maintain service restart --only tomcat
    
  3. Check the log file at /var/log/candlepin/candlepin.log

Hammer

Hammer is a command line user interface for orcharhino.

To enable debug level logging for Hammer perform the following steps:

  1. Ensure the following line is present in Hammer’s cli_config.yml file:

    /etc/hammer/cli_config.yml
    
    :log_level: 'error'
    
  2. Perform any Hammer actions that require logging.

  3. Check the log file at ~/.hammer/log/hammer.log

Apache

The Apache server is used to host orcharhino’s web interface (see also Management UI). The following Apache related log files may be of interest:

/var/log/httpd/foreman_access.log
/var/log/httpd/foreman_error.log
/var/log/httpd/foreman-ssl_access_ssl.log
/var/log/httpd/foreman-ssl_error_ssl.log
/var/log/qdrouterd/qdrouterd.log