Tasks

Note

The documentation in this section of the Troubleshooting chapter, is intended to complement the documentation on Tasks from the Management UI chapter. To familiarize yourself with tasks it is recommended to start reading there. The subsubsection on Task Attributes is of particular interest when troubleshooting Tasks.

Many actions performed in orcharhino are performed as so called “tasks” (see also the Tasks page from the Management UI chapter). Tasks run asynchronously, can be scheduled to run at a specific time, and can perform background actions over an extended period of time.

Tasks are a feature of the foreman-tasks plugin (which is always included with orcharhino) and runs as a background service like any other (named foreman-tasks). (See also: Background Services)

Tasks are of particular importance for troubleshooting purposes, since they are used for the interaction between different system components. (Tasks allow for services to schedule actions performed by another service). Any number of things can go wrong with such interactions, and the foreman-tasks plugin provides inbuilt functionality for troubleshooting and recovery.

Background Tasks

There are two orcharhino tasks, that should always have a running instance. They are “Listen on Candlepin events” and “Monitor Event Queue”. Both are needed for orcharhino’s content management (see also Content Management Guide) to function correctly. (The former for subscription management and the latter for errata calculation).

To check on the “Listen on Candlepin events” and “Monitor Event Queue” tasks, you can filter (see also Filter Bars) the tasks page (see also Tasks) using the following filter keys:

  • label = Actions::Candlepin::ListenOnCandlepinEvents
  • label = Actions::Katello::EventQueue::Monitor

There may be many instances of both tasks, but exactly one of each should have the state = running, and result = pending.

Important

There is a known issue where the “Monitor Event Queue” task reports state = planning in the interface, even though it has progressed to state = running. If your task reports state = planning instead of state = running, the task is probably fine.

In other words, the following filter key should pick out exactly two tasks:

(label = Actions::Candlepin::ListenOnCandlepinEvents and state = running and result = pending) or (label = Actions::Katello::EventQueue::Monitor and result = pending)

If this is not the case, you can try restarting the foreman-tasks service. (See also Managing the services). If the problem persists you should contact ATIX support.

Resource Locks

Tasks may lock various resources they require access to. This mechanism exists to prevent multiple tasks from making changes to the same resource at the same time. There are two types of resource locks. Exclusive resource locks (indicated by a closed lock icon exclusive_lock) prevent other tasks from accessing the resource. (Another task attempting to lock the same resource will fail with a relevant error message). Non-exclusive resource locks (indicated by an open lock non_exclusive_lock) do not block access to the resource. They merely indicate a relation between task and resource.

When viewing a task (see also Viewing a Task) you can see any associated resource locks on the locks tab:

Viewing resource locks
  • Note how this example task blocks both read and write access to a specific product (see also: Products) with an exclusive resource lock (exclusive_lock).
  • The user, provider, and organization, are indicated with non-exclusive resource locks) (non_exclusive_lock)
  • Any other tasks trying to write to the same product at the same time will fail with an error result. If that task supports resuming (see Resuming a Paused Task below), you can do so once the lock is gone.

Tasks Reporting Errors

A failed task will report an error result. It will also change it’s state to stopped or paused. A stopped task with error result can not be resumed. If need be, investigate the errors encountered and then start a new task as appropriate. A paused task with error result may still be manually resumed. (See Resuming a Paused Task below).

In either case you should first investigate the encountered errors. When viewing a task (see also Viewing a Task) you can see any errors encountered on the errors tab:

Errors of a task
  • In this example we attempted to discover a repository of type docker where no docker repository is available (1), resulting in orcharhino requesting a resource that does not exist (404 Not Found) (2).
  • In this case we could try again selecting the correct repository type.
  • If the reported error is not illuminating, and is preventing an important task from succeeding it is probably best to contact ATIX support.

Tasks That Appear Unresponsive

In rare cases a task may appear to be stuck in a running state indefinitely. (Perhaps it is waiting for some service that has crashed). If you suspect a task of having stalled, you can start by checking its “running steps”.

When viewing a task (see also Viewing a Task) you can see its “running steps” on the running steps tab (note that the content on this tab is only available while a task is running):

Running steps of a task
  • In this case we have a bulk action task, that is still waiting for its child task to finish.
  • If we did have a task that was genuinely stuck we could try to restart the relevant background services (see also Background Services). Alternatively, we might simply cancel the task (see Cancelling a Task below).

Resuming a Paused Task

As mentioned already, some tasks support resuming after they have encountered an error. Finding such tasks can be easily achieved by filtering (see also Filter Bars) the tasks page (see also Tasks) using the filter key state = paused.

When viewing a paused task (see also Viewing a Task) a Resume button will be available as follows:

Resuming a paused task
  • Of course it only makes sense to resume the task if you have reason to believe it may now finish successfully. (Perhaps the task encountered a temporary network error before).
  • If you do resume a task, and it does complete successfully, it will change its state to stopped and its result to warning.

Cancelling a Task

Some tasks support manual cancellation. This may be important to free up the resource locks (see Resource Locks above) of a task that has stalled.

When viewing a task that supports manual cancellation (see also Viewing a Task) a Cancel button will be available as follows:

Cancelling a task

Note

It is up to a task to support manual cancellation. If the action is unavailable, there is nothing you can do about that.

Warning

Manually cancelling a task can potentially cause as inconsistent system state. Manually cancelling a task should only be used as a last resort (when you know what you are doing).