Plugins

Restraint relies on plugins to execute tasks in the correct environment and to check for common errors or simply to provide additional logs for debugging issues. Here is a typical outline of how plugins are executed:

run_task_plugins
 \
  10_bash_login
  |
  15_beakerlib
  |
  20_unconfined
  |
  25_environment
  |
  make run
  |\
  | report_result
   \
    report_result

run_task_plugins
 \
  10_bash_login
  |
  15_beakerlib
  |
  20_unconfined
  |
  25_environment
  |
  run_plugins <- completed.d
  \
   98_restore

The report_result commands above cause the following plugins to be executed:

run_task_plugins
 \
  05_linger
  |
  10_bash_login
  |
  15_beakerlib
  |
  20_unconfined
  |
  25_environment
  |
  30_restore_events
  |
  35_oom_adj
  |
  run_plugins <- report_result.d
   \
    01_dmesg_check
    |
    10_avc_check
    |
    20_avc_clear
    |
    30_dmesg_clear

These plugins do not run from the task under test. They run from restraintd process. This allows for greater flexibility if your task is running as a non-root user since a non-root user would not be able to inspect some logs and wouldn’t be able to clear dmesg log.

Task Run

Task run plugins are used to modify the environment under which the tasks will execute. Simply place the executable in /usr/share/restraint/task_run.d. The list of files in this directory will be passed to exec in alphabetical order.

Restraint currently ships with two task run plugins:

  • 05_linger - Enables session bus for user that Restraint is running as. You can disable this with RSTRNT_DISABLE_LINGER=1
  • 10_bash_login - invoke a login shell.
  • 15_beakerlib - Sets env vars to tell beakerlib how to report results in Restraint.
  • 20_unconfined - if selinux is enabled on system run task in unconfined context.
  • 25_environment - Will attempt to guess certain variables if they weren’t defined, (OSARCH, OSMAJOR, etc..).
  • 30_restore_events - Restores Multi-host states after a reboot.
  • 35_oom_adj - sets the oom score low so we are less likely to be killed.

So the above plugins would get called like so:

exec 05_linger 10_bash_login 15_beakerlib 20_unconfined 25_environment 30_restore_events 35_oom_adj "$@"

In order for this to work the task run plugins are required to exec “$@” at the end of the script. Although task run plugins can’t take any arguments they can make decisions based on environment variables.

It should be pointed out that the task run plugins are executed for all other plugins! This is to ensure plugins run with the same environment as your task. When executed under all other plugins the following variable will be defined:

RSTRNT_NOPLUGINS=1

You can do conditionals based on this so lets create a plugin which will start a TCP capture:

# Capture tcpdump data from every task
cat << "EOF" > /usr/share/restraint/plugins/task_run.d/30_tcpdump
#!/bin/sh -x

rstrnt_info "*** Running Plugin: $0"

# Don't run from PLUGINS
if [ -z "$RSTRNT_NOPLUGINS" ]; then
  tcpdump -q -i any -q -w $RUNPATH/tcpdump.cap 2>&1 &
  echo $! > $RUNPATH/tcpdump.pid
fi

exec "$@"
EOF
chmod a+x /usr/share/restraint/plugins/task_run.d/30_tcpdump

Refer to section (Completed) for how to report these results.

Report Result

Every time a task reports a result to Restraint these plugins will execute.

  • 01_dmesg_check - This plugin checks dmesg output for lines containing certain values and also allows lines to be omitted. If any lines are selected, this indicates an error so the task will conclude with failed results.
  • 30_dmesg_clear - This plugin clears dmesg log so the next task will start with a fresh log.

There are 2 variables which manage selection of dmesg output. They are FAILURESTRINGS and FALSESTRINGS. The FAILURESTRINGS variable contain values which allow you to select those lines considered in error. The FALSESTRINGS variable contain values allowing you to omit some lines. This enables you to omit false positives.

There are 3 ways FAILURESTRINGS and FALSESTRINGS configuration are provided. They can be configured by way of environment variables, as files, or defaults. The order of precedence for these variables/files is follows:

1) Task environment variable
2) User defined files
3) and defaults.

FAILURESTRINGS and FALSESTRINGS are processed separately so you could define FAILURESTRINGS as an environment variable while maintaining defaults for FALSESTRINGS.

The default values for FAILURESTRINGS are as follows:

Oops|BUG|NMI appears to be stuck|Badness at

The default values for FALSESTRINGS are as follows:

BIOS BUG|DEBUG|mapping multiple BARs.*IBM System X3250 M4

Both of the above strings can be overridden for each task by passing in your own FAILURESTRINGS or FALSESTRINGS environment variables. This is configured for each task. To define environment variables, refer to instructions for metadata or testinfo.desc files in (see Tasks).

If you want all tasks in a recipe to use the same set of your user-defined FAILURESTRINGS or FALSESTRINGS, you could start your recipe with a task which creates the following files respectively:

/usr/share/rhts/failurestrings
/usr/share/rhts/falsestrings

When configuring these files, each string should be on a separate line instead of separated with ‘|’. For example, failurestrings would contain something like the following:

Oops
BUG
NMI appears to be stuck
Badness at

In some cases, the kernel will produce a multi-line error message (including hardware information and stack trace) in the dmesg output which is delimited by a “cut here” line at the beginning and an “end trace” line at the end. This plugin will capture the entire contents of the multi-line trace and considers it as a single failure. The FALSESTRINGS pattern is applied to the whole trace to check for false positives.

  • 10_avc_check - This plugin searches for AVC (Access Vector Cache) errors that have occurred since the last time a result was reported.
  • 20_avc_clear - This moves the time stamp used by avc_check forward so that we don’t see the same AVC’s reported again, some tests might generate AVC’s on purpose and disable the check but you will still want to move the time stamp forward.

If you need to skip error checking, refer to RSTRNT_DISABLED as described in the Task Environment Variables section.

Local Watchdog

These plugins will only be executed if the task runs beyond its expected time limit. Restraint currently ships with three plugins:

  • 10_localwatchdog - uploads the resultoutputfile.log of the running task.
  • 20_sysinfo - Collects and uploads system information.
    • Uploads system log which contains a collection of system information such as slabinfo, list of blocked tasks derived from sysrq m, t and w, and pre-existing system log messages. Depending if journalctl exists, file journalctl or /var/log/messages is uploaded.
    • Uploads ps-lwd.log which contains a verbose list of running processes.
    • Uploads dmesg log if it contains any output.
    • Uploads user logs listed in $TESTPATH/logs2get.
  • 99_reboot - Simply reboots the system to try and get the system back to a sane state. If you need to skip this step, you can use RSTRNT_DISABLED as described in (see Task Environment Variables).

Completed

These plugins will get executed at the end of every task, regardless if the localwatchdog triggered or not. The only plugin currently shipped with Restraint is:

  • 85_sync_multihost_tasks - Synchronizes tasks between client/server jobs on multihost machines. This will synchronize only if there exists recipes with role=SERVERS as well as role=CLIENTS. For further details on this feature, refer to Beaker Multihost documentation [1].
  • 97_audit_rotate - Searches log files in audit directory to find avc messages.
  • 98_restore - Restores files backed up by either rhts-backup or rstrnt-backup.

To finish our tcpdump example from above we can add the following:

#Kill tcpdump and upload
cat << "EOF" > /usr/share/restraint/plugins/completed.d/80_upload_tcpdump
#!/bin/sh -x

kill $(cat $RUNPATH/tcpdump.pid)
rstrnt-report-log -l $RUNPATH/tcpdump.cap
EOF
chmod a+x /usr/share/restraint/plugins/completed.d/80_upload_tcpdump

If you need to skip file restoration, refer to RSTRNT_DISABLED as described in the environment variable section (see Task Environment Variables).

[1]Beaker Multihost documentation.