BSDwiki/ Review log files to troubleshoot and monitor system behavior

Review log files to troubleshoot and monitor system behavior

Author: Cezary Morga cm@therek.net FreeBSD

Reviewer: name contact BSD flavour

Reviewer: name contact BSD flavour

Concept

Be aware of the importance of reviewing log files on a regular basis as well as how to watch a log file when troubleshooting. Be able to view compressed(((compression))) logs.

Introduction

The review and monitoring of log files can help maintain the health of a system. The tools like dmesg(8), tail(1) and grep(1) all help the administrator to troubleshoot problems. What and how a system logs is controlled by the syslogd(8) program, the amount and verbosity of logging is configured in the syslog.conf file (see Configure system logging). As log files are often rotated and compressed regularly by the system, tools such as zmore(1) and bzcat(1) become useful.

The default directory where the log files are stored is /var/log/. In some situations, i.e. in chrooted environment (see Recognize the BSD methods for restraining a service), log files can also be located elsewhere within the system.

Examples

The dmesg(8) utility displays the contents of the system message buffer. By default, the buffer is read from the currently running kernel. File /var/run/dmesg.boot is a copy of the buffer content taken soon after system boot.

# dmesg
Copyright (c) 1992-2006 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
       The Regents of the University of California. All rights reserved.
FreeBSD 6.1-RELEASE #0: Sun May  7 04:32:43 UTC 2006
   root@opus.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC

<snipped>

ad0: 19092MB  at ata0-master UDMA33
acd0: CDROM  at ata1-master PIO4
Trying to mount root from ufs:/dev/ad0s1a
fxp0: promiscuous mode enabled
fxp0: promiscuous mode disabled
fxp0: link state changed to DOWN
fxp0: link state changed to UP

The tail(1) utility displays the last part of a file. When typed without any additional flags tail displays ten last lines. This default behaviour can be modified by adding -n option with number of lines to be displayed:

# tail -n3 /var/log/cron
Feb 13 22:55:00 ns1 /usr/sbin/cron[90089]: (operator) CMD (/usr/libexec/save-entropy)
Feb 13 22:55:00 ns1 /usr/sbin/cron[90092]: (root) CMD (/usr/libexec/atrun)
Feb 13 22:55:00 ns1 /usr/sbin/cron[90091]: (mailman) CMD (/usr/local/bin/python2.4 -S /usr/local/mailman/cron/gate_news)

Adding -f option causes tail to not stop when end of file is reached, but rather to wait for additional data to be appended to the file, which is very usefull for monitoring changes done to the log file as they come. The syntax is:

tail -f log_file

The grep(1) applicattion searches the named input file for lines containing a match to the given pattern. The pattern is actually a regular expression, which are explained in section Demonstrate proficiency with regular expressions.

To find a simple pattern within a log file execute command like this:

# grep "DHCPREQUEST" /var/log/dhcp
Feb 13 18:01:41 ns1 dhcpd: DHCPREQUEST for 192.168.86.11 (192.168.86.1) from 00:50:bf:b3:a5:00 via xl0

Displaying the context in which the searched pattern appears in the log file is very useful, especially when reviewing log files. This can be achieved through -A and -B options for printing number of lines of adequately trailing and leading context after and before matching lines.

# grep -A1 -B3 "DHCPREQUEST" /var/log/dhcp
Feb 13 18:01:41 ns1 dhcpd: DHCPDISCOVER from 00:50:bf:b3:a5:00 via xl0
Feb 13 18:01:41 ns1 dhcpd: DHCPOFFER on 192.168.86.11 to 00:50:bf:b3:a5:00 via xl0
Feb 13 18:01:41 ns1 dhcpd: DHCPREQUEST for 192.168.86.11 (192.168.86.1) from 00:50:bf:b3:a5:00 via xl0
Feb 13 18:01:41 ns1 dhcpd: DHCPACK on 192.168.86.11 to 00:50:bf:b3:a5:00 via xl0

Practice Exercises

  1. Locate a compressed log file within /var/log, i.e. messages.0.bz2, and display its content using bzcat(1) or bzless (for gziped files use zcat or zmore(1) instead).
  2. Try finding your login name in /var/log/messages and /var/log/auth.log (or authlog on some BSDs) using grep(1).

More information

tail(1), /var/log/*, syslog.conf(5), grep(1), dmesg(8), zmore(1), bzcat(1)