Review log files to troubleshoot and monitor system behavior

Author: Cezary Morga cm@therek.net FreeBSD

Reviewer: name contact BSD flavour

Concept

Be aware of the importance of reviewing log files on a regular basis as well as how to watch a log file when troubleshooting. Be able to view compressed(((compression))) logs.

Introduction

The review and monitoring of log files can help maintain the health of a system. The tools like dmesg(8), tail(1) and grep(1) all help the administrator to troubleshoot problems. What and how a system logs is controlled by the syslogd(8) program, the amount and verbosity of logging is configured in the syslog.conf file (see Configure system logging). As log files are often rotated and compressed regularly by the system, tools such as zmore(1) and bzcat(1) become useful.

The default directory where the log files are stored is /var/log/. In some situations, i.e. in chrooted environment (see Recognize the BSD methods for restraining a service), log files can also be located elsewhere within the system.

Examples

The dmesg(8) utility displays the contents of the system message buffer. By default, the buffer is read from the currently running kernel. File /var/run/dmesg.boot is a copy of the buffer content taken soon after system boot.

""# dmesg ""Copyright (c) 1992-2006 The FreeBSD Project. ""Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 "" The Regents of the University of California. All rights reserved. ""FreeBSD 6.1-RELEASE #0: Sun May 7 04:32:43 UTC 2006 "" root@opus.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC "" ""<snipped> "" ""ad0: 19092MB at ata0-master UDMA33 ""acd0: CDROM at ata1-master PIO4 ""Trying to mount root from ufs:/dev/ad0s1a ""fxp0: promiscuous mode enabled ""fxp0: promiscuous mode disabled ""fxp0: link state changed to DOWN ""fxp0: link state changed to UP

The tail(1) utility displays the last part of a file. When typed without any additional flags tail displays ten last lines. This default behaviour can be modified by adding -n option with number of lines to be displayed:

""# tail -n3 /var/log/cron ""Feb 13 22:55:00 ns1 /usr/sbin/cron[90089]: (operator) CMD (/usr/libexec/save-entropy) ""Feb 13 22:55:00 ns1 /usr/sbin/cron[90092]: (root) CMD (/usr/libexec/atrun) ""Feb 13 22:55:00 ns1 /usr/sbin/cron[90091]: (mailman) CMD (/usr/local/bin/python2.4 -S /usr/local/mailman/cron/gate_news)

Adding -f option causes tail to not stop when end of file is reached, but rather to wait for additional data to be appended to the file, which is very usefull for monitoring changes done to the log file as they come. The syntax is:

""tail -f log_file

The grep(1) applicattion searches the named input file for lines containing a match to the given pattern. The pattern is actually a regular expression, which are explained in section Demonstrate proficiency with regular expressions.

To find a simple pattern within a log file execute command like this:

""# grep "DHCPREQUEST" /var/log/dhcp ""Feb 13 18:01:41 ns1 dhcpd: DHCPREQUEST for 192.168.86.11 (192.168.86.1) from 00:50:bf:b3:a5:00 via xl0

Displaying the context in which the searched pattern appears in the log file is very useful, especially when reviewing log files. This can be achieved through -A and -B options for printing number of lines of adequately trailing and leading context after and before matching lines.

""# grep -A1 -B3 "DHCPREQUEST" /var/log/dhcp ""Feb 13 18:01:41 ns1 dhcpd: DHCPDISCOVER from 00:50:bf:b3:a5:00 via xl0 ""Feb 13 18:01:41 ns1 dhcpd: DHCPOFFER on 192.168.86.11 to 00:50:bf:b3:a5:00 via xl0 ""Feb 13 18:01:41 ns1 dhcpd: DHCPREQUEST for 192.168.86.11 (192.168.86.1) from 00:50:bf:b3:a5:00 via xl0 ""Feb 13 18:01:41 ns1 dhcpd: DHCPACK on 192.168.86.11 to 00:50:bf:b3:a5:00 via xl0

Practice Exercises

Locate a compressed log file within /var/log, i.e. messages.0.bz2, and display its content using bzcat(1) or bzless (for gziped files use zcat or zmore(1) instead).
Try finding your login name in /var/log/messages and /var/log/auth.log (or authlog on some BSDs) using grep(1).

More information

tail(1), /var/log/*, syslog.conf(5), grep(1), dmesg(8), zmore(1), bzcat(1)