System event log linux

How to View and Configure Linux System Logs on Ubuntu 20.04

This tutorial explains the basic administration of a Linux server through system logs. A system log is a file that contains information about the events that happened on the system during runtime.

In this article, you will learn the following Linux logging basics:

• Where the Linux log files are stored, how are they formatted, and how to read them.
• How to read the most important logs (such as syslog ).
• How to configure the Ubuntu syslog daemon.
• What Linux log rotation is all about and how to use the logrotate utility.

Prerequisites

Before proceeding with the rest of this tutorial, ensure that you have a basic knowledge of working with the Linux command line. While many of the concepts discussed in this article are general applicable to all Linux distributions, we’ll be demonstrating them in Ubuntu only so ensure to set up an Ubuntu 20.04 server that includes a non-root user with sudo access.

🔭 Want to centralize and monitor your Linux logs?

Head over to Logtail and start ingesting your logs in 5 minutes.

Step 1 — Finding Linux system logs

All Ubuntu system logs are stored in the /var/log directory. Change into this directory in the terminal using the command below:

You can view the contents of this directory by issuing the following command:

You should see a similar output to the following:

alternatives.log auth.log btmp cloud-init-output.log dmesg dpkg.log journal/ landscape/ private/ ubuntu-advantage-license-check.log ubuntu-advantage-timer.log unattended-upgrades/ apt/ bootstrap.log cloud-init.log dist-upgrade/ dmesg.0 faillog kern.log lastlog syslog ubuntu-advantage.log ufw.log wtmp 

Let’s look at a few of the essential system log files that may be present in the /var/log directory and what they contain:

• /var/log/syslog : stores general information about any global activity in the system.
• /var/log/auth.log : keeps track of all security-related actions (login, logout, or root user activity).
• /var/log/kern.log : stores information about events originating from the Linux kernel.
• /var/log/boot.log : stores system startup messages.
• /var/log/dmesg : contains messages related to device drivers.
• /var/log/faillog : keeps track of failed logins, which comes in handy when investigating attempted security breaches.

The /var/log directory is also used to store various application logs. For example, if your distribution is bundled with Apache or MySQL, or installed later, their log files will also be found here.

Step 2 — Viewing Linux log file contents

Log files contain a large amount of information that are useful for monitoring or analyzing activities performed by the system or a specific application. Therefore, a Linux server administrator must learn the art of reading and understanding the various messages present in log files to effectively diagnose or troubleshoot an issue.

Before we can read log files, we ought to know how they are formatted. Let’s review two basic approaches to log file formatting and storage: plain text and binary files.

Plaintext log files

These logs are plain text files with a standardized content format. Ubuntu uses a log template called RSYSLOG_TraditionalFileFormat . This log format consists of four main fields with a space delimiter:

1. The timestamp indicates the time when a log entry was created in the format MMM dd HH:mm:ss (e.g. Sep 28 19:00:00 ). Notice that this format does not include a year.
2. Hostname is the host or system that originally create the message.
3. Application is the application that created the message.
4. Message contains the actual details of an event.

Let’s go ahead and review some log files in the plaintext format. Run the command below to print the contents of the /var/log/syslog file with the tail utility:

This outputs the last 10 lines of the file:

Mar 23 12:38:09 peter dbus-daemon[1757]: [session uid=1000 pid=1757] Activating via systemd: service name='org.freedesktop.Tracker1' unit='tracker-store.service' requested by ':1.1' (uid=1000 pid=1754 comm="/usr/libexec/tracker-miner-fs " label="unconfined") Mar 23 12:38:09 peter systemd[1743]: Starting Tracker metadata database store and lookup manager. Mar 23 12:38:09 peter dbus-daemon[1757]: [session uid=1000 pid=1757] Successfully activated service 'org.freedesktop.Tracker1' Mar 23 12:38:09 peter systemd[1743]: Started Tracker metadata database store and lookup manager. Mar 23 12:38:40 peter tracker-store[359847]: OK Mar 23 12:38:40 peter systemd[1743]: tracker-store.service: Succeeded. Mar 23 12:39:01 peter CRON[359873]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi) Mar 23 12:39:23 peter systemd[1]: Starting Clean php session files. Mar 23 12:39:23 peter systemd[1]: phpsessionclean.service: Succeeded. Mar 23 12:39:23 peter systemd[1]: Finished Clean php session files. 

You’ll notice that that each record in this file is formatted in the manner described earlier. For example, the last record has its timestamp as Mar 23 12:39:23, hostname as peter, application as systemd[1] and message as Finished Clean php session files.

If you want to view the entire log file, you can use the cat utility or any text editor such as nano or vim .

Binary log files

While plaintext is the dominant storage format for log files, you will also encounter binary log files that cannot be read with a normal text editor. The /var/log directory contains multiple binary files that are related to the user authorization:

• /var/log/utmp : tracks users that are currently logged into the system.
• /var/log/wtmp : tracks previously logged in users. It contains a past data from utmp .
• /var/log/btmp : tracks failed login attempts.

For these binary logs, special command-line tools are used to display the relevant information in human-readable form. For example, to review the contents of the /var/log/utmp file, run the who utility with -H option (this option causes column labels to be printed in the output table):

Источник

Basics- Linux Events Logging

In this post we will cover the basics of Event Logging in Linux systems. We will talk about Syslog: Message structure, the most famous implementations and main configurations.

In this post we will cover the basics of Event Logging in Linux systems. We will talk about Syslog: Message structure, the most famous implementations and main configurations. We will also play with the inner workings of Linux logging using UNIX sockets, logger and syslog services.

What is Syslog?

Syslog is a standard (RFC5424) used for log management. This management can be local or remote. Do not confuse syslog standard with syslog applications like Syslog-ng, Rsyslog, Nxlog”¦

In some of the most famous Linux distros like Ubuntu, Debian or Fedora, Rsyslog is installed by default. This application can be used for local or remote log management.

Message structure

Syslog message structure is similar to this:

PRI value is calculated using the facility and priority values (see tables and formula below). Facility and Severity values are not normative but often used. They are described in the following tables for purely informational purposes. Facility values MUST be in the range of 0 to 23 inclusive.

The formula we will use to calculate the PRI is:

This result will be between 0 and 191.

Log storage

We already know the standard syslog message structure. Now, how are these messages managed?

In Linux environments we have a special type of files called “UNIX sockets”. These files are used to transfer information between processes. Processes will read and write data to the socket in order to communicate. In Debian we can list UNIX sockets with the following command:

In our case, applications will write their log messages into this Syslog dedicated socket. Rsyslog service (in Debian/Ubuntu/Fedora) will be “listening” (reading) from this socket. Every new log entry will then be processed according to Rsyslog configuration (later mentioned).

We can send a random message to this socket in order to test our Rsyslog configuration:

Note that you can specify a PRI with the following command: “logger -p kern.err hola”

Rsyslog service

As we have just seen, Rsyslog must be always listening to any log income from syslog UNIX socket. In order to achieve this, Rsyslog service must be always up and running (obviously). You can check its status with the following command:

One of our main questions is: Which criteria does Rsyslog use to classify and store logs in different files under /var/log? Can this location be modified?

Of course. Rsyslog behaviour is defined in its configuration file and can be easily modified:

This is some of its contents by default:

The storage rules are defined in the first lines following this syntax:

All cron facility logs of any severity will be stored under /var/log/cron.log

If we want to send these logs to a remote machine (usually a Log Collector) we will use the following settings (depending on if we want to use TCP or UDP):

In general, it is suggested to use TCP syslog since it is way more reliable than UDP. Keep in mind that in case of network congestion, messages might be dropped. We UDP these logs will be lost, but with TCP they will be resend after the network issue. Also note that a TLS option is available in order to send logs, but only with TCP. Nevertheless, TCP is not fully reliable and might also introduce some losses (see this post about TCP reliability). In order to increase reliability, see the RELP protocol from Rsyslog (See below) or the standard RFC3195.

In order to use it, install rsyslog-relp in both client and server (collector). In Debian:

Then you should add a couple of lines in the config files:

• module(load=”imrelp”)

• input(type=”imrelp” port=”4444″ maxDataSize=”10k”)

Then make sure you restart Rsyslog service in both machines.

RELP is not part of standard yet and has only been implemented in the following applications:

• librelp – the original C RELP library
• rsyslog (Windows and Linux)
• MonitorWare (Windows)
• logstash

Standard locations under /var/log/

Even though the destination files are fully configurable -as we have just seen-, we will try to maintain some standard names and log files under our /var/log directory:

• Messages (or syslog): General messages. Useful for a first look at logs
• Kern.log: Kernel logs
• Auth.log (or secure): Authorization logs, users’ sessions
• Dmesg: Device driver messages. Use “dmesg” to see its content
• Faillog: Log of failed logins
• Cron: Info related to cron jobs
• Daemon.log: Info about background services (daemons)
• Btmp: Failed login attempts
• Utmp: Current login state by user
• Wtmp: Record of each login/logout
• Lastlog: Record of the last login
• Yum/apt: Logs of package installations
• User.log: Info about all user level logs (coming from user programs)
• Xorg.x.log: Logs about the X (graphic interface)

Log rotation with Rsyslog

Log rotation is the action of saving (usually compressed) old logs for a determined amount of time. In Debian we can achieve this with “logrotate”. This is one of its configuration files (under /etc/logrotate.d/), for Rsyslog it is pretty self-explanatory:

Conclusion

We have seen the basic structure of Syslog messages with its different categories and severities.

Also, we have played with Linux log management and Rsyslog configurations in order to understand its inner workings. In the next post we will introduce Journald, a “new” log manager for Linux. We will see the new features it brings compared to Rsyslog, but also its deficiencies.

Get resources in your mailbox for free

Источник