How to Troubleshoot Problems by using logs on Redhat

How Troubleshoot Problems by using logs/Log Files/Systemd-Journald (journalctl) on Redhat 7/Redhat 8/Redhat 9

Logs is a data that main purpose is to store the activity of user’s, application, operating system etc. The main purpose of logs in a machine identify the error for the resolution of issue. In an operating system log files has created of applications, machine activities, and emails for store the information that will help the administrators to identify then resolve the issues. Actually, logs improve the detection of the problem, minimize the response time for resolve issue, increase overall security of the system and identify the breaches. Here we discuss the logs files and logs types of Redhat base operating system that will helpful to identify the issues for resolve. Accurate time of machine is very import for logs; set current time, time zone and NTP synchronization settings of the system if system is using NTP (Network Time Protocol) or chrony.

The standard logging system on Red Hat Enterprise Linux (RHEL) is based on the syslog protocol. In Red Hat Enterprise Linux (RHEL) syslog messages are handled by two services.

1. systemd-journald.
2.  Rsyslog.

Systemd-Journald: Systemd-journald daemon collect messages (logs) from the kernel, collect information’s of boot process, collect output and errors of daemon startup and running process. Systemd-Journald daemon is an improved log management service, syslog messages are forwarded by Systemd-journald to rsyslog.
Rsyslog: rsyslog sort and write syslog messages on a directory location “/var/log”, rsyslog is an open source high performance log processing. The directory “/var/log” contain various system and services specific log files that manage by rsyslog.

Below are common log files that created inside the directory location “/var/log” on Redhat enterprise Linux machine.

/var/log/messages: Most of the syslog messages log on file “messages” except authentication, email processing, periodically and debugging related logs.
/var/log/secure: The log file “secure” store the information and errors related to security and authentication related.
/var/log/maillog: The log file “maillog” store the messages related to mailserver, email relay and mail.
/var/log/cron: The log file “cron” store the information related to periodically executed tasks (daily, weekly, monthly, yearly).
/var/log/boot.log: The log file “boot.log” store the logs of system startup when the operating system boot.

In Redhat Enterprise Linux (RHEL) log is rotate by the log rotate utility, cron job run the log rotate program daily. Typically a log rotation period is 4 weeks, due to log rotation old log file is remove as per define settings and free the hard disk space.  The rules has written in a configuration file “/etc/rsyslog.conf”, for change rules configuration on file “rsyslog.conf” must restart the service “rsyslog” by type the below mention command.

systemctl restart rsyslog

To view the any log file line by line for deep understand the error or read the information, use command “more”.

more /var/log/messages

To view the last n number lines of log file, type the below mention command.

tail -n 15 /var/log/messages

To check the logs entries by use of systemd-journald to trouble problems or review system status. By default systemd journal logs is store in /run/log, after the machine reboot its log has removed.

To view the all log entries from boot through systemd-journald, type the below mention command.

journalctl

To view the last 10 log entries by use of systemd-journald, type the below mention command.

journalctl -n
or
journalctl -f

To view the last 25 log entries by use of systemd-journald “journalctl”, type the below mention command.

journalctl -n 25

By use of command “journalctl”, you can review the logs by priority like debug, info, notice, warning, err, crit, alert and emerg. For example, here I am showing you the command of information logs by use of command “journalctl”.

journalctl -p info

You can set the time range to find the logs by priority. The time range use attributes like since, until and date with time. By use of this feature, you can find the specific duration logs as per the need and save logs of specific duration by copy through putty session. If date is omitted command is assume for today and if time is omitted command it is assume for 00:00:00, from start to whole day. The some examples to find the logs of specific time duration with priority are as follows.

To find the output all journal entries that are store today, type the below mention command.

journalctl --since today

To find the output of the journal entries from 11 May 2024 00:15:18 to 19 May 2024 23:58:05, type the below mention command.

journalctl --since "2024-05-11 00:15:18" --until "2024-05-19 23:58:05"

To find the output of the journal entries from 11 May 2024 00:15:18 to 19 May 2024 23:58:05 with priority, type the below mention command. Here I have taken a priority of error “err” logs, you can take debug, info, notice, warning, crit, alert and emerg.

journalctl --since "2024-05-11 00:15:18" --until "2024-05-19 23:58:05" -p err

To find the journal entries related to processes that systemd unit file sshd.service and have a process id (PID: 7999). You can also set the time duration for this scenario.

journalctl _SYSTEMD_UNIT=sshd.service _PID=7999
journalctl --since "2024-05-11 00:15:18" --until "2024-05-19 23:58:05" _SYSTEMD_UNIT=sshd.service _PID=7999
journalctl --since "2024-05-11 00:15:18" _SYSTEMD_UNIT=sshd.service _PID=7999 

To find the output of systemd journal message originate from systemd process that have with process id 15.

journalctl _PID=15

To find the systemd journal messages with user id 1001, originate from the system services started.

journalctl _UID=1001

To find the output of the log message since the last boot of the system, type the below mention command.

journalctl -b

Note: The log architecture consist of systemd-journald for collecting logs and rsyslog to sort and write log messages to log file.

Thanks for Read this Article

Comments