Audit backlog limit exceeded linux

Linux Error: audit: backlog limit exceeded [How to Solve]

audit:backlog limit exceeded audit:backlog limit exceeded audit:backlog limit exceeded audit:backlog limit exceeded audit:backlog limit exceeded audit:backlog limit exceeded audit:backlog limit exceeded audit:backlog limit exceeded .

Cause analysis:

The error is Linux kernel logs. The reason for the problem is that the audit service performs audit event operations in a busy system, and there is a bottleneck in the buffer, resulting in the system near crash.

Audit is a service used to record the user’s underlying calls in Linux system. It is used to record the user’s open, exit and other system calls, and write the records to the log file. Audit can add or delete audit rules by using the auditctl command. You can set recording for a user or for a process.

Main command: auditctl audit rules & amp; The system management tool is used to obtain status, add and delete monitoring rules, audit search query audit log tool, and audit report output audit system report

You can try to increase the audit buffer to solve this problem.

The default memory page size for Linux is 4096 bytes. You can obtain the page size through the following command: getconf page_ Size, which can be set to N times of paging

View the current default configuration auditctl – S

backlog_ Limit 320 # my centos7 1 only 320 by default

Optimize the audit service and modify the buffer size auditctl – B 8192. If not set, the system defaults to 64bytes

Settings take effect permanently:

Method 1) modify the rule configuration VIM/etc/audit/audit Rules — D — B 8192 — F 1 Parameter Description: – D delete all rules – B set the audit buffer size. If the buffer is full, the kernel will issue a failure flag – f [0|1|2] set the level of audit acquisition error. There are three values of 0/1/2. 0 is no log output; 1 is the output printk log; 2 is the highest level and will output a large amount of log information -e [0|1] enable/disable audit

Method 2) you can also set CHMOD U + X/etc/rc d/rc. local vim /etc/rc. d/rc. local auditctl -b 8192

Read More:

• [Solved] Linux Error: ENOSPC: System limit for number of file watchers
• [Solved] Linux Error: Error: ENOSPC: System limit for number of file watchers
• [Solved] Linux — 9 — txt files are copied from windows to Linux and read error
• [Solved] Error: ENOSPC: System limit for number of file watchers reached
• [Solved] Linux Error: mipsel-linux-gcc: fatal error: no input files
• Solution of device eth0 does not see to be present, delaying initialization. Error in network card under Linux
• Linux: Configure Network address through Netplan
• [Solved] ERROR: Linux route delete command failed: external program exited with error status: 2
• Installation and configuration of redis in Linux
• [Solved] Go use zoom to connect DM database and start reporting error in Linux
• Linux: How to Solve sudo operate Error
• Linux system service command error: Failed to allocate directory watch: Too many open files
• linux Ubuntu Error: No command ‘setenv‘ found
• Restart and stop of nginx in Linux service
• [Solved] Linux SSH Login Terminal Error: shell request failed on channel 0
• EasyConnect linux (ubuntu 20.04) Run Error [How to Solve]
• [Solved] Error in installing backups.lzma for Python on Linux
• Linux: How to Solve Rabbitmq Plug-in Install Error
• [Solved] Linux WebService Startup Error: BindException: Cannot assign requested address
• [Solved] Linux Service Start Error: *.service: Main process exited, code=exited, status=203/EXEC

Источник

Linux audit buffer configuration

Recently, I encountered a case where a host was in a hung state, and the IPMI console of the host at that time showed the log: audit: backlog limit exceeded , and for some reasons, the NMI signal was not sent in time to trigger the kernel core dump, so I could only troubleshoot according to the existing information, and recorded the following audit buffer related configuration learning.

Audit

The Linux kernel introduced audit in 2.6 to better record various security events in the system, such as file modification events and system call events.

Configuration methods

• Control rules: set some behavior of the audit system and modify its default settings
• File system rules: audit files, record access to special files or directories
• System call rules: record the system call behavior of some special applications

buffer configuration

audit: audit_backlog=321 > audit_backlog_limit=320 audit: audit_lost=44395 audit_rate_limit=0 audit_backlog_limit=320 audit: backlog limit exceeded 

• backlog_limit
  • queue maximum length, when an event is logged that causes the queue length to exceed the limit, then a failure will occur
  • rate, if the number of events in a second exceeds the limit, then the queue will not be added and a failure will occur

  Troubleshooting

  • If the event cannot be logged, then a fault will occur and the handling behavior will be determined by the flag setting
    • 0, silent, silent processing
    • 1, printk (default behavior), prints to the system log, specific print limits based on kernel parameters.

    # sysctl -a | grep kernel.printk_rate kernel.printk_ratelimit = 5 kernel.printk_ratelimit_burst = 10 

    buffer resource calculation

    • queue in memory, you need to set a reasonable backlog_limit value to prevent occupying too much memory resources, each event is around 9000 bytes, if set to 320, then occupy memory resources of 320 * 9000 = 2.7 MiB or so.

    Possible problems encountered

    • audit: backlog limit exceeded
      • The IPMI console prints the above log, indicating that events are not being logged correctly and that the current number of events exceeds the backlog_limit limit, which may cause the system to hang or remain unresponsive.

      An audit buffer queue at or exceeding capacity might also cause the instance to hang or remain in an unresponsive state.

      • Audit system setup parameters are not set properly.
      • File system freeze (usually due to system snapshot)

       1  2  3  4  5  6  7  8  9 10 11 12 

      [root@dogfood-idc-elf-65 audit]# rpm -qa |grep audit audit-2.4.1-5.el7.x86_64 audit-libs-2.4.1-5.el7.x86_64 [root@dogfood-idc-elf-65 audit]# auditctl -s enabled 1 flag 1 pid 1093 rate_limit 0 backlog_limit 320 lost 0 backlog 0 loginuid_immutable 0 unlocked 

       1  2  3  4  5  6  7  8  9 10 11 

      [root@node90 14:16:09 ~]$rpm -q audit audit-2.7.6-3.el7.x86_64 [root@node90 14:16:13 ~]$auditctl -s enabled 1 failure 1 pid 1133 rate_limit 0 backlog_limit 8192 lost 0 backlog 0 loginuid_immutable 0 unlocked 

      Note: auditd disables restart operations (RefuseManualStop = yes), so you cannot use [[systemctl]] to control service start/stop. You can use service instead: service auditd restart

      Summary

      Redhat recommends reasonable configuration of audit-related parameters on online servers to avoid some unexpected situations due to unreasonable parameters. However, the impact of an audit exception is not very clear, as stated in the AWS KB

      An audit buffer queue at or exceeding capacity might also cause the instance to hang or remain in an unresponsive state.

      However, there is no detail in the Redhat KB about what the impact might be, so this needs to be investigated.

      Источник

      EC2 Error Audit: Backlog Limit Exceeded – How to fix it

      Generally, we see the EC2 Error Audit: Backlog Limit Exceeded in the Amazon EC2 Linux instance’s screenshot and system logs.

      Here, at Bobcares, we assist our customers with several AWS queries as part of our AWS Support Services.

      Today, let us discuss why we receive these messages, and how we can prevent them from reoccurring.

      EC2 Error Audit: Backlog Limit Exceeded

      In a Linux system, the audit backlog buffer is to maintain or log audit events. When a new audit event triggers, the system logs the event and adds it to the audit backlog buffer queue.

      The backlog_limit parameter value is the number of audit backlog buffers.

      By default, the parameter is set to 320:

      # auditctl -s enabled 1 failure 1 pid 2264 rate_limit 0 backlog_limit 320 lost 0 backlog 0

      Any event beyond the default number can result in the following errors:

      audit: audit_backlog=321 > audit_backlog_limit=320 audit: audit_lost=44393 audit_rate_limit=0 audit_backlog_limit=320 audit: backlog limit exceeded

      audit_printk_skb: 153 callbacks suppressed audit_printk_skb: 114 callbacks suppressed

      An audit buffer queue at or exceeding capacity might also cause the instance to hang or remain unresponsive.

      To avoid them, we increase the backlog_limit parameter value. Mostly, increasing buffer space helps avoid error messages.

      Here is a calculation of the memory for the auditd backlog. We can use this to determine how large we can make the backlog queue without causing memory stress.

      One audit buffer = 8970 Bytes Default number of audit buffers (backlog_limit parameter) = 320 320 * 8970 = 2870400 Bytes, or 2.7 MiB

      The MAX_AUDIT_MESSAGE_LENGTH parameter defines the size of the audit buffer.

      How to fix further errors?

      In case the instance is inaccessible with backlog limit exceeded messages in the system log, we stop and start the instance.

      Then, we perform the following steps to change the audit buffer value.

      Here we change the backlog_limit parameter value to 8192 buffers.

      1. First and foremost, we access the instance via SSH.

      2. Then we verify the current audit buffer size.

      For Amazon Linux 1 and other OS that don’t have systemd:

      $ sudo cat /etc/audit/audit.rules # This file contains the auditctl rules that are loaded # whenever the audit daemon is started via the initscripts. # The rules are simply the parameters that would be passed # to auditctl. # First rule - delete all -D # Increase the buffers to survive stress events. # Make this bigger for busy systems -b 320 # Disable system call auditing. # Remove the following line if you need the auditing. -a never,task # Feel free to add below this line. See auditctl man page

      For Amazon Linux 2 and other OS that use systemd:

      $ sudo cat /etc/audit/audit.rules # This file is automatically generated from /etc/audit/rules.d -D -b 320 -f 1

      3. After that we with an editor we access the audit.rules file:

      For Amazon Linux 1 and other OS that doesn’t use systemd:

      $ sudo vi /etc/audit/audit.rules

      For Amazon Linux 2 and other OS that use systemd:

      $ sudo vi /etc/audit/rules.d/audit.rules

      4. Now we proceed to edit the -b parameter to a larger value.

      $ sudo cat /etc/audit/audit.rules # This file contains the auditctl rules that are loaded # whenever the audit daemon is started via the initscripts. # The rules are simply the parameters that would be passed # to auditctl. # First rule - delete all -D # Increase the buffers to survive stress events. # Make this bigger for busy systems -b 8192 # Disable system call auditing. # Remove the following line if you need the auditing. -a never,task # Feel free to add below this line. See auditctl man page $ sudo auditctl -s enabled 1 failure 1 pid 2264 rate_limit 0 backlog_limit 320 lost 0 backlog 0

      5. Once done, we restart the auditd service. The new backlog_limit value takes effect.

      In addition, as we can see, it updates in auditctl -s:

      # sudo service auditd stop Stopping auditd: [ OK ] # sudo service auditd start Starting auditd: [ OK ] # auditctl -s enabled 1 failure 1 pid 26823 rate_limit 0 backlog_limit 8192 lost 0 backlog 0

      [Stuck in between? We are here for you]

      Conclusion

      In short, we saw how our Support Techs fix the EC2 error.

      PREVENT YOUR SERVER FROM CRASHING!

      Never again lose customers to poor server speed! Let us help you.

      Our server experts will monitor & maintain your server 24/7 so that it remains lightning fast and secure.

      Источник