Linux audit rate limit exceeded

kernel panic: audit: rate limit exceeded

Возможно, адреса электронной почты являются анонимными для этой группы или вам требуется разрешение на просмотр адресов электронной почты ее участников, чтобы увидеть исходное сообщение.

– epa. @redhat.com, kv. @codeaurora.org, linux. @redhat.com, linux-. @vger.kernel.org, net. @vger.kernel.org, pa. @paul-moore.com, peter. @collabora.com, romain. @collabora.com, stas.y. @gmail.com, syzkall. @googlegroups.com

syzbot found the following crash on:

commit 28b75415ad19fef232d8daab4d5de17d753f0b36
Author: Romain Perier
Date: Wed Aug 23 07:16:51 2017 +0000

wireless: ipw2200: Replace PCI pool old API

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+72461a. @syzkaller.appspotmail.com
Fixes: 28b75415ad19 («wireless: ipw2200: Replace PCI pool old API»)

audit: audit_lost=1 audit_rate_limit=2 audit_backlog_limit=0
Kernel panic — not syncing: audit: rate limit exceeded
CPU: 1 PID: 10031 Comm: syz-executor626 Not tainted 5.6.0-rc2-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x197/0x210 lib/dump_stack.c:118
panic+0x2e3/0x75c kernel/panic.c:221
audit_panic.cold+0x32/0x32 kernel/audit.c:307
audit_log_lost kernel/audit.c:377 [inline]
audit_log_lost+0x8b/0x180 kernel/audit.c:349
audit_log_end+0x23c/0x2b0 kernel/audit.c:2322
audit_log_config_change+0xcc/0xf0 kernel/audit.c:396
audit_receive_msg+0x2246/0x28b0 kernel/audit.c:1277
audit_receive+0x114/0x230 kernel/audit.c:1513
netlink_unicast_kernel net/netlink/af_netlink.c:1303 [inline]
netlink_unicast+0x59e/0x7e0 net/netlink/af_netlink.c:1329
netlink_sendmsg+0x91c/0xea0 net/netlink/af_netlink.c:1918
sock_sendmsg_nosec net/socket.c:652 [inline]
sock_sendmsg+0xd7/0x130 net/socket.c:672
____sys_sendmsg+0x753/0x880 net/socket.c:2343
___sys_sendmsg+0x100/0x170 net/socket.c:2397
__sys_sendmsg+0x105/0x1d0 net/socket.c:2430
__do_sys_sendmsg net/socket.c:2439 [inline]
__se_sys_sendmsg net/socket.c:2437 [inline]
__x64_sys_sendmsg+0x78/0xb0 net/socket.c:2437
do_syscall_64+0xfa/0x790 arch/x86/entry/common.c:294
entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x441239
Code: e8 fc ab 02 00 48 83 c4 18 c3 0f 1f 80 00 00 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 3d 01 f0 ff ff 0f 83 9b 09 fc ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007ffd68c9df48 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 0000000000441239
RDX: 0000000000000000 RSI: 00000000200006c0 RDI: 0000000000000003
RBP: 0000000000018b16 R08: 00000000004002c8 R09: 00000000004002c8
R10: 0000000000000004 R11: 0000000000000246 R12: 0000000000402060
R13: 00000000004020f0 R14: 0000000000000000 R15: 0000000000000000
Kernel Offset: disabled
Rebooting in 86400 seconds..

Источник

Linux audit buffer configuration

Recently, I encountered a case where a host was in a hung state, and the IPMI console of the host at that time showed the log: audit: backlog limit exceeded , and for some reasons, the NMI signal was not sent in time to trigger the kernel core dump, so I could only troubleshoot according to the existing information, and recorded the following audit buffer related configuration learning.

Audit

The Linux kernel introduced audit in 2.6 to better record various security events in the system, such as file modification events and system call events.

Configuration methods

• Control rules: set some behavior of the audit system and modify its default settings
• File system rules: audit files, record access to special files or directories
• System call rules: record the system call behavior of some special applications

buffer configuration

audit: audit_backlog=321 > audit_backlog_limit=320 audit: audit_lost=44395 audit_rate_limit=0 audit_backlog_limit=320 audit: backlog limit exceeded 

• backlog_limit
  • queue maximum length, when an event is logged that causes the queue length to exceed the limit, then a failure will occur
  • rate, if the number of events in a second exceeds the limit, then the queue will not be added and a failure will occur

  Troubleshooting

  • If the event cannot be logged, then a fault will occur and the handling behavior will be determined by the flag setting
    • 0, silent, silent processing
    • 1, printk (default behavior), prints to the system log, specific print limits based on kernel parameters.

    # sysctl -a | grep kernel.printk_rate kernel.printk_ratelimit = 5 kernel.printk_ratelimit_burst = 10 

    buffer resource calculation

    • queue in memory, you need to set a reasonable backlog_limit value to prevent occupying too much memory resources, each event is around 9000 bytes, if set to 320, then occupy memory resources of 320 * 9000 = 2.7 MiB or so.

    Possible problems encountered

    • audit: backlog limit exceeded
      • The IPMI console prints the above log, indicating that events are not being logged correctly and that the current number of events exceeds the backlog_limit limit, which may cause the system to hang or remain unresponsive.

      An audit buffer queue at or exceeding capacity might also cause the instance to hang or remain in an unresponsive state.

      • Audit system setup parameters are not set properly.
      • File system freeze (usually due to system snapshot)

       1  2  3  4  5  6  7  8  9 10 11 12 

      [root@dogfood-idc-elf-65 audit]# rpm -qa |grep audit audit-2.4.1-5.el7.x86_64 audit-libs-2.4.1-5.el7.x86_64 [root@dogfood-idc-elf-65 audit]# auditctl -s enabled 1 flag 1 pid 1093 rate_limit 0 backlog_limit 320 lost 0 backlog 0 loginuid_immutable 0 unlocked 

       1  2  3  4  5  6  7  8  9 10 11 

      [root@node90 14:16:09 ~]$rpm -q audit audit-2.7.6-3.el7.x86_64 [root@node90 14:16:13 ~]$auditctl -s enabled 1 failure 1 pid 1133 rate_limit 0 backlog_limit 8192 lost 0 backlog 0 loginuid_immutable 0 unlocked 

      Note: auditd disables restart operations (RefuseManualStop = yes), so you cannot use [[systemctl]] to control service start/stop. You can use service instead: service auditd restart

      Summary

      Redhat recommends reasonable configuration of audit-related parameters on online servers to avoid some unexpected situations due to unreasonable parameters. However, the impact of an audit exception is not very clear, as stated in the AWS KB

      An audit buffer queue at or exceeding capacity might also cause the instance to hang or remain in an unresponsive state.

      However, there is no detail in the Redhat KB about what the impact might be, so this needs to be investigated.

      Источник

      CentOS

      What does this mean in /var/log/messages and does it need to be fixed?

      kernel: audit: printk limit exceeded

      Also, is this something new with systemd in the latest centos7, I seem to get audit messages every time cron does something every minute.

      Can I quiet it down to show only when there is a problem, or does the fact it shows messages indicate a problem — doesn’t look like it?

       kernel: type=1110 audit(1433591521.801:376901): pid=6793 uid=0 auid=0 ses=53607 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success' kernel: type=1104 audit(1433591521.806:376902): pid=6793 uid=0 auid=0 ses=53607 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success' kernel: type=1106 audit(1433591521.806:376903): pid=6793 uid=0 auid=0 ses=53607 msg='op=PAM:session_close grantors=pam_loginuid,pam_keyinit,pam_limits,pam_systemd acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success' 

      TrevorH Site Admin Posts: 32853 Joined: 2009/09/24 10:40:56 Location: Brighton, UK

      Re: kernel: audit: printk limit exceeded .

      Post by TrevorH » 2015/06/06 12:06:27

      Do you have auditd running? If auditd is running then it handles sending those messages to a different log file — /var/log/audit/audit.log but if it is not then they go via syslog to /var/log/messages.

      The future appears to be RHEL or Debian. I think I’m going Debian.
      Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
      CentOS 5 and 6 are deadest, do not use them.
      Use the FAQ Luke

      Re: kernel: audit: printk limit exceeded .

      Post by _ck_ » 2015/06/06 12:14:25

      Should I have auditd running? Is there a performance hit?

      Can I knock down the verbosity to errors/warnings instead of every activity?

      What’s strange is I don’t remember seeing all these constant messages in /var/log/messages until one of the more recent centos7 updates.

      Did something change or I did I accidentally turnoff auditd when it was running before?

      Aha, I see a /var/log/audit/ directory and it has several rotated audit.log

      There seems to be one each day except not today.

      Maybe I accidentally killed auditd somehow. Nope it is running.

      auditd.service - Security Auditing Service Loaded: loaded (/usr/lib/systemd/system/auditd.service; enabled) Active: active (running) since Thu 2015-05-21 07:28:41 EDT; 2 weeks 2 days ago Main PID: 583 (auditd) CGroup: /system.slice/auditd.service +-583 /sbin/auditd -n 

      Re: kernel: audit: printk limit exceeded .

      Post by _ck_ » 2015/06/06 12:23:36

      Restarted auditd and the problem went away.

      Maybe there was a write-lock on audit.log when it rotated and could not write to it, so started to dump to /var/log/messages

      Thanks for pointing out auditd, saved me a bunch of time.

      • CentOS General Purpose
      • ↳ CentOS — FAQ & Readme First
      • ↳ Announcements
      • ↳ CentOS Social
      • ↳ User Comments
      • ↳ Website Problems
      • CentOS 8 / 8-Stream / 9-Stream
      • ↳ 8 /8-Stream / 9-Stream — General Support
      • ↳ 8 /8-Stream / 9-Stream — Hardware Support
      • ↳ 8 /8-Stream / 9-Stream — Networking Support
      • ↳ 8 /8-Stream / 9-Stream — Security Support
      • CentOS 7
      • ↳ CentOS 7 — General Support
      • ↳ CentOS 7 — Software Support
      • ↳ CentOS 7 — Hardware Support
      • ↳ CentOS 7 — Networking Support
      • ↳ CentOS 7 — Security Support
      • CentOS Legacy Versions
      • ↳ CentOS 5
      • ↳ CentOS 5 — General Support
      • ↳ CentOS 5 — Software Support
      • ↳ CentOS 5 — Hardware Support
      • ↳ CentOS 5 — Networking Support
      • ↳ CentOS 5 — Server Support
      • ↳ CentOS 5 — Security Support
      • ↳ CentOS 5 — Oracle Installation and Support
      • ↳ CentOS 5 — Miscellaneous Questions
      • ↳ CentOS 6
      • ↳ CentOS 6 — General Support
      • ↳ CentOS 6 — Software Support
      • ↳ CentOS 6 — Hardware Support
      • ↳ CentOS 6 — Networking Support
      • ↳ CentOS 6 — Security Support

      Powered by phpBB® Forum Software © phpBB Limited

      Источник