centos 6 nfs: logs not showing anywhere
Can someone please tell me where NFS logs in centos 6? Or perhaps where I can tell NFS to send logs? At the present time, there appears to be no such setting. Trying to get the thing to work without logs is quite frustrating.
[root@houston netshare]# locate nfs| grep log [root@houston netshare]# [root@houston netshare]# grep -Rni "nfs" /var/log /var/log/anaconda.storage.log:23:20:41:33,962 DEBUG : registered device format class NFS as nfs /var/log/anaconda.storage.log:24:20:41:33,962 DEBUG : registered device format class NFSv4 as nfs4
[root@houston ~]# cat /etc/exports /var/netshare 10.10.0.10(rw,sync,no_root_squash) /var/netshare/esxi1 192.168.111.119(rw,sync,no_root_squash) /var/netshare 192.168.111.101(rw,sync.no_root_squash) [root@houston ~]# chkconfig --list | egrep '(nfs|rpc*)' nfs 0:off 1:off 2:off 3:off 4:off 5:off 6:off nfslock 0:off 1:off 2:off 3:on 4:on 5:on 6:off rpcbind 0:off 1:off 2:on 3:on 4:on 5:on 6:off rpcgssd 0:off 1:off 2:off 3:on 4:on 5:on 6:off rpcidmapd 0:off 1:off 2:off 3:on 4:on 5:on 6:off rpcsvcgssd 0:off 1:off 2:off 3:off 4:off 5:off 6:off [root@houston ~]# ps aux | egrep '(nfs|rpc*)' rpc 1482 0.0 0.0 18972 1012 ? Ss Dec06 0:00 rpcbind root 1587 0.0 0.0 0 0 ? S Dec06 0:00 [fc_rport_eq] rpcuser 1648 0.0 0.0 23344 1352 ? Ss Dec06 0:00 rpc.statd root 1690 0.0 0.0 0 0 ? S Dec06 0:00 [rpciod/0] root 1691 0.0 0.0 0 0 ? S Dec06 0:00 [rpciod/1] root 1692 0.0 0.0 0 0 ? S Dec06 0:00 [rpciod/2] root 1693 0.0 0.0 0 0 ? S Dec06 0:00 [rpciod/3] root 29056 0.0 0.0 107304 280 ? Ss 11:36 0:00 rpc.rquotad root 29060 0.0 0.0 21760 1264 ? Ss 11:36 0:00 rpc.mountd root 29111 0.0 0.0 25160 760 ? Ss 11:36 0:00 rpc.idmapd root 29116 0.0 0.0 0 0 ? S 11:36 0:00 [nfsd4] root 29117 0.0 0.0 0 0 ? S 11:36 0:00 [nfsd4_callbacks] root 29118 0.0 0.0 0 0 ? S 11:36 0:00 [nfsd] root 29119 0.0 0.0 0 0 ? S 11:36 0:00 [nfsd] root 29120 0.0 0.0 0 0 ? S 11:36 0:00 [nfsd] root 29121 0.0 0.0 0 0 ? S 11:36 0:00 [nfsd] root 29122 0.0 0.0 0 0 ? S 11:36 0:00 [nfsd] root 29123 0.0 0.0 0 0 ? S 11:36 0:00 [nfsd] root 29124 0.0 0.0 0 0 ? S 11:36 0:00 [nfsd] root 29125 0.0 0.0 0 0 ? S 11:36 0:00 [nfsd] root 29357 0.0 0.0 100992 748 pts/1 S+ 12:56 0:00 egrep (nfs|rpc*) [root@houston ~]# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere anywhere REJECT all -- anywhere loopback/8 reject-with icmp-port-unreachable ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED ACCEPT tcp -- anywhere anywhere tcp dpt:http ACCEPT tcp -- anywhere anywhere tcp dpt:https ACCEPT tcp -- anywhere anywhere tcp dpt:51414 ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ssh ACCEPT udp -- anywhere anywhere state NEW udp dpt:sunrpc ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:sunrpc ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:nfs ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:32803 ACCEPT udp -- anywhere anywhere state NEW udp dpt:filenet-rpc ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:892 ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:47649 ACCEPT udp -- anywhere anywhere state NEW udp dpt:892 ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:rquotad ACCEPT udp -- anywhere anywhere state NEW udp dpt:rquotad ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:pftp ACCEPT udp -- anywhere anywhere state NEW udp dpt:pftp ACCEPT icmp -- anywhere anywhere LOG all -- anywhere anywhere limit: avg 5/min burst 5 LOG level debug prefix `iptables denied: ' DROP all -- anywhere anywhere Chain FORWARD (policy ACCEPT) target prot opt source destination DROP all -- anywhere anywhere Chain OUTPUT (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere anywhere
How to enable NFS related debug logging
This document (7011571) is provided subject to the disclaimer at the end of this document.
Environment
SUSE Linux Enterprise Server 15
SUSE Linux Enterprise Server 12
SUSE Linux Enterprise Server 11
SUSE Linux Enterprise Server 10
Situation
A NFS Server or Client on SUSE Linux Enterprise is not working as expected. The logs don’t provide any details about what goes wrong.
Resolution
There is additional debug logging that can be turned on for NFS related functions. Be warned, however, that this logging is very verbose and very cryptic. It may not be helpful to anyone but a developer. A better approach (than the one here) is probably to pursue analysis of the problem through network packet captures (i.e. tcpdump). Of course, either of these methods (debug logs or packet captures) will require certain skill and experience. If you are getting this debug information with the intention of bringing it to someone else to analyse, then you probably will want that person’s input ahead of time, concerning what method of troubleshooting and data gathering to use for your particular symptoms. You may just waste time if you gather data for someone without their input.
If you are going to gather debug logs of an NFS issue, here is the approach:
1. Set up the environment so that the problematic behavior can be reproduced immediately. It is best to enable additional debugging when the issue is directly reproduced and no other distracting logs have to be read and analysed. Note, also, that the debug logging can be so intensive and verbose that it may slow down performance of the system. It is not very desirable to leave these debug methods turned on for long periods of time. So the basic steps should be:
a. Enable debugging.
b. Promptly reproduce the issue.
c. Disable debugging.
2. rpcdebug is a command-line tool that enables and disables debugging for various nfs related modules, and various categories of debug logging within those modules. The «-m» flag identifies the module were debug is being activated, and the «-s» option identifies which debug flags will be set.
To generate debug logs for NFS Server functions:
To generate debug logs for NFS Client functions:
[Note that the difference between nfs server and nfs client is «nfsd» vs «nfs» respectively.]
In many cases the RPC protocol needs to be debugged as well, and that can be achieved by using:
The additional logging information will appear in /var/log/messages or can be viewed with the command «dmesg».
To disable the respective debugging options, use the same command but with the «-c» option to clear the flags in question.
rpcdebug -m nfsd -c all
rpcdebug -m nfs -c all
rpcdebug -m rpc -c all
In cases where you are pretty sure about where the issue resides (nfs server vs nfs client) you may not need to enable debugging at both ends. But in some cases, it may be beneficial to enable rpc/nfs at the NFS client system, and rpc/nfsd at the NFS Server system.
On some quite old Linux distributions, the «rpcdebug» command does not exist. However, that command is really just a convenient front end to setting values in:
Even without an «rpcdebug» command, you can accomplish the same thing with the sysctl command, or with the echo command.
For example, the sysctl version of the above 3 commands would be:
sysctl -w sunrpc.nfsd_debug=0x7fff
sysctl -w sunrpc.nfs_debug=0xffff
sysctl -w sunrpc.rpc_debug=0x7fff
And setting any of those to 0 or 0x0 would turn that module’s debug off again.
Or the echo versions would be:
echo 0x7fff > /proc/sys/sunrpc/nfsd_debug
echo 0xffff > /proc/sys/sunrpc/nfs_debug
echo 0x7fff > /proc/sys/sunrpc/rpc_debug
And echoing 0 or 0x0 into those locations would turn them off again.
Disclaimer
This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented «AS IS» WITHOUT WARRANTY OF ANY KIND.
- Document ID:7011571
- Creation Date: 03-Jan-2013
- Modified Date:15-Jul-2020
-
- SUSE Linux Enterprise Desktop
- SUSE Linux Enterprise Server
For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com
More information:
I asked a similar question from the Ubuntu client perspective on AskUbuntu. My focus in this question is on the Arch Linux server. In particular, I am looking for logs on the server that will help me understand the problem. Here’s the background: Our small LAN is running an Arch Linux NFS v4 file server. We have several clients running Ubuntu 15.10 and 16.04. We have one client running Ubuntu 14.04. The 14.04 client will not connect to the file server. The others all connect fine. The settings are the same on all clients. And all clients are listed in /etc/exports on the server. I need to find more detailed error information on the Arch linux server. However, journalctl does not show anything related to nfs and it does not contain any entries that are related to the nfs access denied errors. The 14.04 client can ping the fileserver as well as log in via SSH. The user name / ID as well as group match. (I’m using the same user account / uid on both client and server. It is uid 1000.)
Even more info:
$ sudo mount -a (on client) mount.nfs4: access denied by server while mounting fileserver:/export/path/one mount.nfs4: access denied by server while mounting fileserver:/export/path/two
$ ping fileserver PING fileserver (192.168.1.1) 56(84) bytes of data. 64 bytes from fileserver (192.168.1.1): icmp_seq=1 ttl=64 time=0.310 ms
$ ssh fileserver Last login: Tue Aug 16 14:38:26 2016 from 192.168.1.2 [me@fileserver ~]$
$ showmount -e fileserver # on client Export list for fileserver: /export/path/one/ 192.168.1.2 /export/path/two/ 192.168.1.2,192.168.1.3 $ rpcinfo -p fileserver (on client) program vers proto port service 100000 4 tcp 111 portmapper 100000 3 tcp 111 portmapper 100000 2 tcp 111 portmapper 100000 4 udp 111 portmapper 100000 3 udp 111 portmapper 100000 2 udp 111 portmapper 100024 1 udp 58344 status 100024 1 tcp 58561 status 100005 1 udp 20048 mountd 100005 1 tcp 20048 mountd 100005 2 udp 20048 mountd 100005 2 tcp 20048 mountd 100005 3 udp 20048 mountd 100005 3 tcp 20048 mountd 100003 4 tcp 2049 nfs 100003 4 udp 2049 nfs
$ sudo mount -vvv -t nfs4 fileserver:/export/path/one /path/one/ mount: fstab path: "/etc/fstab" mount: mtab path: "/etc/mtab" mount: lock path: "/etc/mtab~" mount: temp path: "/etc/mtab.tmp" mount: UID: 0 mount: eUID: 0 mount: spec: "fileserver:/export/path/one" mount: node: "/path/one/" mount: types: "nfs4" mount: opts: "(null)" mount: external mount: argv[0] = "/sbin/mount.nfs4" mount: external mount: argv[1] = "fileserver:/export/path/one" mount: external mount: argv[2] = "/path/one/" mount: external mount: argv[3] = "-v" mount: external mount: argv[4] = "-o" mount: external mount: argv[5] = "rw" mount.nfs4: timeout set for Tue Aug 16 16:10:43 2016 mount.nfs4: trying text-based options 'addr=192.168.1.1,clientaddr=192.168.1.2' mount.nfs4: mount(2): Permission denied mount.nfs4: access denied by server while mounting fileserver:/export/path/one