Friday, April 24, 2009

Linux NFS and "permission denied" mount problem

Today, I have faced with following strange nfs mount problem.

[root@nfsclient ~]# mount /mymount
mount: XXX.XXX.XXX.XXX:/mymount failed,
reason given by server: Permission denied

My nfs client is FC4 and nfs server is Centos 4.6.First, I have checked the server from nfs client side by using "showmount -e myserver" command. Exports was okay.After that, I decided to check rpc services on server with "rpcinfo -p" command.

[root@nfsclient ~]# rpcinfo -p myserver

program vers proto port
100000 2 tcp 111 portmapper
100000 2 udp 111 portmapper
100024 1 udp 638 status
100024 1 tcp 641 status
100021 1 udp 32768 nlockmgr
100021 3 udp 32768 nlockmgr
100021 4 udp 32768 nlockmgr
100021 1 tcp 32772 nlockmgr
100021 3 tcp 32772 nlockmgr
100021 4 tcp 32772 nlockmgr
100011 1 udp 903 rquotad
100011 2 udp 903 rquotad
100011 1 tcp 906 rquotad
100011 2 tcp 906 rquotad
100003 2 udp 2049 nfs
100003 3 udp 2049 nfs
100003 4 udp 2049 nfs
100003 2 tcp 2049 nfs
100003 3 tcp 2049 nfs
100003 4 tcp 2049 nfs
100005 1 udp 925 mountd
100005 1 tcp 928 mountd
100005 2 udp 925 mountd
100005 2 tcp 928 mountd
100005 3 udp 925 mountd
100005 3 tcp 928 mountd

Mandatory NFS services was working correctly as seen above rpc report.I didn't have a firewall between these machines or iptables on any of them.This was not the problem also here.I haven't check nfs client side because it was mounting other nfs server's mount points without a problem at all. So, I have focused on nfs server side. Restarting nfsd service or re-exporting nfs file systems with exportfs didn't work out. I can see nfs clients that mounted server's nfs mount by looking in /var/lib/nfs/rmtab file (showmount -a) on server. But I couldn't access /proc/fs/nfsd/ directory. It was not mounted on nfs server.So,I manually mounted it by using following mount command.

mount -t nfsd nodev /proc/fs/nfsd

After mounting /proc/fs/nfsd manually, I was able to mount the nfs server from client side again.

If you look at into /etc/modprobe.conf.dist file, you can see that when kernel installs module nfsd , it mounts /proc/fs/nfsd and /var/lib/nfs/rpc_pipefs mount points.

install nfsd /sbin/modprobe --first-time --ignore-install nfsd &&
{ /bin/mount -t nfsd nfsd /proc/fs/nfsd > /dev/null 2>&1 || :; }
install sunrpc /sbin/modprobe --first-time --ignore-install sunrpc &&
{ /bin/mount -t rpc_pipefs sunrpc /var/lib/nfs/rpc_pipefs >
/dev/null 2>&1 || :; }


Anonymous said...

Thanks for this post. I just had this exact same problem and for the life of me I could not figure this out. Don't think I would have ever figured it out without your blog.

Anonymous said...

I also encountered the exact same problem. I had tried all means and still could not solve the problem. I tried googled for solutions and hints, but all gave the normal suggestions of checking /etc/exports etc. Only your blog gave the right solution that save my day.
Thank you for your excellent blog