Ytdyklly

I am having a lot of trouble with my NFS v4 server running over infiniband (IPoIB), every time the load gets high, the nfs server locks up and I have to reboot the server to fix the problem. I get the following messages every few seconds:

Message from syslogd@fruster at Jul 14 12:51:54 ...

  kernel:BUG: soft lockup - CPU#6 stuck for 22s! [nfsd:15806]

I have been trying to debug this and figure out what is going on, and so I checked nfsstat -s to see what was using all of the bandwidth:

Server rpc stats:

calls      badcalls   badclnt    badauth    xdrcall

241998732   0          0          0          0       



Server nfs v4:

null         compound     

3         0% 242004087 99% 



Server nfs v4 operations:

op0-unused   op1-unused   op2-future   access       close        commit       

0         0% 0         0% 0         0% 148697    0% 65006     0% 124611    0% 

create       delegpurge   delegreturn  getattr      getfh        link         

3072      0% 0         0% 10373     0% 6028483   2% 120565    0% 0         0% 

lock         lockt        locku        lookup       lookup_root  nverify      

0         0% 0         0% 0         0% 52306     0% 0         0% 0         0% 

open         openattr     open_conf    open_dgrd    putfh        putpubfh     

90478     0% 0         0% 223       0% 18760     0% 18834351  6% 0         0% 

putrootfh    read         readdir      readlink     remove       rename       

65        0% 12579510  4% 4540      0% 5         0% 24848     0% 20232     0% 

renew        restorefh    savefh       secinfo      setattr      setcltid     

23413     0% 0         0% 20241     0% 0         0% 1073      0% 66        0% 

setcltidconf verify       write        rellockowner bc_ctl       bind_conn    

66        0% 0         0% 5716220   2% 0         0% 0         0% 13        0% 

exchange_id  create_ses   destroy_ses  free_stateid getdirdeleg  getdevinfo   

101       0% 159       0% 58        0% 0         0% 0         0% 0         0% 

getdevlist   layoutcommit layoutget    layoutreturn secinfononam sequence     

0         0% 0         0% 0         0% 0         0% 0         0% 225439523 83% 

set_ssv      test_stateid want_deleg   destroy_clid reclaim_comp 

0         0% 0         0% 0         0% 0         0% 55        0% 

As you can see, 'sequence' is using 83% of the NFS bandwidth, but I have no idea what 'sequence' is, and I can't find it online anywhere because when I search for 'nfsstat -s output sequence' I get a bunch of pages telling me to 'do the following steps in sequence'.

Does anyone have any idea what 'sequence' is? Bonus points if you have a suggestion as to why NFS is freezing up.

Thanks

Definition appears to be as below.

RFC 3530 NFS v4 : Network File System (NFS) version 4 Protocol

NFS4ERR_BAD_SEQID

The sequence number in a locking request is neither the next expected number or the last number processed.

The source for nfsstat didn't reveal any more info unfortunately.

UPDATE 15th July : Possibility of Bug 651409 - BAD SEQID error messages returned by the NFS server being the culprit here.

This is Operation 53 of NFSv4.1

The SEQUENCE operation is used by the server to implement session
request control and the reply cache semantics.

Source

The operation itself doesn't seem to indicate any error or problem.