User Authentication stops working. vastool flush ERROR: Error receiving IPC buffer from server: Invalid argument (4295799)

Symptoms:

1 - Users unable to login on multiple server.

2 -  vasd records multiple errors in /var/log/messages: eg  vasd[4317]: Dispatcher::AdaptiveChildManagement: Not spawning additional process, ChildMax Limit reached. Then we see additional vasd and vasypd processes start up, owned by root. This seems to be the point when end users can't login.

3 - Restarting vasd temporarily resolves the issue

Network DNS issue.  The vasd process is stuck on a DNS query. This could be caused by dropped dns packets or dns timeouts.

TROUBLESHOOTING COMMANDS:

1 - /opt/quest/bin/vastool info processes

2 - ps -ef | grep vasd 

3 - for p in `/opt/quest/bin/vastool info processes | cut -d: -f2`; do echo $p; pstack $p | grep nquerydomain; done

Within this data it will show the vasd process doing the srv dns query. 

RESOLUTION:

Involve the network admin team to investigate DNS timeout or dropped packets.