These messages can be expected and may be related to the Health Checking. Regular queries are performed against each backend server within a Server Group, and if the operation fails for a number of attempts or fails to respond within a given time, the server is marked as 'down' until the server responds during a subsequent health check (this doesn't mean necessarily, that the server was really down; although it could have been)
Reviewing the STDOUT log file, may see that some servers are marked administratively as 'down'.
It is very probable that there may be some clients which don't close connections.
In that case adding some kind of "FE" timeout to close them may help.
1. Go to "Global Parameters -> Global Properties" and activate the "Define Default FE Timeout" checkbox.
For the amount of time you can choose 120 seconds.
2) Restart the instance.
In addition to this, the last two versions of QCVDS (6.0.4 and 6.0.5) have fixes related with 'Simultaneous connections limit' issues. And, regarding the Administration port, these versions already include a "Use FE Timeout" checkbox at the 'Administration' section, in case you are experiencing a lot of connections to this port that are not closed.