Unix enabled large amount of users now system slowness is occurring. What is the recommended setup? (4293615)

Unix enabled large amount of users now system slowness is occurring. What is the recommended setup?

Experiencing a severe slowdown when running vastool commands after unix enabling large amount of users.

To rejoin a server to the domain now takes approximately 15 to 30 minutes per server. Doing a vastool flush also takes approximately 15 minutes per server.

The slowness occurs because Authentication Services (QAS) is like a mini Domain Controller (DC). It is caching all Active Directory (AD) information that is unix related and keeping it up to date. When the amount of unix enabled users are over ten thousand slowness occurs.

We recommend the below configuration on the system when unix enabled users are over ten thousand. In workstation mode, Authentication Services does not pre-cache users unless told. Then when a request for a user is received it finds the information in AD, and caches it. This way it is more like a windows client, only asking AD for information when it needs it. By default all groups will still be loaded, the group-search-path option can still be used to reduce your preloaded groups.

RECOMMENDED CONFIGURATION:

1) Join in workstation mode
2) Set workstation-mode-users-preload to groups of system admins and whoever actively uses the system.
3) lazy-cache-update-interval = 120
4) ws-resolve-uid = true

workstation-mode-users-preload:
When workstation mode is enabled QAS does not use the user-search-path settings to precache all of the accounts in specified OUs (or the whole domain by default). Instead, users who should be preloaded can be added to AD groups specified by this setting.

lazy-cache-update-interval:
Normally QAS asks AD for changes every 15-30 minutes, for both users and groups. As the number of users increase, the chances somethings changed increases. For example, with 30K users and a 30 day password change policy then on average 1000 users are changing their password daily updating their account and making QAS reload all of their account data.

In workstation mode user incremental updates are not done. When a user logs in a by-name update is done on them, so they now have the latest information when it matters, they are logging in. For small environments the more proactive default behaviour is useful, but when the environment gets too large this noise can get to the point where it impacts network activity.

ws-resolve-uid:
By default QAS will only resolve an unknown account that is requested by name, if getpwuid() is called to lookup an unknown UID the lookup will fail. If ws-resolve-uid is enabled vasd will try to resolve the UID against Active Directory and cache the user with the given UID so that the calls to getpwuid() will succeed. Enabling this option will cause getpwuid() to take longer when resolving the UID, but once resolved the UID value is cached and subsequent lookups will be much faster. A cache for unresolvable UIDs is also used to avoid excessive lookups for unknown information. 

The exact number where these effects are seen depends on environmental tolerance for additional users. Typically issues are noticed when the number of unix enabled users reach numbers in the tens of thousands. There is no specific threshold for this however. Once a large number of users are unix enabled in AD anything that changes them at the same time will potentially cause issues when Authentication Services needs to update them all. For example if all users are added to a group or an Exchange policy. So anything that changes a lot of users USN value. This could lead to temporary outages as Authentication Services processes these changes. These issue can be avoided or mitigated by using workstation mode.