Upgrading a DSS to 5.9.x is only possible from Defender 5.8.1 or Defender 5.8.2. A direct upgrade to Defender 5.9.1 on versions earlier then 5.8.1 is not supported and may cause Defender components to stop functional properly..
Delete the dtdsk.dll and aal2dsk.dll from the C:\Program Files (x86)\Quest Software\Defender\DSS Active Directory Edition directory may allow the the service to start.
Revert from a snapshot or backup prior to the upgrade. Upgrade the Defender 5.7 or earlier components to Defender 5.8.2 first, and then upgrade to Defender 5.9.1
Install Defender 5.9.1 onto a different machine.