Using a backup to restore a clustered appliance
In a clustered environment, the objective of a cluster backup is to preserve and allow the restoration of all operational data, including access request workflow, users/accounts, audit logs, and so on. All appliances in a cluster (primary and replicas) can be backed up. However, a backup should only be restored to an appliance in the worst-case scenario where no appliance can be restored using the failover operation.
When a backup is restored to an appliance, the restore on the primary clears the primary's cluster configuration but does not change the replicas' cluster configuration. To avoid issues:
- If possible, unjoin the replicas from the cluster prior to a backup restore.
-
If the primary has been set to encrypt the cluster backups with a password or GPG key, you must have the password or GPG private key to complete the upload and restore operation. For more information, see Backup protection settings..
- Upload and restore the backup on the appliance that will be the primary.
- If you did not unjoin the replicas prior to the backup restore, perform a cluster reset on each replica so they become standalones then join the replicas back into the cluster.
The appliance is restored as a stand-alone primary appliance in Read-only mode with no replicas. However, all the access request workflow, user/account, and audit log data that existed when the backup was taken is retained. This primary appliance can then be activated and replicas can be joined to recreate a cluster.
To take a backup of a physical appliance
- Log in to the appliance as an Appliance Administrator.
- Go to Safeguard Backup and Restore:
web client: Navigate to 
Backup and Retention > Safeguard Backup and Restore.
-
As needed, you can run a backup, set a schedule for the backup, and encrypt the backup for a cluster from the primary. For more information, see Backup and Restore..
To restore a physical appliance from a backup
An Appliance Administrator can restore backups as far back as Safeguard for Privileged Passwords version 8.0.0.21290. Only the data is restored; the running version is not changed.
You cannot restore a backup from a version newer than the one running on the appliance. The restore will fail and a message like the following displays: Restore failed because backup version [version] is newer then the one currently running [version].
The backup version and the running version display in the Activity Center logs that are generated when SPP starts, completes, or fails a restore.
NOTE: If you want to use a backup file taken on a different appliance, that backup file must first be downloaded on the appliance where the backup was taken. The downloaded backup file will then need to be uploaded to the appliance that wants to use it before you can use the Restore option.
- Log in to the appliance to be restored as an Appliance Administrator.
- Go to Safeguard Backup and Restore:
- web client: Navigate to Backup and Retention > Safeguard Backup and Restore.
-
Select the backup to be used and click Restore. If a problematic condition is detected, 
Warning for Restore of Backup displays along with details in the Restore Warnings, Warning X of X message. Click Cancel to stop the restore process and address the warning or click Continue to move to the next warning (if any) or complete the process.
-
If the backup is protected by a password, the Protected Backup Password dialog displays. Type in the password in the Enter Backup Password text box. For more information, see Backup protection settings..
- When the Restore dialog displays, enter the word Restore and click OK. For more information, see Restore a backup..
The appliance is restored as a stand-alone primary appliance in Read-only mode with no replicas.
To rebuild a cluster
- Log in to the primary appliance as an Appliance Administrator.
- Activate the Read-only primary appliance.
- Go to Cluster Management:
- web client: Navigate to
Cluster > Cluster Management.
- Select the node to be activated from the cluster view (left pane).
- Click
Activate.
-
Confirm the activate operation.
For more information, see Activating a read-only appliance..
- One at a time, enroll the replica appliances to rebuild your cluster.
- Go to Cluster Management:
- web client: Navigate to Cluster > Cluster Management.
-
Click 
Add Replica to join a replica appliance to the cluster.
Once the enroll operation completes, repeat to add your appliances back into the cluster as replicas.
NOTE: Enrolling a replica can take up to 24 hours depending on the amount of data to be replicated and your network.
For more information, see Enrolling replicas into a cluster..
Resetting a cluster that has lost consensus
Resetting the cluster configuration allows you to recover a cluster that has lost consensus. If the cluster regains consensus after connectivity is restored, the primary will return to Read-Write mode and password and SSH key check and change will be reenabled. However, if it does not regain consensus, the Appliance Administrator must perform a cluster reset to force-remove nodes from the cluster.
If you are concerned about network issues, reset the cluster with only the new primary appliance. Once the cluster reset operation is complete, enroll appliances one by one to create a new cluster.
|
|
Caution: Resetting a cluster should be your last resort. It is recommended that you restore from a backup rather than reset a cluster. |
Cautions
To avoid issues, consider the following cautions.
- Only reset the cluster if you are certain that consensus has been lost; otherwise, you could introduce a split-brain scenario. (Split-brain scenario is where a cluster gets divided into smaller clusters. Each of these smaller clusters believes it is the only active cluster and may then access the same data which could lead to data corruption.)
- Ensure that no cluster member has Offline Workflow Mode enabled. For more information, see For more information, see Offline Workflow (automatic).. You can ensure that no cluster member is in Offline Workflow Mode on Cluster > Cluster Management. For more information, see For more information, see Cluster Management..
To reset a cluster
- Go to Cluster:
- web client: Navigate to Cluster > Cluster Management.
-
Click the 
Reset Cluster button.
The Reset Cluster dialog displays, listing the appliances (primary and replicas) in the cluster.
-
In the Reset Cluster dialog, select the nodes to be included in the reset operation and use the Set Primary button to designate the primary appliance in the cluster.
NOTE: Nodes must have an appliance state of Online or Online Read-only and be able to communicate to be included in the reset operation. If you select a node that is not online or not available, you will get an error and the reset operation will fail.
- Click Reset Cluster.
-
In the confirmation dialog, enter the words Reset Cluster and click OK.
When connected to the new primary appliance, the Configuring Safeguard for Privileged Passwords Appliance progress page displays, showing the steps being performed as part of the maintenance task to reset the cluster.
- Once the maintenance tasks have completed, click Restart.
- If an appliance is cluster reset as a standalone appliance, it will be placed in StandaloneReadonly mode (not online) and will require activation to avoid a split-brain scenario. For more information, see Activating a read-only appliance..
Once reset, the cluster only contains the appliances that were included in the reset operation.
Performing a factory reset
As an Appliance Administrator, you can use the Factory Reset feature to reset a Safeguard for Privileged Passwords Appliance to recover from major problems or to clear the data and configuration settings on the appliance. A factory reset of a physical appliance may be initiated from:
- The web client
- The Recovery Kiosk
- The Support Kiosk
- Using the API
A Safeguard for Privileged Passwords virtual appliance is reset by the recovery steps to redeploy and not a factory reset. For more information, see Virtual appliance backup and recovery..
|
|
Caution: Care should be taken when performing a factory reset against a physical appliance, because this operation removes all data and audit history, returning it to its original state when it first came from the factory. Performing a factory reset will NOT reset the BMC/IPMI interface or the IP address. However, the BMC/IPMI interface will need to be reenabled after the reset has completed (for more information, see Lights Out Management (BMC)). The appliance must go through configuration again as if it had just come from the factory. For more information, see Setting up Safeguard for Privileged Passwords for the first time..
In addition, performing a factory reset may change the default SSL certificate and default SSH host key.
The appliance resets to the current Long Term Support (LTS) version. For example, if the appliance is running version 6.6 (feature release) or 6.0.6 LTS (maintenance Long Term Support release) and then factory reset, the appliance will reset down to 6.0 LTS and you will have to patch up to your desired version. For more information, see Long Term Support (LTS) and Feature Releases.. |
Factory reset on a clustered appliance
Performing a factory reset on a clustered hardware appliance will not automatically remove the appliance from a cluster. The recommended best practice is to unjoin an appliance from the cluster before performing a factory reset on the appliance. After the unjoin and factory reset, the appliance must be configured again. For more information, see Setting up Safeguard for Privileged Passwords for the first time..
To perform a factory reset from the web client
- Go to Factory Reset on hardware (not virtual machine):
- Navigate to Appliance Management > Appliance > Factory Reset.
- Click Factory Reset.
-
In the Factory Reset confirmation dialog, enter the words Factory Reset and click Factory Reset.
The appliance will go into Maintenance mode to revert the appliance. If the appliance was in a cluster, you may need to unjoin the factory reset appliance. The factory reset appliance must be configured again. For more information, see Setting up Safeguard for Privileged Passwords for the first time.. In addition, when you log in to the appliance, you will be prompted to add your Safeguard for Privileged Passwords licenses.
To perform a factory reset from the Recovery Kiosk
|
|
CAUTION: As part of the factory reset process, you will be performing a challenge response operation.
If the challenge response operation is invalidated, try restarting the process to generate a new challenge response. If that fails, contact One Identity Support for assistance. |
- To perform a hardware factory reset, go to the Recovery Kiosk. For more information, see Recovery Kiosk (Serial Kiosk)..
-
Select Factory Reset.
-
Press the right arrow.
-
At Name or Email, enter your email or name and press the Tab key (or down arrow).
-
Select Submit.
-
At View Challenge, press the Enter key. Safeguard for Privileged Passwords produces a challenge. (If the challenge is not shown, maximize Putty.)
-
Copy and paste the challenge into a text document and send it to One Identity Support. A challenge response is only good for 48 hours.
IMPORTANT: Do not reboot the machine during the challenge response process.
- When you get the response from One Identity Support, copy and paste the response into the kiosk screen and select Factory Reset. The response is only valid for 24 hours from when it was generated by One Identity.
-
Once the factory reset is completed the appliance will need to be reconfigured.
See the following Knowledge Base Article for details on using the MGMT network interface for factory reset: KB 232766: What are the steps to perform a factory reset from the recovery kiosk or MGMT network interface on physical devices?
To perform a factory reset from the Support Kiosk
|
|
CAUTION: As part of the factory reset process, you will be performing a challenge response operation. To avoid invalidating the challenge response, do NOT navigate away from the page or refresh.
If the challenge response operation is invalidated, try restarting the process to generate a new challenge response. If that fails, contact One Identity Support for assistance. |
- To perform a hardware factory reset, on the web management console, click
Support Kiosk. For more information, see Support Kiosk..
- Select Factory Reset. (This option is not available if you are attached to the console of a virtual machine. The options is only available for hardware.)
-
Complete the challenge/response process:
- In Full Name or Email, enter your name or email to receive the challenge question.
- Click Get Challenge.
- To get the challenge response, perform one of the following (see the illustration that follows).
- When you get the response from One Identity Support, copy and paste the response into the kiosk screen and select Factory Reset.
Unlocking a locked cluster
In order to maintain consistency and stability, only one cluster operation can run at a time. To ensure this, Safeguard for Privileged Passwords locks the cluster while a cluster operation is running, such as enroll, unjoin, failover, patch, reset, session module join, update IP, and audit log maintenance. While the cluster is locked, changes to the cluster configuration are not allowed until the operation completes.
The lock notification displays as follows:
- web client: The Appliance State will show a red lock icon (
).
You should never cancel the cluster lock for an SPP unjoin, failover, cluster reset, restore, patch, or IP address update. Other considerations:
- If a SPP join (enroll) is taking a long time, you may cancel it during the streaming audit data step.
- If a patch distribution is taking a long time, you may cancel it and upload the patch to the replicas directly.
- If an audit log synchronize operation is taking a long time, or you have reason to believe it will not complete due to a down appliance in the cluster, you may cancel it. Canceling this operation requires monitoring as detailed in Cancel Audit Log Maintenance from the Audit Log Maintenance page.
-
If an audit log archive or purge operation is taking a long time, or you have reason to believe it will not complete due to a down appliance in the cluster, you may cancel it. Canceling this operation requires monitoring as detailed in Cancel Audit Log Maintenance from the Audit Log Maintenance page.
To unlock a locked cluster
- Go to Cluster Management:
- web client: Navigate to Cluster > Cluster Management.
- Click the lock icon in the upper right corner of the warning banner.
-
In the Unlock Cluster confirmation dialog, enter Unlock Cluster and click OK.
This will release the cluster lock that was placed on all of the appliances in the cluster and close the operation.
IMPORTANT: Care should be taken when unlocking a locked cluster. It should only be used when you are sure that one or more appliances in the cluster are offline and will not finish the current operation. If you force the cluster unlock, you may cause instability on an appliance, requiring a factory reset and possibly the need to rebuild the cluster. If you are unsure about the operation in progress, do NOT unlock the cluster.
