One Identity Safeguard for Privileged Passwords 2.9 - Administration Guide

Disaster recovery and clusters

Safeguard for Privileged Passwords Appliances can be clustered to ensure high availability. Clustering enables the recovery or continuation of vital technology infrastructure and systems following a natural or human-induced disaster. This reduces down time and data loss.

Another benefit of clustering is load distribution. Clustering in a managed network ensures the load is distributed to ensure minimal cluster traffic and to ensure appliances that are closest to the target asset are used to perform the task. The Appliance Administrator defines managed networks (network segments) to effectively manage assets, account, and service access requests in a clustered environment to distribute the task load.

Primary and replica appliances

A Safeguard for Privileged Passwords cluster consists of three or five appliances. An appliance can only belong to a single cluster. One appliance in the cluster is designated as the "primary". Non-primary appliances are referred to as "replicas". All vital data stored on the primary appliance is also stored on the replicas. In the event of a disaster, where the primary appliance is no longer functioning, you can promote a replica to be the new primary appliance. Network configuration is done on each unique appliance, whether it is the primary or a replica.

The replicas provide a read-only view of the security policy configuration. You cannot add, delete, or modify the objects or security policy configuration on a replica appliance. You can perform password change and check operations and make password release and session access requests. Users can log into replicas to request access, generate reports or audit the data. Also, passwords and sessions can be requested from any appliance in a Safeguard cluster.

Supported cluster configurations

Current supported cluster configurations follow.

• 3 Node Cluster (1 Primary, 2 Replicas): Consensus is achieved when two of the three appliances are online and able to communicate. Valid states are: Online or ReplicaWithQuorum. For more information, see Appliance states.
• 5 Node Cluster (1 Primary, 4 Replicas): Consensus is achieved when three of the five appliances are online and able to communicate. Valid states are: Online or ReplicaWithQuorum. For more information, see Appliance states.

Consensus and quorum failure

Some maintenance tasks require that the cluster has consensus (quorum). Consensus means that the majority of the members (primary or replica appliances) are online and able to communicate. Valid states are: Online or ReplicaWithQuorum. For more information, see Appliance states.

Supported clusters have an odd number of appliances so the cluster has a consensus equal to or greater than 50% of the appliances are online and able to communicate.

If a cluster loses consensus (also known as a quorum failure), the following automatically happens:

• The primary appliance goes into Read-only mode.
• Password check and change is disabled.

When connectivity is restored between a majority of members in a cluster, consensus is automatically regained. If the consensus members include the primary appliance, it automatically converts to read-write mode and enables password check and change.

Health checks and diagnostics

The following tools are available to perform health checks and diagnose the cluster and appliances.

• Perform a health check to monitor cluster health and appliance states. For more information, see Maintaining and diagnosing cluster members.
• Diagnose the cluster and appliance. You can view appliance information, run diagnostic tests, view and edit network settings, and generate a support bundle. For more information, see Diagnosing a cluster member.

Shut down and restart an appliance

You can shut down and restart an appliance.

• Shut down an appliance. For more information, see Shutting down the appliance.
• Restart an appliance. For more information, see Restarting the appliance.

Run access request workflow on an isolated appliance in Offline Workflow Mode

You can enable Offline Workflow Mode either automatically or manually to force an appliance which no longer has quorum to process access requests using cached policy data in isolation from the remainder of the cluster. The appliance will be in Offline Workflow Mode.

• For general information on Offline Workflow Mode, see About Offline Workflow Mode.
• To manually enable offline workflow or manually resume online workflow, see Manually control Offline Workflow Mode.
• To configure automatic Offline Workflow Mode and, optionally, automatically Resume Online Workflow, see Offline Workflow (automatic). When automation is turned on, you can still also manually control Offline Workflow Mode.

Primary appliance failure: failover and backup restore

If a primary is not communicating, perform a manual failover. If that is not possible, you can use a backup to restore an appliance.

• Failover: If the primary is not communicating, you can perform a manual failover if there is a quorum (the majority have consensus). For more information, see Failing over to a replica by promoting it to be the new primary.

Enrolling replicas into a cluster

Prior to the Appliance Administrator enrolling cluster members into a Safeguard for Privileged Passwords cluster, review the enrollment considerations which follow.

Considerations to enroll cluster members

• If there is an appliance in Offline Workflow Mode, resume online operations before adding another replica. For more information, see About Offline Workflow Mode.
• Update all appliances to the same appliance build (patch) prior to building your cluster. During the cluster patch operation, access request workflow is available so authorized users can request password releases and session access.
• To enroll an appliance into a cluster, the appliance must communicate over port 655 UDP/TCP and port 443 TCP and must have IPv4 or IPv6 network addresses (not mixed). For more information, see Appendix A: Safeguard ports.
• You can only enroll replica appliances to a cluster when logged into the primary appliance (using an account with Appliance Administrator permissions).
• You can only add one appliance at a time. The maintenance operation must be complete before adding additional replicas.
• Enrolling a replica can take as little as 5 minutes or as long as 24 hours depending on the amount of data to be replicated and your network.

• During an "enroll replica" operation, the replica appliance goes into Maintenance mode. The existing members of the cluster can still process access requests as long as the member has quorum. On the primary appliance, you will see an "enrolling" notice in the status bar of the cluster view, indicating that a cluster-wide operation is in progress. This cluster lock prevents you from doing additional maintenance activities.

  Once the maintenance operation (enroll replica operation) is complete, the diagram in the cluster view (left pane) shows the link latency on the connector. The appliances in the cluster are unlocked and users can once again use the features available in Safeguard for Privileged Passwords.

  TIP: The Activity Center contains events for the start and the completion of the enrollment process.

• The primary appliance's objects and security policy configuration are replicated to all replica appliances in the cluster. Any objects (such as users, assets, and so on) or security policy configuration defined on the replica will be removed during enroll. Existing configuration data from the primary will be replicated to the replica during the enroll. Future configuration changes on the primary are replicated to all replicas.

To enroll a replica

1. It is recommended that you make a backup of your primary appliance before enrolling replicas to a cluster.
2. Log into the primary appliance as an Appliance Administrator.
3. In Administrative Tools, navigate to Settings | Cluster | Cluster Management.
4. Click  Add Replica to join a Safeguard for Privileged Passwords Appliance to a cluster. 
5. In the Add Replica dialog, enter a network DNS name or the IP address of the replica appliance into the Network Address field and click Connect.

6. Safeguard for Privileged Passwords connects to the replica and displays the login screen for the replica appliance.

   1. Enter a valid account with Appliance Administrator permissions.

   2. In the Add Replica confirmation dialog, enter the words Add Replica and click OK to proceed with the operation.

   Safeguard for Privileged Passwords displays (synchronizing icon) and (lock icon) next to the appliance it is enrolling and puts the replica appliance in Maintenance mode while it is enrolling into the cluster.

   On all of the appliances in the cluster, you will see an "enrolling" banner at the top of the cluster view, indicating that a cluster-wide operation is in progress and all appliances in the cluster are locked down.

   Once the maintenance operation (enroll replica operation) is complete, the diagram in the cluster view (left pane) shows the link latency on the connector. The appliances in the cluster are unlocked and users can once again make access requests.

7. Log into the replica appliance as the Appliance Administrator.

   Notice that the appliance has a state of Replica (meaning it is in a Read-Only mode); and contains the objects and security policy configuration defined on the primary appliance.

Unjoining replicas from a cluster

Safeguard for Privileged Passwords allows the Appliance Administrator to unjoin replica appliances from a cluster. Prior to unjoining a replica from a Safeguard for Privileged Passwords cluster, review the unjoin considerations which follow.

Considerations to unjoin cluster members

• You can only unjoin replica appliances from a cluster.

• To promote a replica to be the new primary and then unjoin the 'old' primary appliance, you can use the Failover option if the cluster has consensus (the majority of the appliances are online and able to communicate). For more information, see Failing over to a replica by promoting it to be the new primary. If the cluster does not have consensus, use the Cluster Reset option to rebuild your cluster. For more information, see Resetting a cluster that has lost consensus.
• To perform an unjoin operation, the replica appliance to be unjoined can be in any state; however, the remaining appliances in the cluster must achieve consensus (online and able to communicate).
• You can unjoin a replica appliance when logged into any appliance in the cluster that is online using an account with Appliance Administrator permissions.

• When you unjoin a replica appliance from a cluster, the appliance is removed from the cluster as a stand-alone appliance that retains all of the data and security policy configuration information it contained prior to being unjoined. After the replica is unjoined, the appliance is placed in a Read-only mode with the functionality identified in Read-only mode functionality. You can activate an appliance in Read-only mode so you can add, delete and modify data, apply access request workflow, and so on. For more information, see Activating a read-only appliance.

To unjoin a replica from a cluster

1. Log into an appliance in the cluster, as an Appliance Administrator.
2. In Administrative Tools, navigate to Settings | Cluster | Cluster Management.
3. In the cluster view (left pane), select the replica node to be unjoined from the cluster.
4. In the details view (right pane), click Unjoin.

5. In the Unjoin confirmation dialog, enter the word Unjoin and click OK to proceed.

   Safeguard for Privileged Passwords displays (synchronizing icon) and (lock icon) next to the appliance it is unjoining and puts the replica appliance in Maintenance mode while it is unjoining from the cluster.

   Once the operation has completed, the replica appliance no longer appears in the cluster view (left pane).

Maintaining and diagnosing cluster members

When a node is selected in the Cluster view (left pane) of the Cluster settings page, the appliance details and cluster health view (right pane) displays details about the selected appliance. From this pane you can run the following maintenance and diagnostic tasks against the selected appliance.

• Unjoin: Click Unjoin to remove a replica from the cluster. For more information, see Unjoining replicas from a cluster.
• Failover: Click Failover to promote a replica to the primary appliance. For more information, see Failing over to a replica by promoting it to be the new primary.

• Activate: Click Activate to activate a Read-only appliance so it can add, modify and delete data. For more information, see Activating a read-only appliance.

  CAUTION: Activating an appliance that is in Read-Only mode will take it out of the Read-only state and enable password check and change for managed accounts. Ensure that no other Safeguard for Privileged Passwords Appliance is actively monitoring these accounts, otherwise access to managed accounts could be lost.

• Diagnose: Click Diagnose to open the Diagnostics pane where you can perform the following:
  • View appliance information. For more information, see Appliance Information.
  • Run diagnostic tests against the appliance. For more information, see Network Diagnostics.
  • Perform a factory reset. For more information, see Factory Reset from the desktop client.
  • View or edit networking settings. For more information, see Networking.
  • Generate a support bundle. For more information, see Support Bundle.
  • View or edit time settings. For more information, see Time.
• Check Health: Click Check Health to capture and display the current state of the selected appliance.
• Restart: Click Restart to restart the selected appliance. Confirm your intentions by entering a Reason and clicking Restart.

To fix more serious issues with a cluster, you can perform additional operations depending on the state of the cluster members. Some such operations include:

• Patching cluster members
• Using a backup to restore a clustered appliance
• Performing a factory reset
• Resetting a cluster that has lost consensus
• About Offline Workflow Mode