Use the Cluster settings to create a clustered environment, to monitor the health of the cluster and its members, and to define managed networks for high availability and load distribution.
It is the responsibility of the Appliance Administrator or the Operations Administrator to create a cluster, monitor the status of the cluster, and define managed networks.
Before creating a Safeguard for Privileged Passwords cluster, become familiar with the Disaster recovery and clusters chapter to understand:
Go to the following:
web client: Navigate to 
Cluster.
Cluster Management allows you to create and diagnosis clusters.
When using Cluster Management from the web client, performing operations against other members of the cluster will incur a Cross-Origin Resource Sharing (CORS) HTTP request. This may require you to change the Trusted Servers, CORS, and Redirects setting to allow the specific host name being used in your web browser.
Navigate to Cluster | Cluster Management.
Cluster Management grid
- Health indicators: Health indicators display in the first column in the Cluster Management grid. Cluster members periodically query other appliances in the cluster to obtain their health information. Cluster member information and health information is cached in memory, with the most recent results displayed.
The health indicators on the nodes indicate if cluster members are in any of these states:
error: Indicates a definite problem impacting the functionality of the cluster
warning: Indicates a potential issue with the cluster
locked: Indicates the cluster is locked
(green) healthy state.
Expand the View More section to see more details.
- Name: The name of the appliance.
- Network Address: The IPv4 address (or IPv6 address) of the appliance configuration interface. You can modify the appliance IP address. For more information, see How do I modify the appliance configuration settings.
- Primary: Displays Yes if the appliance is the primary.
- Appliance State: Indicates the appliance state. For a list of available states, see Appliance states.
When you select an appliance, the details for the appliance display on the right. The grid information displays: name, network address, primary, and state. This additional information is available:
- Disk Space: The amount of used and free disk space.
- Version: The appliance version number.
- Last Health Check: Last date and time the selected appliance's information was obtained.
- Uptime: The amount of time (days, hours, and minutes) the appliance has been running.
- If the replica is selected, this additional information displays for the Primary:
- Network Address: The network DNS name or the IP address of the primary appliance in the cluster
-
MAC Address: The media access control address (MAC address), a unique identifier assigned to the network interface for communications
-
Link Present: Displays either Yes or No to indicate if there is an open communication link
-
Link Latency: The amount of time (in milliseconds) it takes for the primary to communicate with the replica. Network latency is an expression of how much time it takes for a packet of data to get from one designated point to another. Ideally, latency is as close to zero as possible.
- Errors and warnings are reported:
-
Errors: Errors are reported. For example, if an appliance is disconnected from the primary (no quorum), an error message may be: Request Workflow: Cluster configuration database health could not be determined.
-
Warnings: Warnings are reported. For example, if an appliance is disconnected from the primary (no quorum), a warning message may be: Policy Data: There is a problem replicating policy data. Details: Policy database slave IO is not running. The Safeguard primary may be inaccessible from this appliance.
Toolbar actions
In order to maintain consistency and stability, only one cluster operation can run at a time. To ensure this, Safeguard for Privileged Passwords locks the cluster while a cluster operation is running, such as enroll, unjoin, failover, patch, reset, session module join, update IP, and audit log maintenance. While the cluster is locked, changes to the cluster configuration are not allowed until the operation completes.
The lock notification displays as follows:
- web client: The Appliance State will show a red lock icon (
).
You should never cancel the cluster lock for an SPP unjoin, failover, cluster reset, restore, patch, or IP address update. Other considerations:
- If a SPP join (enroll) is taking a long time, you may cancel it during the streaming audit data step.
- If a patch distribution is taking a long time, you may cancel it and upload the patch to the replicas directly.
- If an audit log synchronize operation is taking a long time, or you have reason to believe it will not complete due to a down appliance in the cluster, you may cancel it. Canceling this operation requires monitoring as detailed in Cancel Audit Log Maintenance from the Audit Log Maintenance page.
-
If an audit log archive or purge operation is taking a long time, or you have reason to believe it will not complete due to a down appliance in the cluster, you may cancel it. Canceling this operation requires monitoring as detailed in Cancel Audit Log Maintenance from the Audit Log Maintenance page.
To unlock a locked cluster
- Go to Cluster Management:
- web client: Navigate to Cluster | Cluster Management.
- Click the lock icon in the upper right corner of the warning banner.
-
In the Unlock Cluster confirmation dialog, enter Unlock Cluster and click OK.
This will release the cluster lock that was placed on all of the appliances in the cluster and close the operation.
IMPORTANT: Care should be taken when unlocking a locked cluster. It should only be used when you are sure that one or more appliances in the cluster are offline and will not finish the current operation. If you force the cluster unlock, you may cause instability on an appliance, requiring a factory reset and possibly the need to rebuild the cluster. If you are unsure about the operation in progress, do NOT unlock the cluster.
Managed networks are named lists of network segments serviced by a specific Safeguard for Privileged Passwords (SPP) or Safeguard for Privileged Sessions (SPS) appliance in a clustered environment. Managed networks are used for scheduling tasks, such as password or SSH key change, account discovery, sessions recording, and asset discovery to distribute the task load. Using managed networks, you can:
- Distribute the load so there is minimal cluster traffic.
- Specify to use the appliances that are closest to the target asset to perform the actual task.
An SPP cluster has a default managed network that consists of all cluster members. Other managed networks can be defined.
|
|
CAUTION: If the role of a managed host that belongs to a linked SPS cluster is changed or if a managed host is added or removed from the cluster, SPP will detect the change by querying each Central Management node and attempt to stay in sync with the SPS cluster topology. If the Central Management node is down, SPP warns the administrator there may be invalid policies with a message like: The session connection policy was not found, in addition to flagging each broken Access Request Policy with an Invalid notation (Security Policy Management | Entitlements | Access Request Policies tab). Based on the size of your network and other factors, this will take one to 10 minutes and, during this time window, an unavailable managed host may continue to appear on the Managed Networks page. Any requests made will be invalid and will not be able to be launch sessions. |
Task delegation
A Safeguard for Privileged Passwords' cluster delegates platform management tasks (such as password and SSH key check and password and SSH key change) to appliances based on platform task load. The primary appliance performs delegation and evaluates cluster member suitability using an internal fitness score that is calculated by dividing the number of in-use platform task threads by the maximum number of allowed platform task threads.
The maximum number of allowed platform task threads can be adjusted using the Appliance/Settings API and adjusting the MaxPlatformTaskThreads value. By adjusting this number, you can tune task distribution.
IMPORTANT: Adjusting the MaxPlatformTaskThreads will impact SPP's available resources for handling access requests and may impact user experience. Best practice is to engage Professional Services if the value may need to be changed.
Increasing the maximum number of allowed platform task threads will decrease the fitness score thus increasing the number of tasks passed to that appliance.
The fitness score is cached and is recalculated in 8-minute intervals when the scheduler is not busy. When the scheduler is running tasks, the fitness score is calculated more frequently so the scheduler can dynamically adjust.
The selection of a Safeguard for Privileged Sessions (SPS) Appliance is primarily dependent on managed network rules. However, if there aren't any managed network rules or if the managed network rules result in more than one SPS appliances selected, a fitness score is used as the tie breaker. The fitness score is calculated based on the percentage of disk available minus the overall load average of the SPS appliance. (Load average is a Linux metric which provides a numerical indication of the overall resource capacity in use on the server.) The higher the fitness score, the more likely that the corresponding appliance will be selected.
Precedence
The selection made on the Entitlement | Access Request Policy tab takes precedence over the selections on Appliance Management | Cluster | Managed Networks page. If a Managed Networks rule includes nodes from different SPS clusters, SPP will only select the nodes from the same cluster that was assigned on the Session Settings page of the Access Request Policy tab.
IMPORTANT: Discovery, password and SSH key check and change will not work if a managed network has been configured with a subnet but is not assigned to an appliance (the appliance is blank). If the managed network does not have an assigned appliance, a message like the following displays: No appliances in network '<NameOfEmptyNetwork>' available to execute platform task request. To resolve the issue, assign at least one appliance to manage the passwords, SSH key, and/or sessions or delete the managed network entry.
Go to Managed Networks:
- web client: Navigate to Cluster | Managed Networks.
The Managed Networks page displays the following information about previously defined managed networks. Initially, this page contains the properties for the Default Managed Network, which implicitly includes all networks and is served by all appliances in the cluster.
Table 38: Managed Networks: Properties
|
Name |
The name assigned to the managed network when it was added to Safeguard for Privileged Passwords. |
| Subnets |
A list of subnets included in the managed network.
Double-click an entry in the Managed Networks grid to display details about the subnets associated with the selected managed network.
If you have linked Safeguard for Privileged Sessions, the following apply:
- Passwords Managed By: The SPP appliance ID, which includes the MAC address followed by the IP address of the node.
- Sessions Managed By: If applicable, the SPS appliance host name followed by the IP address of the SPS node.
|
|
Passwords Managed By |
The host name and IP address of the appliances and the MAC address assigned to manage the specified subnets. |
|
Sessions Managed By |
The host name and IP address of the cluster nodes. |
|
Description |
The descriptive text entered when defining the managed network. |
Use these toolbar buttons to define and maintain your managed networks.
Table 39: Managed Networks: Toolbar
 New |
Add a managed network. For more information, see Adding a managed network. |
 Delete Selected |
Remove the selected managed network from Safeguard for Privileged Passwords. You cannot delete the Default Managed Network. |
 Refresh |
Update the list of managed networks. |
 Edit |
Modify the selected managed network configuration. You can not modify the Default Managed Network. |
|
Resolve Network text box |
Locate an IP address in a managed network's list of subnets. For more information, see Resolving IP address. |
Unjoin: Click 
Unjoin to remove a replica from the cluster. 
Failover: Click 
Failover to promote a replica to the primary appliance. 
Activate: Click 
Activate to activate a read-only appliance so it can add, modify and delete data. 
Diagnose: Click 
Diagnose to open the Diagnostics pane where you can perform the following:
Check Health: Click 
Check Health to capture and display the current state of the selected appliance.
Restart: Click 
Restart to restart the selected appliance. Confirm your intentions by entering a Reason and clicking Restart. 
Reset Cluster: Reset a cluster to recover a cluster that has lost consensus. 
Enable Offline Workflow: This button is available if the appliance has lost consensus, you are logged into the selected appliance, and you have not already put the appliance in Offline Workflow Mode. The state of the appliance will be 
Resume Online Operations: This button is available if the appliance has lost consensus, you are logged into the selected appliance, and the appliance is in Offline Workflow Mode. The state of the appliance will be