Prior to the Appliance Administrator enrolling cluster members into a Safeguard for Privileged Passwords cluster, review the enrollment considerations that follow.
Considerations to enroll cluster members
- If there is an appliance in Offline Workflow Mode, resume online operations before adding another replica. For more information, see About Offline Workflow Mode.
- Update all appliances to the same appliance build (patch) prior to building your cluster. During the cluster patch operation, access request workflow is available so authorized users can request password and SSH key releases and session access.
- To enroll an appliance into a cluster, the appliance must communicate over port 655 UDP and port 443 TCP, and must have IPv4 or IPv6 network addresses (not mixed). If both IPv4 and IPv6 are available for the connection then IPv6 will be used.For more information, see Safeguard ports.
- You can only enroll replica appliances to a cluster when logged in to the primary appliance (using an account with Appliance Administrator permissions).
- You can only add one appliance at a time. The maintenance operation must be complete before adding additional replicas.
- Enrolling a replica can take as little as five minutes or as long as 24 hours depending on the amount of data to be replicated and your network.
During an enroll replica operation, the replica appliance goes into Maintenance mode. The existing members of the cluster can still process access requests as long as the member has quorum. On the primary appliance, you will see an enrolling notice in the status bar of the cluster view, indicating that a cluster-wide operation is in progress. This cluster lock prevents you from doing additional maintenance activities.
Once the maintenance operation (enroll replica operation) is complete, the diagram in the cluster view (left pane) shows the link latency on the connector. The appliances in the cluster are unlocked and users can once again use the features available in Safeguard for Privileged Passwords.
TIP: The Activity Center contains events for the start and the completion of the enrollment process.
The primary appliance's objects and security policy configuration are replicated to all replica appliances in the cluster. Any objects (such as users, assets, and so on) or security policy configuration defined on the replica will be removed during enroll. Existing configuration data from the primary will be replicated to the replica during the enroll. Future configuration changes on the primary are replicated to all replicas.
To enroll a replica
- It is recommended that you make a backup of your primary appliance before enrolling replicas to a cluster.
- Log in to the primary appliance as an Appliance Administrator.
- Go to Cluster Management:
- web client: Navigate to Cluster > Cluster Management.
- Click Add Replica to join a Safeguard for Privileged Passwords Appliance to a cluster.
- In the Add Replica dialog, enter a network DNS name or the IP address of the replica appliance into the Network Address field, and click Connect.
Your web browser redirects to the login page of the replica. Log in as normal, including any two-factor authentication. After successful log in, your web browser is redirected back to the web client.
- Enter a valid account with Appliance Administrator permissions.
In the Add Replica confirmation dialog, enter the words Add Replica and click OK to proceed with the operation.
Safeguard for Privileged Passwords displays (synchronizing icon) and (lock icon) next to the appliance it is enrolling and puts the replica appliance in Maintenance mode while it is enrolling into the cluster.
On all of the appliances in the cluster, you will see an "enrolling" banner at the top of the cluster view, indicating that a cluster-wide operation is in progress and all appliances in the cluster are locked down.
View the link latency: Once the maintenance operation (enroll replica operation) is complete, click on an appliance to see the link latency. The appliances in the cluster are unlocked and users can once again make access requests.
- Log in to the replica appliance as the Appliance Administrator.
Notice that the appliance has a state of Replica (meaning it is in a Read-Only mode) and contains the objects and security policy configuration defined on the primary appliance.
Safeguard for Privileged Passwordsallows the Appliance Administrator to unjoin replica appliances from a cluster. Prior to unjoining a replica from a Safeguard for Privileged Passwords cluster, review the unjoin considerations that follow.
Considerations to unjoin cluster members
You can only unjoin replica appliances from a cluster.
- To promote a replica to be the new primary and then unjoin the 'old' primary appliance, you can use the Failover option if the cluster has consensus (the majority of the appliances are online and able to communicate). For more information, see Failing over to a replica by promoting it to be the new primary. If the cluster does not have consensus, use the Cluster Reset option to rebuild your cluster. For more information, see Resetting a cluster that has lost consensus.
- To perform an unjoin operation, the replica appliance to be unjoined can be in any state; however, the remaining appliances in the cluster must achieve consensus (online and able to communicate).
- You can unjoin a replica appliance when logged in to any appliance in the cluster that is online, using an account with Appliance Administrator permissions.
When you unjoin a replica appliance from a cluster, the appliance is removed from the cluster as a stand-alone appliance that retains all of the data and security policy configuration information it contained prior to being unjoined. After the replica is unjoined, the appliance is placed in a Read-only mode with the functionality identified in Read-only mode functionality. You can activate an appliance in Read-only mode so you can add, delete and modify data, apply access request workflow, and so on. For more information, see Activating a read-only appliance.
To unjoin a replica from a cluster
- Log in to an appliance in the cluster, as an Appliance Administrator.
- Go to Cluster Management:
- web client: Navigate to Cluster > Cluster Management.
- In the cluster view on the left, select the replica node to be unjoined from the cluster.
- In the details view on the right, click Unjoin.
In the Unjoin confirmation dialog, enter the word Unjoin and click OK to proceed.
Safeguard for Privileged Passwords displays (synchronizing icon) and (lock icon) next to the appliance it is unjoining and puts the replica appliance in Maintenance mode while it is unjoining from the cluster.
Once the operation has completed, the replica appliance no longer appears.
Login during Maintenance mode
If you log in to the replica appliance while Safeguard for Privileged Passwords is processing an unjoin operation, you will see the Maintenance mode screen. At the end of the Maintenance mode, there will be a button indicating that the unjoin operation completed successfully:
- web client: Continue
Maintain and diagnosis cluster members from Cluster Management:
- web client: Navigate to Cluster > Cluster Management.
When a node is selected in the Cluster view, the right of the pane displays details about the selected appliance. From this pane you can run the following maintenance and diagnostic tasks against the selected appliance.
To fix more serious issues with a cluster, you can perform additional operations depending on the state of the cluster members. Some such operations include:
To ensure password and SSH key consistency and individual accountability for privileged accounts, when an appliance loses consensus in the cluster, access requests are disabled. In the event of an extended network partition, the Appliance Administrator can either automatically or manually place an appliance in Offline Workflow Mode to run access request workflow on that appliance in isolation from the rest of the cluster. When the network issues are resolved and connectivity is reestablished, the Appliance Administrator can either automatically or manually resume online operations to merge audit logs, drop any in-flight access requests, and return the appliance to full participation in the cluster.
Offline workflow considerations
- In Offline Workflow Mode, an appliance functions apart from the other members of the cluster. Users can request passwords and sessions.
- Settings for Offline Workflow are set on an individual appliance.
- Suspend/Restore account does not work in Offline Workflow mode.
Passwords and SSH keys in Offline Workflow Mode
- In Offline Workflow Mode, the appliance is enabled to request, approve, and release passwords, SSH key, and sessions without a quorum, using cached policy data.
In Offline Workflow Mode, when policy requires change after check-in, the requirement is bypassed to allow for subsequent check out. In this case, a Access Request Password or SSH Key Reset By-passed Event is generated, stating: An access request subsequent check out is available as password [or SSH key] reset was by-passed.
Password and SSH key changes will be rescheduled and will possibly complete when network connectivity is restored even while the appliance is in Offline Workflow Mode.
- Users may still request a password or SSH key from the primary or another replica on the cluster with consensus; password and SSH key check and changes works as usual. The result is that passwords or SSH keys may get out of sync on the appliance running Offline Workflow Mode. This is expected behavior and the password and SSH key will remain out of sync until the partition is healed.
On a network partition where one or more appliances are in Offline Workflow Mode, it is possible for two individuals to have the same password and SSH key at the same time. Tying actions back to a single responsible individual is not possible. It will still be possible to identify each person that had access to the password and SSH key at the time.
Policies in Offline Workflow Mode
Policy will be enforced as it existed at the time the appliance, now in Offline Workflow Mode, lost network connectivity to the rest of the cluster.
Policy requiring a password and SSH key change after check-in is bypassed and subsequent check-out from the appliance in Offline Workflow Mode is allowed.
- Policy is Read-only. Therefore, update and delete configuration operations are not allowed on the appliance in Offline Workflow Mode.
Policy changes are only allowed if directed at an online primary within the cluster. Policy changes on the online primary do not affect the appliance in Offline Workflow Mode. Once the offline workflow appliance has resumed online operations the policy changes will be distributed.
Work flow in Offline Workflow Mode
User experience: Enable Offline Workflow Mode
Users that are requesting a password and SSH key in Safeguard are returned to the Home page. Password and SSH key requests prior to the switch to Offline Workflow Mode are not displayed.
- When the switch to Offline Workflow Mode starts, this message displays: Safeguard is switching to Offline Workflow Mode. Please wait until this process is complete before proceeding with any current work. The bottom of the Home page displays this information: (Switching to Offline Workflow Mode...) and Disconnected. If the user clicks Refresh, the banner is replaced with: The service is unavailable.
- When the switch to Offline Workflow Mode is complete, a banner with this information is displayed: Safeguard is currently in Offline Workflow Mode. Previous access requests are temporarily unavailable. You may submit new requests to continue working in Offline Workflow Mode. The bottom of the Home page displays these messages: (Offline Workflow Mode) and the connection status: Connecting then Connected.
Administrators can view the workflow status on the Cluster View pane where a message like this displays: Offline Workflow Enabled (This appliance is running access workflow in isolation from the cluster.) For more information, see Cluster Management.
User experience: Resume Online Operations
When the switch to Resume Online Operations has begun, this message displays: Safeguard is returning to normal operations. Please wait until this process is complete before proceeding with any current work. The bottom of the Home page displays this information: (Returning to normal operations) and Disconnected.
Once online operations are restored, the bottom of the Home page displays this information: Connected.
The Appliance Administrator is notified when an appliance has lost consensus (quorum) via the ApplianceStateChangedEvent.
- The following events can be configured for email notifications and are written to the audit log:
- All access request notifications are still generated.
The Notification service identifies whether access workflow is available on an appliance via the IsPasswordRequestAvailable, IsSSHKeyRequsteAvailable, and IsSessionsRequestAvailable properties. The following API endpoint can be used to make this determination:
https://<hostname or IP>/service/notification/v4/Status/Availability
Audit logs in Offline Workflow Mode
Avoid modifications to the cluster configuration
It is recommended that no changes to cluster membership are made while an appliance is in Offline Workflow Mode. The online operations must be automatically or manually resumed before adding or removing other nodes to ensure the appliance can seamlessly reintegrate with the cluster.
The Appliance Administrator is advised to resume the online operations as soon as possible for individual password or SSH key accountability, policy adherence, and audit integrity.
Cluster patching is not allowed
During a cluster patch, Offline Workflow Mode cannot be triggered manually or automatically on any of the clustered appliances.
Considerations to resume online operations
- The network partition must be corrected before resuming online operations with full functionality.
You can resume online operations of an appliance in Offline Workflow Mode without a quorum. To resume online operations, it is highly recommended that network connectivity is restored between a majority of the cluster members, including the member in Offline Workflow Mode.
When resuming online operations, any access requests that are in flight on the appliance that is running in Offline Workflow Mode will be dropped.
- While it is possible to resume online operations if the appliance is not connected, making access requests will no longer be available.
Automatic versus manual workflow