One Identity Safeguard for Privileged Passwords 2.9 - Administration Guide

About Offline Workflow Mode

To ensure password consistency and individual accountability for privileged accounts, when an appliance loses consensus in the cluster access requests are disabled. In the event of an extended network partition, the Appliance Administrator can either automatically or manually place an appliance in Offline Workflow Mode to run access request workflow on that appliance in isolation from the rest of the cluster. When the network issues are resolved and connectivity is reestablished, the Appliance Administrator can either automatically or manually resume online operations to merge audit logs, drop any in-flight access requests, and return the appliance to full participation in the cluster.

Offline workflow considerations

In Offline Workflow Mode, an appliance functions apart from the other members of the cluster. Users can request passwords and sessions.

Passwords in Offline Workflow Mode

• In Offline Workflow Mode, the appliance is enabled to request, approve, and release passwords and sessions without a quorum using cached policy data.

• In Offline Workflow Mode, when policy requires change after check-in, the requirement is by-passed to allow for subsequent check out. In this case, the Access Request Password Reset By-passed Event is generated, stating: An access request subsequent check out is available as password reset was by-passed.

• Password changes will be re-scheduled and will possibly complete when network connectivity is restored even while the appliance is in Offline Workflow Mode.

• Users may still request a password from the primary or another replica on the cluster with consensus; password check and changes works as usual. The result is that passwords may get out of sync on the appliance running Offline Workflow Mode. This is expected behavior and the password will remain out of sync until the partition is healed.

• On a network partition where one or more appliances are in Offline Workflow Mode, it is possible for two individuals to have the same password at the same time. Tying actions back to a single responsible individual is not possible. It will still be possible to identify each person that had access to the password at the time.

Policies in Offline Workflow Mode

• Policy will be enforced as it existed at the time the appliance, now in Offline Workflow Mode, lost network connectivity to the rest of the cluster.

• Policy requiring a password change after check-in is by-passed and subsequent check-out from the appliance in Offline Workflow Mode is allowed.

• Policy is Read-only. Therefore, update and delete configuration operations are not allowed on the appliance in Offline Workflow Mode.

• Policy changes are only allowed if directed at an online primary within the cluster. Policy changes on the online primary do not affect the appliance in Offline Workflow Mode. Once the offline workflow appliance has resumed online operations the policy changes will be distributed.

Work flow in Offline Workflow Mode

• Regular work flow approval rules apply.
• Time-based constraints and emergency access apply.
• For the few minutes the appliance is switching to or from Offline Workflow Mode, Application to Application and any command line password-fetching operations will be suspended.

User experience: Enable Offline Workflow Mode

Users that are requesting a password in Safeguard are returned to the Home page. Password requests prior to the switch to Offline Workflow Mode are not displayed.

• When the switch to Offline Workflow Mode starts, this message displays: Safeguard is switching to Offline Workflow Mode. Please wait until this process is complete before proceeding with any current work. The bottom of the Home page displays this information: (Switching to Offline Workflow Mode...) and Disconnected. If the user clicks Refresh, the banner is replaced with: The service is unavailable.
• When the switch to Offline Workflow Mode is complete, a banner with this information is displayed: Safeguard is currently in Offline Workflow Mode. Previous access requests are temporarily unavailable. You may submit new requests to continue working in Offline Workflow Mode. The bottom of the Home page displays these messages: (Offline Workflow Mode) and the connection status: Connecting then Connected.

Administrators can view the workflow status on the Cluster View pane where a message like this displays: Offline Workflow Enabled (This appliance is running access workflow in isolation from the cluster.) For more information, see Cluster view pane.

User experience: Resume Online Operations

When the switch to Resume Online Operations has begun, this message displays: Safeguard is returning to normal operations. Please wait until this process is complete before proceeding with any current work. The bottom of the Home page displays this information: (Returning to normal operations) and Disconnected.

Once online operations are restored, the bottom of the Home page displays this information: Connected.

Notifications

• The Appliance Administrator is notified when an appliance has lost consensus (quorum) via the ApplianceStateChangedEvent.

  • A primary will change from Online to PrimaryNoQuorum.
  • A replica will change from Online to one of the following:
    • ReplicaNoQuorum (connected to primary, does not have quorum)

    • ReplicaDisconnected (disconnected from primary, does not have quorum).

    • ReplicaWithQuorum (disconnected from primary, has quorum)

    For more information, see Appliance states.

• The following events can be configured for email notifications and are written to the audit log:

  • ClusterPrimaryQuorumLostEvent

  • ClusterPrimaryQuorumRestoredEvent

  • ClusterReplicaQuorumLostEvent

  • ClusterReplicaQuorumRestoredEvent

• All access request notifications are still generated.

• The Notification service identifies whether access workflow is available on an appliance via the IsPasswordRequestAvailable and IsSessionsRequestAvailable properties. The following API endpoint can be used to make this determination:

  https://<hostname or IP>/service/notification/v2/Status/Availability

Audit logs in Offline Workflow Mode

• Prior to network connectivity being restored, everything that happens on the appliance running in Offline Workflow Mode is only audited on that appliance.

• The audit logs merge when network connectivity is restored between the offline member and any other member in the cluster, even while in Offline Workflow Mode.
• The audit data on any cluster member operating in Offline Workflow Mode will be lost unless the appliance is returned to the cluster using the resume online operations steps.
• All cluster members that were capable of processing access and session requests must have network connectivity restored to the remainder of the cluster to ensure the cluster wide audit history is maintained.

Avoid modifications to the cluster configuration

• It is recommended that no changes to cluster membership are made while an appliance is in Offline Workflow Mode. The online operations must be automatically or manually resumed before adding or removing other nodes to ensure the appliance can seamlessly reintegrate with the cluster.

  The Appliance Administrator is advised to resume the online operations as soon as possible for individual password accountability, policy adherence, and audit integrity.

Cluster patching is not allowed

During a cluster patch, Offline Workflow Mode cannot be triggered manually or automatically on any of the clustered appliances.

Considerations to resume online operations

• The network partition must be corrected before resuming online operations with full functionality.

• You can resume online operations of an appliance in Offline Workflow Mode without a quorum. To resume online operations, it is highly recommended that network connectivity is restored between a majority of the cluster members, including the member in Offline Workflow Mode.

• When resuming online operations, any access requests that are in flight on the appliance that is running in Offline Workflow Mode will be dropped.

• While it is possible to resume online operations if the appliance is not connected, making access requests will no longer be available.

Automatic versus manual workflow

• You can configure automatic triggering of Offline Workflow Mode and automatic resumption of online workflow. For more information, see Offline Workflow (automatic).
• You can manually enable Offline Workflow Mode and manually resume online workflow. Manually control Offline Workflow Mode.

Manually control Offline Workflow Mode

The Appliance Administrator can manually control Offline Workflow Mode using the following steps. Manual intervention is possible when automatic Offline Workflow Mode is enabled. For more information, see Offline Workflow (automatic).

To manually enable Offline Workflow Mode

1. Navigate to Administrative Tools | Settings | Cluster | Cluster Management.

2. In the cluster view (left pane) of the offline appliance, click the member of the cluster that is offline.

3. In the appliance details and cluster health pane (right pane), review the errors and warnings to verify the appliance has lost consensus.

4. On the offline appliance, click Enable Offline Workflow. (This option is only available when the appliance has lost consensus with the cluster.)

   A message like the following displays:

   This appliance will run access workflow in isolation from the cluster to work around loss of consensus with the cluster. Users will be able to request, approve and release passwords and sessions via this appliance using cached data. When connectivity is restored, you should resume online operations to reintegrate this appliance with the cluster and merge audit logs.

   Type 'Enable Offline Workflow' in the box below to confirm.

   See KB263580 for more information.

5. In the dialog box, type in Enable Offline Workflow and click Enter. The appliance is in Offline Workflow Mode and enters maintenance. In the Activity Center, the Event for the appliance goes from Enable Offline Workflow Started to Enable Offline Workflow Completed.
6. You can verify that new requests are enabled and view the following health checks on the Cluster Management window:
   • If there is communication to the other members in the cluster, while connected to the member in Offline Worflow mode, a message like this displays at the top of the messages: Cluster connectivity detected. When communication is reestablished, you can manually resume online operations to the appliance.
   • A warning icon displays next to an appliance in Offline Workflow Mode. An error icon is displayed if viewed from any other member in the cluster if the member is unable to communicate with the member in Offline Workflow Mode. At any time, you can click Check Health to update the information.
   • A warning message like the following will display: Request Workflow: Access workflow on this appliance is operating in offline isolation from the cluster. This warning will persist until online operations are resumed by an Appliance Administrator.

To manually resume online operations

Before resuming online operations, see Considerations to resume online operations.

1. Navigate to Administrative Tools | Settings | Cluster | Cluster Management.

2. In the cluster view (left pane), click the member of the cluster that is offline.

3. On the appliance in Offline Workflow Mode, click Resume Online Operations. (This operation is only available when the appliance is in Offline Workflow Mode.)

   A message like the following displays:

   The appliance will be reconfigured for online operations. The appliance will attempt to reintegrate with the cluster and merge audit logs. Refer to the to the Admin Guide for more information.

   Type 'Resume Online Operations' in the box below to confirm.

4. In the dialog box, type in Resume Online Operations and click Enter.
5. When maintenance is complete, click Restart Desktop Client. The appliance is returned to Maintenance mode.
6. You can verify health checks on the Cluster Management window. If a warning icon still displays next to the appliance, select the appliance and click Check Health to re-run the cluster health check and display the most up-to-date health information.

Failing over to a replica by promoting it to be the new primary

Safeguard for Privileged Passwords allows you to failover to a replica appliance by promoting it to be the new primary.

To promote a replica to be the new primary in a cluster

1. Log into a healthy cluster member as an Appliance Administrator.
2. In Administrative Tools, select Settings | Cluster | Cluster Management.
3. In the cluster view (left pane), select the replica node that is to become the new primary.
4. In the details view (right pane), click Failover.

5. In the Failover confirmation dialog, enter the word Failover and click OK to proceed.

   During the failover operation, all of the appliances in the cluster are placed in Maintenance mode.

   Once the failover operation completes, the selected replica appliance appears as the primary with a state of online. All other appliances (including the "old" primary) in the cluster appear as replicas with a state of online.

Activating a read-only appliance

Appliances that have been unjoined from a Safeguard for Privileged Passwords cluster or restored from a backup are placed in a Read-only mode.

You can activate an appliance in Read-only mode so you can add, delete and modify data, apply access request workflow, and so on.

The appliance in Read-only mode must be online in order to use the Activate task. If it is offline or the cluster does not have consensus (that is, the majority of the remaining members are offline/unable to communicate), you must use the Cluster Reset option to rebuild your cluster. For more information, see Resetting a cluster that has lost consensus.

To activate a read-only appliance

1. Log into the read-only appliance as an Appliance Administrator.

2. In Administrative Tools, navigate to Settings | Cluster | Cluster Management.

   The cluster view (left pane) displays one primary appliance with a yellow warning icon indicating the appliance is in a Read-only mode.

3. In the cluster view (left pane), select the read-only node to be activated.
4. In the details view (right pane), click Activate.

5. In the Activate confirmation dialog, enter the word Activate and click OK to proceed.

   The appliance's node in the cluster view (left pane) no longer displays the yellow warning icon and the state is now Online.