Understanding a Locked Safeguard for Privileged Passwords (SPP) Cluster During Audit Log Maintenance
When performing audit log maintenance on your One Identity Safeguard for Privileged Passwords (SPP) cluster, you might observe that the cluster enters a 'locked' state. This Knowledge Base article explains why this occurs, what it signifies, and important considerations.
Why Does the Cluster Lock During Audit Log Maintenance?
The SPP system is designed to maintain consistency and stability across its clustered environment. To achieve this, only one cluster operation can run at a time. Audit log maintenance, which involves tasks like synchronizing, archiving, or purging audit logs, is considered a critical cluster operation that requires a temporary lock. Other operations that also lock the cluster include enrolling new appliances, unjoining replicas, failovers, patching, resetting the cluster, and IP address updates.
What Happens When the Cluster is Locked?
When the cluster is locked:
- The Appliance State will display a red lock icon in the web client.
- Changes to the cluster configuration are not permitted until the ongoing operation (in this case, audit log maintenance) is complete.
- The cluster is unavailable for other cluster operations.
- Individual appliances within the cluster may enter a Maintenance mode and be temporarily unavailable for approximately 5 minutes when audit logs are archived or purged.
- During synchronization, the cluster is locked for "ensuring data consistency".
How Long Does the Lock Last?
The duration of the cluster lock due to audit log maintenance can vary. It may take hours depending on:
- The amount of audit log data on the appliance.
- The volume of data being archived or purged.
- The network conditions between the synchronizing nodes in the cluster.
Can the Cluster Lock be Released or Cancelled?
Yes, a locked cluster can be unlocked or cancelled. You can cancel Audit Log Maintenance from either the Audit Log Maintenance page itself or from the Cluster Management page.
Important Considerations When Unlocking:
Extreme caution must be exercised when unlocking a locked cluster. It is only recommended if you are absolutely certain that one or more appliances in the cluster are offline and will not complete the current operation. Forcing the cluster unlock when it's not truly necessary could lead to instability on an appliance, potentially requiring a factory reset and even a complete rebuild of the cluster.
Steps to Cancel Audit Log Maintenance:
To cancel Audit Log Maintenance from the web client:
- Navigate to Backup and Retention > Audit Log Maintenance.
- While Audit Log Maintenance is running, a Cancel button will be available.
- Click Cancel. A confirmation dialog, Unlock Cluster, will appear.
- Type "Unlock Cluster" into the text box and click OK. This will immediately release the cluster lock.
Important Post-Cancellation Monitoring:
After cancelling, it's crucial to monitor the Activity Center to ensure the operations are truly complete. The required events to look for depend on the specific audit log maintenance action that was running:
-
If you were synchronizing data and audit logs only:
- The lock is released immediately.
- You must wait for the SynchronizingDataCompletedEvent and then the SynchronizingAuditLogsCompletedEvent to appear in the Activity Center before performing other clustering operations. This ensures all nodes have all audit data.
- The cluster will attempt to complete the audit log synchronization again the next day at the configured start time.
-
If you were synchronizing after archiving and deleting audit logs, or just deleting audit logs:
- The lock is released immediately.
- Monitor the Activity Center for either a SchedulerJobSucceeded or SchedulerJobFailed event, containing
Job Id = core.AuditLogMaintenance
. This indicates that the archive/purge portion has completed. - Audit Log Maintenance will continue to synchronize regardless of whether the archive/purge succeeded or failed. You may need to cancel again if the cluster locks for "Ensuring data consistency" during this subsequent synchronization.
Where to Monitor the Status:
You can monitor the status and progress of cluster operations, including audit log maintenance, in the following areas of the SPP web client:
- Activity Center: Provides detailed events and user activity logs.
- Cluster Management: Displays the overall health and status of the cluster, including any lock notifications.