-
Title
How does Safeguard handle failures for scheduled platform tasks? -
Description
There are two different ways that management tasks can be executed on a One Identity Safeguard for Privileged Passwords appliance: manually, as when a person initiates a task, or by schedule, as when executed as part of a profile.
Test / Check Connection can only be performed manually. Here manage means “perform platform tasks,” specifically:
- Test / Check Connection- Account Discovery
- Account Password Check
- Account Password Change
- Account Suspend
- Account Restore
-
Resolution
Manual Task BehaviorIf a task is manually executed against an asset that is disconnected from the network, it will fail because Safeguard cannot reach the managed asset.
Scheduled Task Behavior
If a task is scheduled for execution and fails, as it will for a disconnected asset, Safeguard’s scheduler repeatedly attempts to execute the task until it succeeds. It does so in the following manner:- The scheduler follows a back off algorithm to reschedule the task with increasing delay between attempts, up to the user-defined Max Platform Task Retries (default 50, configurable via Web UI > Appliance Management > Settings > Platform Task Settings)
- As the delay increases, it eventually exceeds the profile-defined amount of time until next scheduled task
- The scheduler stops following the back off algorithm and continues to attempt the task according to the profile-defined schedule
- Once successful, the scheduler clears the retry count and if tasks fail again will again follow a back off algorithm
Failed attempts are visible by querying the AssetAccount or DirectoryAccount API, or by viewing the Admin Dashboard in the Safeguard Desktop Client.
Back off Algorithm Details
The back off algorithm follows a quadratic curve to increase the number of seconds between retries and is implemented differently for different tasks.
For Password Change, Account Discovery, Account Suspend, Account Restore
(1 x 10)^2 seconds, (2 x 10)^2 seconds, ..., (n x 10)^2 seconds
Where n = number of attempts.
For Password Check
(1 x 20)^2 seconds, (2 x 20)^2 seconds, …, (n x 20)^2 seconds
Where n = number of attempts.
The scheduler uses this algorithm to calculate how many seconds after the initial failure to schedule the next retry, rounded down to the nearest minute.
Example Table of Scheduled Task Intervals, in SecondsAttempt # Algorithm 1 Interval Algorithm 2 Interval 1 100s after initial failure 100 400s after initial failure 400 2 400s after initial failure 300 1600s after initial failure 1200 3 900s after initial failure 500 3600s after initial failure 2000 4 1600s after initial failure 700 6400s after initial failure 2800
Password Change Schedule Example
Assume a Safeguard appliance has been configured with a password change schedule that executes a password change every 10 minutes on an asset that is unreachable.
Scheduled Task 1 (1:00 PM) – the appliance executes the password change per the schedule, but it fails. The back off algorithm shows that retry 1 should occur 100 seconds (1 minutes 40 seconds) later, so the scheduler rounds down to 1 minute (1m 40s rounded down) and reschedules the task for 1:00 PM + 1 minute (1:01 PM).
Retry 1 (1:01 PM): This, too fails. The back off algorithm shows that Retry 2 should occur 300 seconds (5m) after Retry 1, so the scheduler reschedules the task for 1:01 PM + 5 minutes (1:06 PM).
Retry 2 (1:06 PM): The second retry fails. The back off algorithm shows that Retry 3 should occur 500 seconds (8m 20s) after Retry 2, so the scheduler rounds down to 8 minutes (8m 20s rounded down) and reschedules task for 1:06 PM + 8 minutes (1:14 PM).
Retry 3 (1:14 PM): The third retry fails. The back off algorithm shows that Retry 4 should occur 700 seconds (11m 40s) after Retry 3. Note that the algorithm has reached the point where the next retry (11 minutes from now) would occur after the next scheduled change (10 minutes from now) according to the user-defined Password Change Schedule. This causes the scheduler to stop following the back off algorithm and schedule the next task for 1:24 PM, per the password change schedule.
Scheduled Task 2 (1:24 PM): This task fails. The scheduler does not use the back off algorithm to retry, instead scheduling the next task for 1:34 PM, per user-defined Password Change Schedule.
Scheduled Task 3 (1:34 PM): This task succeeds, as the asset has become available again. The scheduler clears the retry count and schedules the next task for 1:44 PM. If that task fails, this process restarts and repeats, including the back off algorithm.