During System Restore the Error Message 'Connection Failed' May Appear as a False Positive (4380790)

During the restoration of a backup that takes longer than 200 seconds, an automatic message will pop up on the screen indicating a failure. It is possible the restoration is ongoing despite the error message. 

This is caused by a timeout value that is hardcoded into the product and may expire before the restoration is done. 

While there are many reasons a restoration could fail, including network issues, this message is misleading and may cause someone to prematurely restart the system, which would result in the restore failing. 

Workaround # 1 

If the host being restored is 8.0 or above and is not a search master a simple test to see if the appliance can process sessions will indicate if the restore was successful. 

During the restore sessions cannot be processed and will be denied. 

Workaround # 2 

A potentially more useful workaround is to use the REST API to collect status from the appliance. This does have some pre-requisites and requires some planning. 

An external host and a user with credentials is required. 

On the external host a local user, which is a member of a group with permissions to run commands. This assumes that 'curl' and 'jq' are available on the system. 

Authenticated 

curl [--insecure] -s -b /tmp/cookies -c /tmp/cookies --user <username>:<password> https://<address-of-sps>/api/authentication [--insecure] -b /tmp/cookies -c /tmp/cookies https://<address-of-sps>/api/health-status | jq .health_status.disruptions

This should display "System restore is running" during a restore and when no restore operation is in progress, this should display:

null

[] 

Unauthenticated 

Alternatively the following unauthenticated command should return a status as well.

curl [--insecure] https://<address-of-sps>/api/proxy-status

This should display "status": "not ok" during a restore and "status": "ok" when no restore operation is in progress.

Workaround # 3

A detailed progression of the restore operation can also be queried.
To do so, issue the following command in the core shell of the SPS.

tail -F /var/log/messages | grep backup

Once the restore process finished the following line will be visible.

scb/backup[963533]: INFO (root@localhost) Finished system restore

Resolution

This issue is being tracked as defect #509908 for potential changes in a future release of the product.