Can I merge data after a split brain situation? (4300673)

KB article and Admin Guide's split brain recovery guide explains how to recover from a split brain situation, but doesn't explain the details why and how can it happen, and what can we do with the saved data.

This KB tries to address this gap.

In case of a split brain situation it might happen that in spite of our suggested best practice, customers use our HA pairs in different datacenters, and a partitioned network leads to the aforementioned split-brain situation.

In this case both node think they are master, since they don't see any activity from the other node, because of the partitioned network.

In this case both nodes continue its operation to audit sessions.

Part of the session metadata is stored in a relational database (PostgreSQL). In a relational database it's best practice to have a primary key field on your tables. Since both nodes continue its operation as master and both node's relational database will add new records to their own version of the metadata it is impossible to merge this two database later because of the colliding primary keys.

So the short answer to the question in the title is: No.

That is why in our recovery guides we suggest to choose one of the nodes as "to be master" and the other as "to be slave". Ideally the "to be master" is the one which audited most of the traffic after the partitioning event, and the "to be slave" is the one with less new audit trails.

The best what we can suggest in our recovery guide is to do a full backup of the "to be slave" node, before these differences would be wiped with the "to be master"'s data. Choose this backup space wise: If you leave this on the same place where the "HA" (the to be master node in this case) do it's backup, the cluster will override these salvaged data! So it's imperative to put this backup elsewhere to actually salvage these differences.

If you need to be able to search in these salvaged audit trails later, the best what you can do is to spin up a VM and install a fresh SPS into that VM than do a full restore from this "elsewhere" space. After the restore, you can remove the license to not violate it with using that outside where it is entitled. To be able to search in the recorded sessions you don't need the license.

The full backup and restore procedure is explained in our migrating procedure KB#306491.

To recover from the split-brain situation you can check these resources:

• Admin Guide's chapter: Recovering from split brain situation.
• KB: 264443