The SQL AlwaysOn listener is likely configured with DNS Round Robin instead of a geographically-aware load balancer.
To confirm this, perform an NSLOOKUP against the listener name which is configured in the Active Roles Configuration Center. If multiple IP Addresses are returned, the listener is likely configured with DNS Round Robin.
Configure the SQL AlwaysOn listener DNS alias to be load balanced by a geographically-aware appliance.
On the Active Roles host, configured the HOSTS file so that the DNS alias of the SQL AlwaysOn listener resolves to an IP address in the local site.