The QER update trigger for DialogGroup does check whether the changed DialogGroup is assigned to the Application Role and in this case prohibits the change of "IsRoleBasedOnly". However, "IsRoleBasedOnly" is not evaluated in the QBM trigger. This has the following consequences:
1. If "IsRoleBasedOnly" goes from 1 to 0, this group will not be assigned to any viadmin-like admin users (IsAdmin=1, IsServiceAccount=0).
2. If "IsRoleBasedOnly" goes from 0 to 1, this role-based group remains assigned to admin users.
This is a product defect (34098).
This will be fixed in a future release of the product. If you require this immediately corrected, please contact Support for a hotfix referencing the defect ID 34098.