Do you have SSO enabled in your community as well? (I was also curious as to this one - I assume technically possible, but not something I am adept at as a non-technical person)....
I believe the property is site wide. It is on or off and if I recall correctly, can be set in days before deactivation. I don't know of any way to split this by permission group.