Use group policy to force certain security groups to log off remote desktop sessions

Use group policy to force certain security groups to log off remote
desktop sessions

I'm having difficulty creating a GPO that will be applied only to certain
computers and security groups. Here is what I would like to do.
I have an OU with several computers in them. I would like to apply a GPO
that causes remote sessions that have been idle for x amount of time
disconnect and for disconnected sessions to log out after y amount of
time. I need this to apply to only users in a specific security group.
I have created the GPO and changed the 2 settings in Computer
Configuration -> Policies => Administrative Templates -> Windows
Components -> Remote Desktop Services -> Remote Desktop Session Host ->
Session Time Limits.
Next I Link the GPO to my target OU and turn on the Enforce check box. I
change the Scope Security Filtering to remove Authenticated Users (because
if I don't it gets applied to everyone) and then I add in my security
group. I also add in the Domain Computers group (because if I don't it
doesn't get applied to anyone)
The problem is that the GPO still gets applied to users that are not part
of the security group. If I change the GPO from Computer Configuration to
User Configuration then the GPO does not get applied to any users at all.
I have also tried moving the security group to a separate OU and linking
the GPO to that OU instead, but that did not work either.
It seems like my configuration should work, but I cannot figure out why
it's applying it to users not within the group. Any help is appreciated!