Thanks for posting about this, Chris.
I suggest going ahead and posting a support case about this either way, to raise visibility to the current functionality. It's behaving "as designed" currently, but I can definitely see how it could be improved by sending to the recipients who had not already joined, rather than making you start all over again.
I just tested this on my 5.0.1 community and found that while the error states the invite was not sent, it was indeed sent to the ones who did not get the original invite.
My test case:
1. Created Private Group
2. Invited two fake members: Emre and Brenda
3. Invited two more members: Emre and Joan
4. Got the error
5. Logged in as Joan and she had the invite
Hopefully this is the same as your site is seeing. If not, then it's not working as intended.
Either way, the message needs to reflect who was not sent the invite so it's not misleading.