While there is not a "don't send email notification" option when inviting users to a social group, there are a few options available if you are not wanting group invitation emails sent.
1. Have users configure their preferences so that they are not receiving email notifications - Customizing Notifications
2. Break the message template associated with the "invite to group" action (Editing Email Notification Templates ). This would prevent an email from being sent even if a user has their preferences set to receive an email, however would be a global action in your instance.
Lastly, if you are interested in seeing a "don't send email notification" option I would recommend helping to vote up the following idea found in the Ideas for Jive