Well... it's not inviting me specifically, it's just inviting people. I'd put an actual response in if you actually mentioned me by name, but if it's just a general message of 'hey you are all invited', I also don't feel compelled to answer specifically.
If I send a message to a group, it's intended for everyone in the group - what's casual about that?