Sometimes a "no" is as important as a "yes" when gauging interest and reminding people to sign up.
If someone has stated "no" you know who you should be sending the event reminders to, and you should be able to target just those who have actually not responded yet.