Hey there
I can confirm the behavior, but your statement isn't accurate I think
Without getting too much into it, I think your issue reside in the fact that you reactivate it in the 24hrs "blackout time" following the previous automatic message.
Every couple of sender/receiver is logged, and is, by default, kept for 24hrs (I think).
So if you get the reply, disable then re-enable this feature : using the same sender will not get the new notification
You can see those in the "autoreplies_log" table.
If you delete the "couple" sender/receiver, you should get the notification again (once).