When an object with toggle reference is notifying a change we just
assume that this is true because of previous checks.
However, while locking, another thread may have removed the toggle
reference causing the waiting thread to abort (as no handler is set at
that point).
To avoid this, once we've got the toggle references mutex lock, check
again if the object has toggle reference, and if it's not the case
anymore just ignore the request.
Add a test that triggers this, it's not 100% happening because this is
of course timing related, but this is very close to the truth.
Fixes: #2394
468246bb3b