Well you could check with GetInvocationList if the Eventmanager is already attached. Check if it is already attached with that manager with a specific target on the other hand i don't know if thats possible. When looking at the PropertychangedEventManager though, it is perfectly fine to add multiple listeners to one manager with one target(distinguished with different property names)
For me the correct way of calling RemoveListener is to use IDisposable. The problem with the finalizer is that your object lives longer, because it will be on the finalizer queue, and might get events at that time(not sure about that). Maybe this will help