Exit poll loop on POLLHUP or anything unexpected

In my opinion, we really shouldn't even get POLLHUP, but
we do. In case we get anything else odd, quite polling on
that fd.