Remove a peer or upward answer from a participant's view