Sometimes being around certain people is worse for them than not being around them. They might not understand why you keep your distance at certain times. And they might find your absence hurtful. But they might find your behavior even more perplexing and hurtful when you are around them.
Keeping your distance is similar to moderation of speech. In other words, we don’t say everything that comes to mind. Physical distance from other people can be a way of moderating yourself. Because sometimes the presence, words, or actions of other people can influence our words and behavior, and not always for the better.