No, it's silly to just hate someone you enjoy simply because of their association, now if they begin to adopt those traits that you dislike that's different, but simply engaging with a person you don't particularly like shouldn't make you like the person less, I think that says more about you than it does them.