I think everything can be explained by Iske's speech to Ellen about him only putting up with Freya because she's his sister's friend. This leads me to the conclusion that he never even allowed other women to bother him, and because Freya was the only one he didn't actively push away, the rumors were born.