I was asked this once before lol. I think realistically speaking probably. Rinne definitely does, he obviously did not get along with their father. Hiiro doesn't seem to take issue with their father but was pressured to fill in for Rinne after he ran away. I don't think it's specified, but I wouldn't be surprised if Papa Amagi was among the people in the village to not really "get" Hiiro. I don't think he would be cruel to his son, but there's the possibility he found Hiiro frustrating. That would affect their relationship.