I think that Zhongli is most aggravated by a lack of decorum or respect. That's why he struggles so much with Hu Tao - she's completely irreverent. He expects traditions, laws, and social contracts to be followed to the letter, so when a brash, reckless Snezhnayan diplomat comes to Liyue Harbor, flashing a too-wide smile and laughing at inappropriate moments, it gets on his nerves. Childe either doesn't know much about Liyuen customs or doesn't care, but either way he fails to show Zhongli, the Qixing, and the general populace the respect they are entitled to.

Later on, Childe's bold departure from tradition will seem endearing, commendable even. But that's when he's wearing the rose colored glasses of someone hopelessly in love.