We normally see him at his angriest, or at least the most emotional, whenever he's being reminded of his past in the port mafia and more specifically what he had done. For example in the guild arc his dislike for Yumeno after coming face-to-face with them for the first time in the years was very apparent and filled with hostility as they're probably a representative of Dazai's past mistakes (he was earlier reminded about his past being mafia black according to his records). So when Jouno points out his rage when he's being arrested due to his past crimes in the port mafia it's likely him being angered from how the worst parts of his past always seems to haunt him. Dazai is not really the type who likes to confront his past, which is understandable, and so his hidden anger stems from those reminders that makes him remember what he's been trying to move on from.