Stephen Colbert and Bill O’Reilly had a fraught relationship. Yes, Colbert spoofed Fox’s conservative firebrand on a nightly basis as host of The Colbert Report, but without O’Reilly, Colbert’s alter ego would have never existed. When O’Reilly appeared on The Late Show last July, the two men even seemed to get along. Batman may be ideologically opposed to the Joker, but he’d be nothing without him.

So naturally, the real Colbert had some conflicted feelings about O’Reilly’s stunning ouster at Fox News. “Bill and I did not see eye to eye on … anything,” he said on his Late Show monologue Wednesday, just hours after O’Reilly was fired. “But he’s been a guest on this show and I take no pleasure in his downfall, okay? I’m not going to sit here and publicly gloat.” So instead, he decided to gloat off-camera, while the audience cheered him on, as all Colbert’s attempts to take the moral high ground had gone out the window. At one point, he read a Fox memo that called O’Reilly “one of the most accomplished TV personalities in the history of cable news,” by ratings standards, to which Colbert replied: “By ratings standards, he is. By moral standards, he was a self-righteous landfill of angry garbage.”

Even Colbert’s former alter ego, who used to lovingly refer to O’Reilly as “Papa Bear,” couldn’t help but get a shot in. “Shame on you. You failed Bill O’Reilly,” he said. “You didn’t deserve this great man. All he ever did was have your back — and if you’re a woman, you know, have a go at the front too.”