It is a definitional and logical conclusion that a concept cannot tolerate its anathema and inverse.

This is a pretty good rewording removing ambiguity.

As for my experience seeing this point brought up, its usually to silence a voice, and then this logical statement is equaled to the moral reasoning and justification in one, instead of reasoning inside that case how a “removal” would be required.

What if the other party in question is of the opinion they didn’t break it, yet the other claims it has been. Who gets to decide it?

You misunderstand the point of this paradox. By default you become intolerant when you start “removing” people. it is explicitly not a justification for whatever action you claim moral superiority on.

Since almost every political decision will affect at least some fraction of society negatively (even if it would ethically be for the greater good), you can carelessly throw this around to eliminate any opponents for this arbitrary tolerance reason. The only way to make sure the “removal” is fair, as a society absolutely needs these tools to function, is to clearly outline the case when it needs to happen and bring the barrier such that those capable of improvement do not get ostracized into further radicalization. And that barrier needs to be significant.

You bring up “fascist”, at which line does it happen? Genocide execution, support, inaction, Swastika wearing, illegal membership, legal membership of ultra radical parties, support of conservative oligarchs? What is greedy? Robbery, theft, tax evasion, corruption, cheating with the girlfriend of a friend? What is bigotry? You get the idea.