Protester's Free Speech vs. Speaker's Free Speech

Although a fierce believer in free speech, I must admit that I have never been a great fan of protesters who interrupt and disrupt speeches (whether it is at a political rally for either party or the State of the Union Address). Of course, political rally protesters have free speech rights, but my common sense tells me that the rights of the speaker who has paid for the venue, advertised the event and invited the public should prevail in the battle of voices. Also, as a practical matter, if protesters were under no limits to express themselves, then that could be the demise of all public free speech, because the protesters exercising their rights could silence and drown out the speech rights of everyone else.

So conceding that limiting the free speech of protesters is a practical necessity in some, but not all, instances, brings me to my question: What is the authority that allows protesters to be grabbed and dragged out of a rally by force? The enforcers are most often private security guards, not Secret Service -- who are there to protect the candidate from harm -- not oust anyone who opposes the candidate. Let's take the most benign (and probably most popular) example: a protester holding up a sign saying, "Trump is a Racist", and doing nothing more. Some burly guys will come along and usually rip up the sign, grab the protester and drag him or her out of the arena. Isn't that an assault? Possibly a battery? False arrest or false imprisonment?

Holding up a sign while someone else is speaking cannot possibly be a crime. Even if more active protest occurs and for some reason is denoted as criminal activity -- which is difficult to imagine, how do personal security guards have the right to enforce it. I see protesters being removed at every rally, and sit and wonder: How is it justified? And they are not gently escorted out -- they are physically grabbed and dragged out of a place they have a right to be. So when principle and practicality come in to conflict -- who should win?