Authority is not rule

Libertarians often make the mistake of confusing authority with rule. The referee of a basketball game is not the ruler of the game. But he does have the authority to decide when someone has committed a foul and when they should be thrown out of the game.

Furthermore, what makes the referee an authority is not some "magical" power he possesses that allows him special insight into the sport of basketball, or some sterling virtues that make him a better person than the players. No, he is an authority because he has been designated as one in an authorized procedure: the league has appointed him. And if some referee abuses his authority and attempts to act like a ruler, that does not discredit the office of referee, but only the person of this referee.

And note: the fact that he makes mistakes in these calls does not diminish his authority. Players and coaches may not remove him even if both teams believe he has made a series of bad calls. For an authority to be removed properly requires again an authoritative procedure: in this case, a league review of the referee's conduct, a hearing, etc. (In the case of government officeholders, the equivalent procedures would be elections, impeachment hearings, and so on.)

Officeholders in a constitutional government are authorities, not rulers. And if in our time they have often come to act as if they are rulers, then a portion of the blame falls at the feet of those who have kept calling them that, and who have been trying to persuade the average person to shun politics, which is the only possible way of correcting this (political) problem.