The referee in a sporting event can be a neutral arbiter because the question of the rules themselves is not at play during the event: the referee's only job is to enforce a set of pre-existing rules, and not to decide what the rules should be. The process of deciding the rules inherently cannot be neutral, because certain sets of rules will favor one participant, and other sets other ones. For instance, adding a 3-point shot in basketball helped smaller, more skilled players at the expense of bigger, more physical ones.
This is why liberalism's pretense to being a neutral arbiter amongst different value systems was never a possible state of affairs. Any set of rules will favor one value system over others, and what liberalism has always meant is favoring liberal rules, rules hat privilege the liberal value system.
Take, for example, rules about public modesty. Liberals often try to present their preferred arrangement, which could be called, "Everyone dresses whatever way they want," as neutral, because, "If you want to dress modestly, you can, and if others want to dress provocatively, they can."
But this way of deciding the matter, regarding public presentation as purely a matter of individual choice, is liberal through and through, and privileges liberal values. Furthermore, it is quite extraordinary in terms of the history of human societies, in which it has almost always been the case that the way one dresses has been a matter of social choice, not personal preference. And this also demonstrates how liberals can create the illusion that their solutions are "neutral": they present them as if they were the default, and that any other solution would be some extraordinary imposition, when, in fact the opposite is the case: traditional communities have had, again and again, to have liberalism forced upon them through state force.