Is, as economists would say, that privilege exists along many margins. Of course there are situations where merely having white skin is an edge: gaining membership in the Klu Klux Klan is an obvious example. And there are many others where the legacy of past discrimination has accumulated to white people's advantage. But there are other situations where white skin clearly is not an advantage. For instance, my son played for a few weeks in an elite Brooklyn basketball program. Regularly, he would hear his opponents taunt him with, "Yeah, white boy, I'm gonna..." whatever. (I told him he should at least get them to taunt him correctly, as "Half-white, half-Filipino boy.") After a few weeks he couldn't take any more of this and dropped out.
And there are many, many other aspects to privilege: being beautiful can make you privileged. (Beyonce appears to be in a fairly privileged position to me.) Being the president's kids isn't a bad deal. (The kids of some meth-addled trailer-park mom in Appalachia will no doubt be amused to discover that their "white privilege" gives them an edge over Malia and Natasha Obama.) Being the son of the greatest international reggae superstar is an edge. (Respect due, Ziggy.) Being very tall makes one privileged in some regards: I recently read that well over 10% of males in the US over 7 feet tall and between 20 and 40 years of age are currently in the NBA!
So one problem I see with the idea of white privilege is not that it is total nonsense but that it is a one-sided emphasis on one aspect of privilege amongst many others. But there is a further problem: the whole project of harping on privilege itself seems to assume that we could create a human society in which privilege does not exist. But we have never, ever seen such a society: even in the most egalitarian of hunter-gatherer cultures, being the chief's son was no doubt an advantage. And current efforts to fight one privilege typically just create privilege for a different group: affirmative action has largely benefited the children of upper middle-class blacks, especially black immigrants, so that, say, Nigerians are four times more likely to be doctors in the US then is the average American, while doing very little for the most under-privileged blacks. (Note: this is just what Pareto described. New political projects are forwarded by a rising elite trying to displace an established elite: their non-elitist elements are mere "derivations" created to collect non-elite support for the rising elite.)
A just society can try to ameliorate the effects of privilege. But to try to eliminate those effects is a utopian project, which, when taken to its logical extreme, is likely to result in things like the death of everyone who wears glasses.