I was reading Landsburg's Fair Play today, and he was pretty harsh about people who wanted to restrict immigration. (I'll probably review his book formally so no more commentary on it here...)
Anyway, today it struck me that one self-interested reason I could see current WASPs being afraid of immigrants who look different is that you then lose the camaraderie with the police. I mean let's face it, whitey readers, whether you realize it or not, you really do have a lot more leeway with cops.
We live really close in Nashville to where the seriously loaded/important people live; like Al Gore material. And I got a speeding ticket once and remarked to my wife how incredibly courteous the cop was. She said something like, "Well they're probably being careful since they don't know who they're pulling over."
I imagine if I my skin were darker and I had on a baseball cap, the officer would have been quite sure I wasn't a city building contractor who played golf with the mayor.
So back to the main point, if you suddenly become the ethnic minority in your community, you could be subject to real persecution. I.e. it's not simply, "Eww, I don't want to live next to people who cook that stuff for dinner." (Though I did have a white lady ask me on time when I told her where I lived during grad school, "Ugh, how you can stand the curry smell?")
Naturally, I don't think it's a great argument for restricting other consensual transactions, simply because you're worried that it will down the road indirectly lead to government oppressing you more. None of this has changed my personal views on the best policies in this arena.
But I still think it's a lot more than merely racism. Racism helps it out, for sure, but I think the utilitarian reasoning exists independent of that.
Finally, this is just another reason why the State is bad: It makes groups view each other with fear because that group really could hurt you if it seizes political power. In a free society, you wouldn't fear the police anymore than you worry that construction crews will use their big powerful machines to smash your house down against your will.
The police are scary not because they have guns, but because they have a monopoly.