First of all, separate the titular question from two different questions:
1) Is Birtherism dumb?
Yes, I think it is. There is little question that Obama was born the USA.
2) Are some Birthers racists?
Of course. Some opposition to Obama is driven by racism. But not that much of it. How can I assert that? Most people who are driven nuts by Obama would have been enthusiastic about voting for Colin Powell in 2008 or 2012. And most of those people were even more virulent in their opposition to Bill Clinton. (I was actually around in the 1990s, so I remember! Think about it: Clinton was actually impeached. Nothing remotely like that has happened to Obama.)
Opposition to Obama is driven chiefly by the fact he is a liberal, not by his race. However, it is very convenient for Democrats to write off all opposition to their president as racism: now they don't have to make any argument, just engage in name calling!
(By the way, I supported Obama's election in 2008 and 2012. I am not an "Obama hater" trying to excuse my own opposition to him! But I know many people who fiercely oppose him, and I know they would have voted for Powell in a heartbeat. And many of them disliked Clinton even more than they dislike Obama.)
So, on to:
3) Is Birtherirsm racist?
Once we have handled 2) it should be obvious that the answer to 3) is "Don't be stupid."
People try to discredit the politicians they oppose. Both Democrats and Republicans do this, Christian Democrats and Christian Socialists do this, Conservatives and Labourites do this. They find semen stains on a dress left by the politician. They laugh at how the politician can't check out at the grocery store. They question whether he dodged military duty. They contend that maybe he isn't really legally qualified to hold office. They call him a racist. They say he is "dog whistling."
Folks, just like Geico saves you 15% or more on your car insurance, smear attacks are what partisans do. It's not pretty, and we may wish it were otherwise, but when accusation like this are flung at a black politician by members of the other party, that's not racism! That is partisan politics. What actually might be a wee bit racist is the notion that black politicians alone should be exempt from this rough-and-tumble.