It's a common theme in TV shows and movies to see people positively boasting that they are "terrible at math." This is interesting: you don't see anyone boasting that they are terrible at reading, or terrible at choosing breakfast.
But mathematics displays the reality of objective, transcendental truth. And this is a serious threat to the world of doxa, of mere opinion. So it is best to portray it as something only an elite few can possibly understand.