If I fear spiders even though I know they won't hurt me then that's a phobia. I fear muslims because I know that in many situations they will harm me. That's not a phobia, it's perfectly rational. If I were walking around Paris now, I'd be looking over my shouilder and with good reason.

If this attitude makes me Islamophobic then so be it.