It is this mans name Nudger and all you have done is point out something that could be abused in a racist way, you can guarantee that the people accusing you were "laying in wait" for someone to make a comment so they could attack! Don't get too bent out of shape about it they will be on the lookout for something else soon and will move on to that/them, probably something like canvassing for Chess and Draught peices to be Blue versus Red?
Racism and Discrimination is wrong in all forms and right minded people know this and respect all people regardles of any of the protected charachteristics.