A think it goes back to the days of segregation and apartheid Ollie.
When the US Southern States unofficially segregated blacks and whites in schools, transport, employment blacks were described as "Coloureds"
The Official apartheid policy in South Africa had premises, transport etc signed as for "Coloureds" or "Non-Coloureds" or sometimes "Whites Only" and "Coloureds Only" and this was strictly enforced.
The feelings attached to the word "Coloured" still run deep.