I always thought about this word as a place where poor people usually live. I associated class rather than race to this word. However according to Longman dictionary it might also have racial connotations.
This is one of the definitions of ghetto in Longman dictionary:
"A part of a city where people of a particular race or class, especially people who are poor, live separately from the rest of the people in the city. This word is sometimes considered offensive."