Is calling someone "Trash" offensive? If you say no, then is calling someone "brown trash" offensive?
If you say yes to the 2nd question but no the first question then what you are saying is that calling someone trash is ok but to distinguish what kind of trash they are based on their skin colour is not ok.
If you just called someone "brown" and didn't consider that to be offensive then it's a dumb position to take because your argument lies in the colour of their skin rather than whether or not they are trash.