When is the word 'boy' derogatory?

My family is from Arkansas.  I don't recall ever hearing them refer to a black man as 'boy'.  But while visiting Arkansas for many family reunions, I know that the word 'boy' is not used in a good way, though usage has waned over the years.  It's not always meant in a bad way, but is part of the language spoken in the South and, when spoken to an adult, is exclusively used when talking to black men.  Some people say it because that is what their family and friends do.  Not an excuse, but rather what happens.  I'd like to believe that once most people hear that 'boy' is demeaning to a black man, they would stop using it.

Which is why I found this article about the use of 'boy' at a Tyson Foods plant in Alabama interesting.  The Supreme Court said there were 'no racial overtones' in using the term.  Read the entire article and see what you think.

Appeals Court in Atlanta Again Rejects Racial Discrimination Claim (NY Times)

My question would be, did the managers call any white men 'boy'?  If they did, then there is probably not much of a case.  If they did not, as seems the case, then the case is stronger.