When Roly Gregoire quit football in 1980 after suffering a knee injury, he had faced so much racism that he could not bear to watch football for many years.