A cocktail party fundraiser has been cancelled hours before it was due to begin after anti-racism protesters vowed to demonstrate outside the venue.