Prince Harry’s rep revealed his invitation to stay at Buckingham Palace during his upcoming U.K. visit was “withdrawn at the last moment.”