The Conservative leader claimed that retailers were being “let down by a policy that is driving their customers away”