While the SNP’s recent issues put off some voters, the Tories’ support for North Sea oil drilling proved more decisive