Regulator finds inadequate monitoring by firm could have exposed vulnerable customers ‘to clear risk of harm’