There has been a "substantial loss" to taxpayers, as a result of the termination of agreements with a number of potential asylum seeker accommodation providers.