Victorian Labor ministers and backbench MPs from both the Left and Right factions are more frustrated than ever.