Ben Waddell is leaving "Summer House" after just one season as production is set to begin for Season 11 over the Fourth of July weekend.