Conservative Party backers have cautioned that the party is set for a voting ‘disaster’ unless it takes action on policies such as immigration.