The comments come after Liberal Party leader Peter Dutton argued against giving visas to Gaza refugees temporarily.