Make sure you're using a classic wood drill bit and make sure the drill is NOT on hammer or masonry modes ect. Take it slow.
The plastic sounds brittle as none of mine cracked. In this case try using a tiny drill bit, a bit bigger one followed by a bit bigger one. They shouldn't crack if you do...