Maintain a steady speed and a loose grip on the clip ons and you'll be fine. It's when you freak out and either brake or try and correct your front wheel by steering that you'll run into trouble.

Same thing with concrete/pavement that has been grated to be repaved at a later date.