to quote a phrase I think I've seen on this forum.....
Give time in the bucket... and give time in the bottle.
No matter how quick your brew ferments ( some go like rockets some are like us wrinklies... slow and steady... but dependable ) try and give a period in the FV of around 10 days. This gives plenty of time for the fermentation process to be completed, the rubbish to settle down and for the yeasts to clear up some proteins and enzymes they have made....
The brew will if properly sealed come to no harm, and instead will improve and absorbs many " flavours " from the ingredients. The rubbish at the bottom will not hurt the brew in this time scale. Of course leaving the brew on the rubbish for much longer is not recommended and its better to get on with the process, to minimise any chance of infections getting a hold.
A good maxim to work to is look at your brew and if you think its ready to bottle,leave it another 24-48 hours. As well as taking your gravity readings.