Ahhh... I always thought it comes off as the fermentation finishes and drops away. Every brew I notice big swings in the middle or later stages of the fermentation but it always seems to right itself (well, it did anyway).However, that isn't the case with final gravity. Each fermentation will be different and a different amount of yeast will stick to the bottom each time. And a different amount will stick to the top each time. This will throw off the reading by a random amount each brew
I'm going to fish it out before bottling just to see if this is what has happened.