I wonder if this is a problem with polar alignment. I have been making sure my PA has been within a few Arc Seconds and noticed my drift virtually went away.
Last night I let my system run for 4 hours, taking images every minute and dithering every 4 frames. Four hours later the target was no more than a minute off.