Welcome to the joys (and heartache) that is astro photography! In your case, you would want to align the camera and the mount at the same time, and then leave the ball head as is. Move only the mount controls, and never the ball head after alignment. The way you describe it, the mount and the software still think they are pointed at Polaris, even though the camera is somewhere else.