I have always had this issue with my CGEM mount as well. It also comes up with crazy numbers for how much the mount should be moved during PA. I set the PA option to manual move, then I open up the "motion control' panel in the mount tab and note the 'hour angle' I then use the 'motion control' panel to move the mount 1 hour for each 15 degrees of motion I intend to use for PA. I don't have to go outside to see the mount, the panel tells me what direction the mount is moving, I can control the speed of motion with the 'motion control' panel. It works fine and quick and performs PA exactly as it should. The only time I have to go to the mount is to make the final PA adjustment, for that I start the PA looping (after it has presented the correction arrow or line) and at the scope I monitor the progress of my 'adjustments' using a tablet that is linked wirelessly to the Rpi running Ekos using VNC.
I never have resolved why the PA angle motion numbers are so large or that it does not move as intended, I have just continued to manually move it and it works just fine.