×

INDI Library v2.0.7 is Released (01 Apr 2024)

Bi-monthly release with minor bug fixes and improvements

Read More...

Ekos Scheduler did not wake up

  • Posts: 1224
  • Thank you received: 566

Replied by Hy Murveit on topic Ekos Scheduler did not wake up

Lee,

Please share .esq and .esl files. I really doubt that's it. Here's something else...I put together a toy schedule that ran one job (one image capture) and then slept waiting to wake up several hours later. In my (successful) simulation here's what happened:
[2023-04-01T16:14:31.476 PDT INFO ][ org.kde.kstars.ekos.scheduler] - "Job 'Capella' is complete."
...
[2023-04-01T16:14:32.966 PDT INFO ][ org.kde.kstars.ekos.scheduler] - "Job 'Alioth' scheduled for execution at 01/04 21:47. Observatory scheduled for shutdown until next job is ready."
[2023-04-01T16:14:32.967 PDT DEBG ][ org.kde.kstars.ekos.scheduler] - Checking shutdown state...
[2023-04-01T16:14:32.967 PDT INFO ][ org.kde.kstars.ekos.scheduler] - Starting shutdown process...
[2023-04-01T16:14:33.965 PDT DEBG ][ org.kde.kstars.ekos.scheduler] - Checking shutdown state...
[2023-04-01T16:14:33.966 PDT INFO ][ org.kde.kstars.ekos.scheduler] - "Parking mount in progress..."
...
[2023-04-01T16:14:45.694 PDT INFO ][           org.kde.kstars.indi] - Telescope Simulator :  "[INFO] Mount is parked. "
[2023-04-01T16:14:45.695 PDT INFO ][     org.kde.kstars.ekos.align] - Target coordinates updated to JNow RA: "03h 46m 02s" DE: " 90° 00' 00\""
[2023-04-01T16:14:45.695 PDT INFO ][           org.kde.kstars.indi] - Telescope Simulator :  "[INFO] Telescope slew is complete. Parked "
[2023-04-01T16:14:45.716 PDT DEBG ][     org.kde.kstars.ekos.mount] - Mount status changed from  "Parking"  to  "Parked"
[2023-04-01T16:14:45.965 PDT DEBG ][ org.kde.kstars.ekos.scheduler] - Checking shutdown state...
[2023-04-01T16:14:45.965 PDT INFO ][ org.kde.kstars.ekos.scheduler] - "Mount parked."
[2023-04-01T16:14:46.965 PDT DEBG ][ org.kde.kstars.ekos.scheduler] - Checking shutdown state...
[2023-04-01T16:14:47.965 PDT DEBG ][ org.kde.kstars.ekos.scheduler] - Checking shutdown state...
[2023-04-01T16:14:48.967 PDT DEBG ][ org.kde.kstars.ekos.scheduler] - Checking shutdown state...
[2023-04-01T16:14:48.967 PDT INFO ][ org.kde.kstars.ekos.scheduler] - Disconnecting INDI...
[2023-04-01T16:14:48.968 PDT DEBG ][           org.kde.kstars.ekos] - Disconnecting  "Telescope Simulator"
[2023-04-01T16:14:48.968 PDT DEBG ][           org.kde.kstars.ekos] - Disconnecting  "CCD Simulator"
[2023-04-01T16:14:48.968 PDT DEBG ][           org.kde.kstars.ekos] - Disconnecting  "Focuser Simulator"
[2023-04-01T16:14:48.968 PDT DEBG ][           org.kde.kstars.ekos] - Disconnecting  "Guide Simulator"
[2023-04-01T16:14:48.968 PDT INFO ][           org.kde.kstars.ekos] - "Disconnecting INDI devices..."
[2023-04-01T16:14:49.010 PDT INFO ][           org.kde.kstars.indi] - Telescope Simulator :  "[INFO] Telescope simulator is offline. "
[2023-04-01T16:14:49.011 PDT DEBG ][           org.kde.kstars.ekos] - "Telescope Simulator"  is disconnected.
[2023-04-01T16:14:49.011 PDT INFO ][           org.kde.kstars.ekos] - "Telescope Simulator is disconnected."
[2023-04-01T16:14:49.013 PDT DEBG ][           org.kde.kstars.ekos] - "CCD Simulator"  is disconnected.
[2023-04-01T16:14:49.013 PDT INFO ][           org.kde.kstars.ekos] - "CCD Simulator is disconnected."
[2023-04-01T16:14:49.019 PDT DEBG ][           org.kde.kstars.ekos] - "Focuser Simulator"  is disconnected.
[2023-04-01T16:14:49.019 PDT INFO ][           org.kde.kstars.ekos] - "Focuser Simulator is disconnected."
[2023-04-01T16:14:49.024 PDT DEBG ][           org.kde.kstars.ekos] - "Guide Simulator"  is disconnected.
[2023-04-01T16:14:49.025 PDT INFO ][           org.kde.kstars.ekos] - "Guide Simulator is disconnected."
[2023-04-01T16:14:49.028 PDT DEBG ][ org.kde.kstars.ekos.scheduler] - Scheduler INDI status is 0
[2023-04-01T16:14:49.965 PDT DEBG ][ org.kde.kstars.ekos.scheduler] - Checking shutdown state...
[2023-04-01T16:14:49.965 PDT INFO ][ org.kde.kstars.ekos.scheduler] - "INDI devices disconnected."
[2023-04-01T16:14:49.967 PDT INFO ][ org.kde.kstars.ekos.scheduler] - Stopping Ekos...
[2023-04-01T16:14:49.975 PDT DEBG ][           org.kde.kstars.ekos] - Resetting Ekos Manager...
[2023-04-01T16:14:49.993 PDT INFO ][           org.kde.kstars.ekos] - "INDI services stopped."
[2023-04-01T16:14:50.006 PDT DEBG ][ org.kde.kstars.ekos.scheduler] - Scheduler Ekos status is 0
[2023-04-01T16:14:50.965 PDT DEBG ][ org.kde.kstars.ekos.scheduler] - Checking shutdown state...
[2023-04-01T16:14:50.965 PDT INFO ][ org.kde.kstars.ekos.scheduler] - "Ekos stopped."
[2023-04-01T16:14:50.967 PDT INFO ][ org.kde.kstars.ekos.scheduler] - "Shutdown complete."
[2023-04-01T16:14:50.967 PDT INFO ][ org.kde.kstars.ekos.scheduler] - Scheduler is stopping...

Note all the stuff that happens after "Mount Parked". Your log had none of that. Your scheduler didn't do anything after "Mount Parked", and thus probably didn't shutdown properly and thus (my guess is) probably didn't set its timer to wake up.

You say you are debugging the mount. Do you think there might be some communications issue that might be confusing INDI and somehow causing it to hang?
Hy
1 year 3 weeks ago #91600

Please Log in or Create an account to join the conversation.

  • Posts: 1224
  • Thank you received: 566

Replied by Hy Murveit on topic Ekos Scheduler did not wake up

Lee,

Since you're a retired NASA programmer, perhaps you can join in and help debug this.

After the scheduler gets the "PARK_PARKED" message from INDI
invent.kde.org/education/kstars/-/blob/m.../scheduler.cpp#L6344

There's a complex state machine that the scheduler works through, waking up every second to check on things.

In any case, please make sure that verbose logging is enabled.
Perhaps that's why you were missing all those other messages, not sure.
I'm happy to look again at the verbose log tomorrow.

Hy
1 year 3 weeks ago #91601

Please Log in or Create an account to join the conversation.

  • Posts: 24
  • Thank you received: 0

Replied by Lee Peterson on topic Ekos Scheduler did not wake up

Morning Hy,

Unfortunately last night's run crashed again in the middle of the first run due to a mount error. I bet this is behind the problem with the scheduler not waking up. Here is the error message that repeated every second all night:

iOptron v3 : "[ERROR] bool IOPv3::Driver::getCoords(double*, double*, IOPv3::IOP_PIER_STATE*, IOPv3::IOP_CW_STATE*): Expected 20 bytes but received 23. "

This particular mount is an iOptron CEM 40 EC. Previously I had been communicating with it via a powered USB bus, and had problems with it dropping intermittently over the past year, often requiring a restart of ekos. I recently tracked down that there was some EMI issue with the mount's USB controller almost every time it slewed. I surmise that was due to an issue in the mount's power system or something emitted by the stepper motors. It actually was creating issues with all the USB devices. So I moved the mount comm over to the wifi interface and it solved the EMI problem and had been running fine, apparently.

But now I think the wifi interface has created another issue with getting the mount state, maybe. I found this interesting comment in the source code.

github.com/indilib/indi/blob/master/driv...e/ioptronv3.cpp#L808

That's right where where the status gets set in the driver. This particular stellar mate has indilib 1.9.7 installed. I'm going to try some more debugging runs and I'll let you know if I can see what is happening.
1 year 3 weeks ago #91616

Please Log in or Create an account to join the conversation.

  • Posts: 24
  • Thank you received: 0

Replied by Lee Peterson on topic Ekos Scheduler did not wake up

Hello Hy,

I've studied the code and logic, and had a look at the iOptron RS-232 syntax. I ran another schedule last night the same as before, and encountered the same stuck state. Here's what I've found:

- The iOptron error I mentioned above did not recur. The debug output was indicating that the mount responses to get_status and get_position were mixed up. That is, the command string ":GLS#" for status was returning the response for the command string ":GEP#" and visa versa. That looks like a mount firmware problem, or perhaps something wrong in a FIFO buffer somewhere. At any rate, the error did not repeat, and so I'll ignore it for now.

- What I did see is that, when the scheduler went to sleep, it parked the mount, emitting a "Mount parked" info string into the log. It then went into a perpetual "Checking shutdown state" loop. Looking at the code, the next state would have been SHUTDOWN_PARK_DOME. I presume it got stuck there as the message "Checking shutdown state" repeated all night.

I have a dome, but that is on another system. This particular system does not have a dome. But to my surprise, the schedule I am running has "Park Dome" checked. I generate my .esl files in python using templates. I bet I imported a template .esl file that was generated on my other system that has a dome.

So my hypothesis is that the scheduler was trying to park a dome that does not exist. By checking "Park Dome" in the GUI, the scheduler state machine would see that parkDomeCheck->isChecked() is True. But inside parkDomeCheck, domeInterface.isNull() also returns True since there is no dome. But the immediate return from domeInterface.isNull() does not set the shutdownState to SHUTDOWN_SCRIPT, which would be the next step. Instead, it stays in SHUTDOWN_PARK_DOME, causing the loop to repeat.

Tonight I will have a chance to run this again with "Park Dome" unchecked in the schedule. Weather and full moon might wash out tonight's run, so it might be a couple days.
1 year 3 weeks ago #91663

Please Log in or Create an account to join the conversation.

Replied by Jasem Mutlaq on topic Ekos Scheduler did not wake up

Excellent analysis Lee! I just submitted a fix that would mark an error at least (it's not safe to continue to the next step automatically). So at least you'll know what's going on and what caused the error.
1 year 3 weeks ago #91665

Please Log in or Create an account to join the conversation.

  • Posts: 24
  • Thank you received: 0

Replied by Lee Peterson on topic Ekos Scheduler did not wake up

Thank you, Jasem!

I saw your commit with the changes and I would like to test it tonight. But looking at the binary-factory Jenkins dashboard for MacOS, It doesn't look like the recently built commit 918bc20506c4b139ca35ca4d53372496b73b05e0 corresponds to any commit on invent.kde.org/education/kstars. So there must be something up with the git SHA. At any rate, is your commit in the latest build #1853? If so I'll test it tonight.

Oh and by the way I'd like to add my observatory info to my signature, but the link in your signature is broken. Can you tell me where to do that on this site?
1 year 3 weeks ago #91672

Please Log in or Create an account to join the conversation.

  • Posts: 1224
  • Thank you received: 566

Replied by Hy Murveit on topic Ekos Scheduler did not wake up

Yes, the signature editing is well hidden.

To add/edit your signature, click on the "My Details" tab near the top of the forum page (or go to indilib.org/forum/edit-details.html) and then click on the "Profile Information" tab, then go all the way to the bottom (just above "Save") and edit your signature. Then click "Save".
1 year 3 weeks ago #91673

Please Log in or Create an account to join the conversation.

  • Posts: 24
  • Thank you received: 0

Replied by Lee Peterson on topic Ekos Scheduler did not wake up

I am happy to report that everything ran perfectly last night. By unchecking the "Park Dome" on the scheduler GUI, the scheduler shutdown the system properly and then woke up as planned.

Thank you everyone for your help!
1 year 3 weeks ago #91711

Please Log in or Create an account to join the conversation.

Time to create page: 0.573 seconds