×

INDI Library v1.9.7 Released (29 Jul 2022)

Bi-monthly INDI Library released with new drivers and bug fixes.

Announcing a new Scheduling Algorithm

  • Posts: 834
  • Thank you received: 405
The Greedy Scheduler

I've recently submitted an Ekos change that will allow you to choose a new scheme for scheduling jobs. (In the nightly release) there should now be a scheduling algorithm choice. If you set it to "Classic" (the default), nothing has changed. If you set it to "Greedy" you will see the changes described below.  Note: these changes concern scheduling--deciding which jobs run and when. They do not affect the job management aspects of the scheduler which remain unchanged.

In both the Classic and Greedy schedulers, jobs are listed as "earlier on the job list means higher priority". With the Classic scheduler, priority** is of the highest importance. It will not schedule a lower priority job until the higher priority job is done, even if that job takes several nights, and even if the higher-priority job cannot run at the current time, e.g. due to altitude/terrain/etc.  In contrast to this, the Greedy scheduler attempts to keep Ekos busy as much as possible. Although it gives priority to earlier-listed-jobs, it will run later-listed ones if the earlier one can't run. Of course, the lower priority job will get preempted when the high-priority job can finally start to run.

**In this discussion, "lower priority" means "further down on the job list". and similarly, higher priority is higher/earlier on the job list. We will be phasing out the priority number that can be assigned to jobs on the Classic scheduler.

If you start the scheduler with only one job, there is no difference between Classic and Greedy scheduling. However, if you have more than one job, depending on your setup, there is a good chance that the Greedy scheduler will schedule more imaging time than Classic.

Here's a recommended way to try this out. Let's assume you have a main target for which you want to collect as many images as possible. Set that target up as first on the scheduler list, have it start ASAP and set its completion condition as "Repeat Until Terminated". It should be scheduled to image whenever possible (even across multiple nights) until you turn off the scheduler or Ekos. Add several other targets as well, ones that you might also be interested in, and that can be imaged in other parts of the night. Make sure those are listed below the primary target on the jobs list. Set those the same way (ASAP/RepeatUntilTerminated). They will be scheduled to run whenever the primary target can't be imaged.  Of course, make sure the twilight restriction is set for all your targets. Altitude and terrain restrictions are important as well--if Ekos doesn't know that there's a tree or house blocking your target, it can't be smart about scheduling it.

Since jobs will be preempted/restarted more often with Greedy than with Classic, the "Remember Job Progress" option is now more important.  You can find this setting in the KStars Setting Menu --> Ekos --> Scheduler --> "Remember job progress".  This option only works if you are storing images on the same machine where the scheduler is running. You should enable "Remember Job Progress" with this scheduler to get the most benefit, assuming your capture sequences use multiple/different filters. If your capture sequences are just used with one type of filter/or OSC then it probably doesn't matter. (RememberJobProgress has also be updated a bit, and should now do a better job of picking up where the last attempt finished.)

Here is a screenshot of the scheduler running with this new scheme. Note that there are 4 jobs, and they are listed in order of precedence. You can see the next start times for each of the jobs on the scheduler table, and the highlighted (4th) job is the one currently running. A schedule for the next 48 hours is also printed in the log window at the bottom.

 

The scheduler is (minimally) integrated with Analyze.  During testing I let the Greedy Scheduler run for 3 days. Here's a screenshot of the Analyze timeline for the 3 days. (Note some timelines didn't display well because of screen resolution.)  The top line is the new scheduler timeline in Analyze. The different colors correspond to different jobs that were run, each one keeping its same color on different stars (yes, red is an unfortunate choice of colors and I've already submitted code that changes the color scheme for that line). You can see the different jobs ran as expected each night.

 

Please test this (currently in the nightly release) and let me know how it goes.
Hy
AP1100 & Orion Atlas Pro, WO/ZS105 w/Moonlight V2 focus, GSO RC10 w/RSF focus
ZWO ASI1600, Astronomik Filters, ST80, QHY 5L-IIm.
KStars/Ekos/Indi on NUC10 & RPi4 w/SSD -- Ubuntu
Projects: Greedy Scheduler, Terrain, Polar Align, Analyze, Linear Focuser, SEP MultiStar & GPG Guide, FITS autostretch.
The following user(s) said Thank You: Jasem Mutlaq, Rafa Barberá, John, Chris Madson
5 months 3 days ago #81233
Attachments:

Please Log in or Create an account to join the conversation.

  • Posts: 350
  • Thank you received: 51
This new Greedy scheduler looks fantastic. Can't wait to try it.
I have one related question... I have a visual 360º terrain image of my backyard overlaid in the sky view. What is the current mechanism to make use of the alpha channel of that image to establish / use the altitude / terrain limits for the scheduler?

Thanks!
120 MM Skywatcher Esprit on Celestron CGX, ZWO ASI 224MC guiding, Pegasus FocusCube v2+PPBA
Nikon Z7 8256 x 5504, 35.9 x 23.9mm 4.34 um. Triad Ultra Quad NB Filter
1) Odroid-N2 Ubuntu-mate 20.04.4 2) StellarMate RPI4 8G
The good, the bad, the ugly at sciencedowneast.no-ip.ca/zenphoto/
5 months 2 days ago #81249

Please Log in or Create an account to join the conversation.

  • Posts: 11
  • Thank you received: 1
I just pulled the latest from master and tried it out. No joy!

The Scheduler has stopped working completely;
The moment I add a new target, it shows up with status "Complete". Clicking "Reset" does nothing. Starting the scheduler just reports that there are no jobs left in the queue.
Not being able to use it at all at the moment. Neither Classic or Greedy.

  

 
Last edit: 5 months 2 days ago by Jussi Saarivirta.
5 months 2 days ago #81258
Attachments:

Please Log in or Create an account to join the conversation.

  • Posts: 834
  • Thank you received: 405
Jussi,

It seems to me that you likely have "Remember Job Progress" checked, and that your job is set to finish at Sequence Completion.
Remember Job Progress is hidden in the Configure KStars Menu, Ekos tab, Scheduler subtab.

To get your job to run you can do one of the following:
- uncheck and apply 'Remember Job Progress' (you may have to do something to get it to reschedule, like push play, not sure).
- Move the folder with all your images to a folder with a different name
- change your job to one of the other settings, e.g. Repeat for 2, or Repeat forever, ...

Let me know if this doesn't work.
Hy
AP1100 & Orion Atlas Pro, WO/ZS105 w/Moonlight V2 focus, GSO RC10 w/RSF focus
ZWO ASI1600, Astronomik Filters, ST80, QHY 5L-IIm.
KStars/Ekos/Indi on NUC10 & RPi4 w/SSD -- Ubuntu
Projects: Greedy Scheduler, Terrain, Polar Align, Analyze, Linear Focuser, SEP MultiStar & GPG Guide, FITS autostretch.
The following user(s) said Thank You: Jussi Saarivirta
5 months 2 days ago #81261

Please Log in or Create an account to join the conversation.

  • Posts: 11
  • Thank you received: 1
Thank you Hy! That did the trick!

I must say, the setting as a default setting seems a bit odd. Also it might be a good idea to tell a little more about why the behavior was as it was in the log - if it had said something about "Remember Job Progress" then I'd have gone digging in the settings.
But yeah, I must have forgotten it since I think that's the first thing I've always turned off after a fresh install and then forgot completely about its existence
5 months 2 days ago #81263

Please Log in or Create an account to join the conversation.

  • Posts: 834
  • Thank you received: 405
Jerry,

There are two steps involved.

Step 1: You add the terrain 360-degree image and make sure it's pretty well aligned with the actual view from your telescope. It sounds like you've already done this.

Step 2: The scheduler doesn't automatically get the info from the alpha channel--perhaps it should. However, there is a manual step to give it that information. In the KStars main (SkyMap) window,  go to the Settings menu, and click on "Artificial Horizon". There is a process from there to click on the sky map and add points to a series of line segments that define artificial horizons. It isn't too onerous, given that you already have the overlay with terrain picture.  You can have/enable more than one artificial horizon. Make sure that one or more are enabled/checked, and hit apply. 

Now, the scheduler should be aware of these, and you should see that in your Greedy plan. You may need to trigger a re-plan to see it (e.g. by moving a job up and down, etc). The replan thing shouldn't be a worry in real production as it replans on start, and checks jobs each second, and replans on preemption, etc etc.

Here's a pic of the interface for the manual clicking on the SkyMap.
Hy

 
AP1100 & Orion Atlas Pro, WO/ZS105 w/Moonlight V2 focus, GSO RC10 w/RSF focus
ZWO ASI1600, Astronomik Filters, ST80, QHY 5L-IIm.
KStars/Ekos/Indi on NUC10 & RPi4 w/SSD -- Ubuntu
Projects: Greedy Scheduler, Terrain, Polar Align, Analyze, Linear Focuser, SEP MultiStar & GPG Guide, FITS autostretch.
The following user(s) said Thank You: AstroMuni
5 months 2 days ago #81265
Attachments:

Please Log in or Create an account to join the conversation.

  • Posts: 834
  • Thank you received: 405
Jussi,

Since you are an early tester, and it's the first thing you ran into, it seems that we should find a way to alert users about this. What do you think the system should do?  Clearly there are cases where the behaviour is right and the job should not be run because it is completed. 

Should 'Remember Job Progress' be promoted to appear on the scheduler page? Should there be a popup when all jobs are completed and the user hits play, suggesting he look at this? ...

Feel free to suggest UI improvements.
Hy
 
AP1100 & Orion Atlas Pro, WO/ZS105 w/Moonlight V2 focus, GSO RC10 w/RSF focus
ZWO ASI1600, Astronomik Filters, ST80, QHY 5L-IIm.
KStars/Ekos/Indi on NUC10 & RPi4 w/SSD -- Ubuntu
Projects: Greedy Scheduler, Terrain, Polar Align, Analyze, Linear Focuser, SEP MultiStar & GPG Guide, FITS autostretch.
5 months 2 days ago #81266

Please Log in or Create an account to join the conversation.

  • Posts: 11
  • Thank you received: 1
@Hy

It might not be a bad idea to promote 'Remember Job Progress' to the scheduler page. It is after all a setting that affects the scheduler behavior in a big way. Also if possible, if there is a reasonably sensible way to detect the situation it might be a good idea to write to the log something like "Job '[x]': All images [number of images] were found in the target directory, nothing to do. Job progress is considered complete (Remember Job Progress is enabled)". Not sure if there's a need for a popup, although I suppose it might not be a bad idea either? In any case by instinct I immediately looked at the log to figure out what was going on, but in my case it didn't help.
5 months 2 days ago #81274

Please Log in or Create an account to join the conversation.

  • Posts: 834
  • Thank you received: 405
Thanks. FWIW, there is plenty (perhaps too much) log spam related to this, though it probably doesn't mention 'Remember Job Progress'.
Look for lines with the words "completed its sequence" in them.
See: invent.kde.org/education/kstars/-/blob/m.../scheduler.cpp#L5772
Of course, you'd need to have logging enabled for the scheduler.

Hy

 
AP1100 & Orion Atlas Pro, WO/ZS105 w/Moonlight V2 focus, GSO RC10 w/RSF focus
ZWO ASI1600, Astronomik Filters, ST80, QHY 5L-IIm.
KStars/Ekos/Indi on NUC10 & RPi4 w/SSD -- Ubuntu
Projects: Greedy Scheduler, Terrain, Polar Align, Analyze, Linear Focuser, SEP MultiStar & GPG Guide, FITS autostretch.
5 months 2 days ago #81276

Please Log in or Create an account to join the conversation.

  • Posts: 126
  • Thank you received: 21
Thats great news @Hy. As you mentioned that this is minimally integrated with Analyse, are there any improvements in this release for figuring out progress of images stored in Remote folders? Reason is that I notice that Analyse is aware of the number of images taken but Scheduler wasnt and continued taking images past the target number.

Celestron Astromaster 130 on HEQ5 Pro mount, ZWO ASI224mc, 30mm guidescope with ASI120mm mini. All managed using Kstars/Ekos on RPi with Astroberry build.
Last edit: 5 months 1 day ago by AstroMuni.
5 months 1 day ago #81289

Please Log in or Create an account to join the conversation.

  • Posts: 834
  • Thank you received: 405
@AstroMuni It wouldn't be hard for the scheduler to keep track of what has been captured in the current Ekos session, which is basically what Analyze is doing. The issue is, though, that Remember Job Progress keeps track of all sessions by looking at the files captured. That is, if you ran you job using Ekos/Scheduler the past 3 nights, but turned off your computer during the daytime and restarted Ekos at night, the current scheme still knows about all those past captures and balances the captures appropriately.

What would be needed to implement this where images aren't stored on the scheduler/Ekos machine is some kind of persistent storage to keep that information (and some way for the user to reset that, if desired). The persistent storage now is the filesystem itself.

I suppose I could test if the images are being stored locally, and if not, then just keep track of the current Ekos sessions' captures. Not as good as "all the captures", but better than nothing. Do you have an alternate approach/suggestion? 

 
AP1100 & Orion Atlas Pro, WO/ZS105 w/Moonlight V2 focus, GSO RC10 w/RSF focus
ZWO ASI1600, Astronomik Filters, ST80, QHY 5L-IIm.
KStars/Ekos/Indi on NUC10 & RPi4 w/SSD -- Ubuntu
Projects: Greedy Scheduler, Terrain, Polar Align, Analyze, Linear Focuser, SEP MultiStar & GPG Guide, FITS autostretch.
5 months 1 day ago #81305

Please Log in or Create an account to join the conversation.

  • Posts: 297
  • Thank you received: 55
Thanks for this -- a huge improvement that greatly simplifies multi-target multi-night imaging!

Also thank you for your clarification about the terrain imagery and artificial horizon data being independent and unaware of one another. I hadn't realized that and had wondered why my terrain had no effect on my scheduled imaging. All fixed now!
5 months 11 hours ago #81328

Please Log in or Create an account to join the conversation.

Time to create page: 1.096 seconds