×

INDI Library v2.0.6 is Released (02 Feb 2024)

Bi-monthly release with minor bug fixes and improvements

Resuming a schedule...

Thanks for looking in into this. There are more issues reported here: bugs.kde.org/show_bug.cgi?id=389738

These are partially resolved as well. One issue I partially resolved recently is when running identical jobs and it used to mark them all as complete since it checks the file system for completed files and viola all requested files within the job are there so it is marked as complete. However, now it subtracts the file system # from each job sequentially. Similar, the scheduler sends the signature (prefix) of a job with the number of jobs to resume from. Suppose you have 2 identical scheduler jobs, each captures 5 images. When the scheduler runs, and it finds that 8 images exist on the file system for this particular signature, it marks the first job as complete but then proceed to the 2nd one and runs it. It then informs capture module that 3 images are already completely and that the sequence should start after that.

Changing the sequence file should result in resetting the estimation again, hopefully this is a simple fix. I already thought of the job index but as you can check in the bug report above, this is problematic when it is time to examine the data later on, so instead I approached it differently. So yeah, the scheduler is a complex beast but that is totally expected. There are probably more cases where it fails. For me, I usually just have a single job with a single sequence file so I don't suffer much!
The following user(s) said Thank You: Eric
6 years 1 month ago #23465

Please Log in or Create an account to join the conversation.

  • Posts: 1029
  • Thank you received: 301

Replied by Eric on topic Re:Resuming a schedule...

Sorry I forgot to check the KDE bugzilla!

OK that's clear about the jobs. I think the duplication of jobs should be kept for those of us whom run short unguided captures.

From what I understand from your post, when the scheduler is started anew, and that multiple duplicated jobs are in the list, the algorithm will "stack" the captures onto the jobs, starting with the first, whatever what happened during the last session.

I propose to add a column indicating the progress for each job. If jobs are duplicated, the first run will for instance say "10/10, 6/10, 3/10, 0/10", whatever the reason, and the second run will consolidate this into "10/10, 9/10, 0/10, 0/10". Then with that visual feedback, I will make sure "evaluate jobs" is redistributing the jobs that way, just like when opening the scheduler for the first time.

Probably I should enforce this evaluation anytime a job is modified, but that's debatable. There's a problem with this: we may have jobs that are duplicates in terms of captures, but are different in terms of which module is enabled and which is not (too much flexibility). We will still be distributing the captures to the first jobs with no regard for what option is enabled and what option is not. That's what will happen we we start the scheduler anew, of course. We can probably consider that a limitation of the current structure and live with it. I could find a way to warn the user about this: same sequence file, same target, but different options.


-Eric
6 years 1 month ago #23473

Please Log in or Create an account to join the conversation.

  • Posts: 1029
  • Thank you received: 301

Replied by Eric on topic Re:Resuming a schedule...

I checked what I could do without reworking the scheduler algorithm and structure:

- I'll have the estimation run anytime there is a change in a scheduler job. If possible, I'll have it trigger when a sequence job is updated in the Capture Sequence tab for clarity. But that might be overkill as when looking at it, the estimation has to take place when starting the scheduler too. In other words, re-evaluate everything unconditionally.

- I'll add a column with the light frame capture count to each scheduler job, so that the "stacking" effect of estimation is clearer when the output folder is identical between jobs. This means two duplicated jobs have to get a frame count increase anytime a capture is finished, and thus that this update must propagate amongst scheduler jobs, based on their output signature. This could be an unconditional re-estimation at each step actually.

- I'll search previous sequence jobs of the same scheduler job when estimating the required frame count for a scheduler job, in order to support multiple capture sequence jobs outputting to the same folder. This is probably an unnecessary feature, but because the sequence editor allows it and is unrelated to the scheduler editor, this is required to avoid sequence jobs not counted in when the total frame count should be larger than what is currently in the output folder. Oof :)

Eventually, highly debatable:

- I'll add HFR info next to the focus checkbox, for the user to notice whether it is the sequence job which is enforcing refocus, or only the scheduler. It's a problematic point, because it's not possible to know which object the focuser will use as target. I've had occurrences where this would work perfectly by running an initial manual focus, but also less than perfectly when the focus changed to another very small star (HFR limit was too high) or a galaxy core (HFR limit could not be achieved). I believe this could be fixed by raising the full-frame initial exposure that will be used to select a star.

- I'll make persistent stats about the duration of each step in order to better evaluate the duration of each step, and provide a way to customize these.

-Eric
The following user(s) said Thank You: Jasem Mutlaq
6 years 1 month ago #23691

Please Log in or Create an account to join the conversation.

  • Posts: 1029
  • Thank you received: 301

Replied by Eric on topic Re:Resuming a schedule...

Oh, and I'll probably make the Alignment tab remember which plate solver it needs to use. It's annoying to default to online when there's no network connection :)

-Eric
6 years 1 month ago #23692

Please Log in or Create an account to join the conversation.

It's doesn't default to online except when you first install KStars. It always remember your last choice. So if there is a problem with that then it's another issue.
6 years 1 month ago #23694

Please Log in or Create an account to join the conversation.

  • Posts: 1029
  • Thank you received: 301

Replied by Eric on topic Re:Resuming a schedule...

Probably related to the Windows build then. There are several items that won't save whatever my efforts :) I'll check the Linux version in parallel.

-Eric
6 years 1 month ago #23696

Please Log in or Create an account to join the conversation.

  • Posts: 1029
  • Thank you received: 301

Replied by Eric on topic Resuming a schedule...

I pushed a draft work, non-up-to-date vs. current branch, to github.com/TallFurryMan/kstars/tree/bugfix__duplicated_jobs, based on 79001ff59.

- Added a column with current number of captures for the sequence job associated to the scheduler job.
- Reworked score/altitude/priority comparison, there's probably a bug in how the scheduler manages this.
- Bounded the quantity of displayable logs to an arbitrary limit, still pushed to file nonetheless.
- Warned about the use of FINISH_LOOP, proposed solution.
- Reworked widget cell management, and resized table to column content.
- Added twilight check to dawn verification.
- Reworked checkJobStage/updateCompletedJobsCount/estimateJobTime to work with repeated and duplicated jobs.
- Evaluated jobs more often, so their status is up-to-date.
- Made aborted jobs evaluate again and reschedule properly.

More info when I rebase that work against the current codebase.
-Eric
The following user(s) said Thank You: Jasem Mutlaq
5 years 11 months ago #24904

Please Log in or Create an account to join the conversation.

  • Posts: 1029
  • Thank you received: 301

Replied by Eric on topic Resuming a schedule...

Getting delayed by @knro latest changes :lol: Now the scheduler goes to sleep without being started, and jobs schedule at other times than their altitude calculation require... But apart from that I'm pretty happy with where the Scheduler is going. Nice job with the report from the Messier Marathon, I'll try to obtain an automated (non-optimized because that would remove the fun) schedule with the code I have.

I attach the base .esl I use for tests. I'll prepare a Messier .esl too.
The following user(s) said Thank You: Jose Corazon
5 years 11 months ago #24977
Attachments:

Please Log in or Create an account to join the conversation.

Replied by Jasem Mutlaq on topic Resuming a schedule...

Now I'm afraid that your changes would introduce regressions. The scheduler code really needs some unit testing among other things to make sure such changes do not break it. For the time being, please try to test with your changes in all possible configurations.
5 years 11 months ago #25003

Please Log in or Create an account to join the conversation.

  • Posts: 1029
  • Thank you received: 301

Replied by Eric on topic Re:Resuming a schedule...

Indeed. This is why it takes so much time... I started the activity more than two months ago! While the code managing the evaluation changes because of my commits, the resulting list of actions which processes the job is still effective.

Besides, I believe the scheduler sleeping while it is not started is a regression in the current trunk code ;)

-Eric
5 years 11 months ago #25011

Please Log in or Create an account to join the conversation.

Time to create page: 0.308 seconds