New Internal Solver for Mac, Windows, and Linux -- Testing/ Experiments needed

Rob,
I just updated my Raspberry Pi to the latest KSARS using the AstroPi3 script with the new StellarSolver. During testing with the simulators I kept getting failed to solve errors. I eventually realised that the reason for the failure was I had only downloaded the relevant index files for my 80mm x 500mm FL imaging setup, the default simulator scope setup at 120mm x 900mm FL would not solve as the relevant index files were not present. On the Astronomy.net solver it would flash a warning that the index files were missing. Would it be possible to add the warning message to StellarSolver? It may help out newcomers to resolve any setup issues.

I must say once I had changed the simulator telescope settings to match those of my imaging setup, it worked flawlessly on both GSC and uploaded images, very nice work indeed.

Many Thanks

Alan

@El Corazon
you pointed me in the right direction.
Finally I used Synaptic to install libraw-dev while downgrading the libraw package.

Hi Alan,

So the message you are referring to was actually a function of KStars, not astrometry.net. Yes we could make it print that message again saying that you don't have the right indexes, but actually I didn't like that message because it was often incorrect. It would just say you don't have such and such index files even though you didn't need them for your image scale and I got tired of seeing that message. It is actually hard to say which index files you actually need to solve a particular field size. Yes, the index files are definitely meant for certain scales, and we could just do a table lookup based upon the scale size you/Kstars indicate the image should be and to just search for the index files with sky marks that size and the ones a little larger and a little smaller than your scale. But one issue you might encounter is that sometimes the index files that are significantly larger or smaller than your scale are the ones that actually solve the image. So it can be hard to say exactly which ones should be included. I did write an algorithm in the index file downloader a few years ago that attempts this. Maybe we could use similar code here. I mean I guess it is just a warning so you could ignore if it is wrong. We can look into it.

Thanks,

Rob

Rob,
Yes I see that, I have seen the message when I did have all the relevant indexes installed. Also what you have said explains why I found if I was slightly over generous when downloading the index files with Astronomy.net, the solving speeds actually improved. On a good night a first solve of 15 to 20 seconds and subsequent solves in 8 to 12 seconds. I wonder if the message could be made more of a basic / some type of prompt to check the installed index files, rather than just fail. It would act as a guide or starting point to sorting the issue of a failed solve.

By the way, we had the first clear night here for a few weeks last night and the new StellarSolver in KSTARS worked great right from the off, not a single failed solve during the session.

Many thanks

Alan

rlancaste wrote: Hi Alan,

So the message you are referring to was actually a function of KStars, not astrometry.net. Yes we could make it print that message again saying that you don't have the right indexes, but actually I didn't like that message because it was often incorrect. It would just say you don't have such and such index files even though you didn't need them for your image scale and I got tired of seeing that message. It is actually hard to say which index files you actually need to solve a particular field size. Yes, the index files are definitely meant for certain scales, and we could just do a table lookup based upon the scale size you/Kstars indicate the image should be and to just search for the index files with sky marks that size and the ones a little larger and a little smaller than your scale. But one issue you might encounter is that sometimes the index files that are significantly larger or smaller than your scale are the ones that actually solve the image. So it can be hard to say exactly which ones should be included. I did write an algorithm in the index file downloader a few years ago that attempts this. Maybe we could use similar code here. I mean I guess it is just a warning so you could ignore if it is wrong. We can look into it.

Thanks,

Rob

Rob,

The parallel solving speeds up the solver significantly (about 4-5 fold under my conditions). It now solves routinely in under 1 s on my Pi4/8Gb, but it takes up to 5 seconds otherwise. However, for this to work I need to restrict the astrometry files the Solver loads into RAM to a minimum. Since I am using a wide-field scope at 250 mm focal length, that is feasible on a Pi4. However, when I select all the astrometry files needed to solve on any of my scopes, including an 8" RC, then the file size exceeds the RAM available and solving slows down dramatically.
At the moment, I am managing this by manually moving the astrometry files that are not needed for my short focal length into a subfolder from which they are not loaded. I then can manually move them back when I am putting a larger scope on the mount.

What I am wondering is whether it would be possible for the Solver to just load those files it ANTICIPATES it needs for parallel solving BASED ON FOV into RAM and ignore the rest for first pass. Only when solving is unsuccessful, it would then include the remainder of the astrometry files for sequential solving.

That way, it would not be necessary for the observer to make that decision and then, if solving fails, having to move back larger astrometry files into the folder on which the Solver is looking for them.

I would think it should be possible to do that in an automated fashion?

Thoughts?

Jo

Jo,

This is certainly a good idea for the future, but right now unfortunately it might be kind of hard to implement. As I mentioned, it is slightly hard to say exactly which index file will solve any particular image. You can say that certain index files tend to work pretty well with a certain setup for sure, but you never know with certainty which exact one will solve it. So then the question would be, how would we determine which indexes to load? Do we load just the ones closest toe the scale we think it is? How much bigger or smaller do we go? Should that be somehow user configurable?

Also I would want to make sure that whichever way we do this for the internal solver, it will also work for the local solver as well. Right now, both the internal and external solvers are configured to just load all the indexes the given folder list. It is certainly possible to load just specific files for both the internal and external methods, but that would require a significant change to the code. Right now the only options are "load all the indexes" or don't do it and load them sequentially. I have considered a third option to have it automatically determine which one to do. This would be like a fourth option?

Another problem is that I'm not 100% sure that the RAM calculation that I have in there is perfect yet. It is supposed to shut off the "load all indexes in memory" option if there is not enough RAM. Before we would play with loading only certain indexes, I would want to be certain that that calculation works.

rlancaste wrote: Jo,

This is certainly a good idea for the future, but right now unfortunately it might be kind of hard to implement. As I mentioned, it is slightly hard to say exactly which index file will solve any particular image. You can say that certain index files tend to work pretty well with a certain setup for sure, but you never know with certainty which exact one will solve it. So then the question would be, how would we determine which indexes to load? Do we load just the ones closest toe the scale we think it is? How much bigger or smaller do we go? Should that be somehow user configurable?

Also I would want to make sure that whichever way we do this for the internal solver, it will also work for the local solver as well. Right now, both the internal and external solvers are configured to just load all the indexes the given folder list. It is certainly possible to load just specific files for both the internal and external methods, but that would require a significant change to the code. Right now the only options are "load all the indexes" or don't do it and load them sequentially. I have considered a third option to have it automatically determine which one to do. This would be like a fourth option?

Another problem is that I'm not 100% sure that the RAM calculation that I have in there is perfect yet. It is supposed to shut off the "load all indexes in memory" option if there is not enough RAM. Before we would play with loading only certain indexes, I would want to be certain that that calculation works.

Rob,

Making it user configurable would be one option. What I did was just move all the index files the astrometry page in the solver options told me were optional (i.e. those marked with an asterisk, presumably this is based on the calculated FOV of my telescope (?)), into a subfolder so they would not be loaded. That made a TREMENDOUS difference in speed.

I was hoping that would be easy to implement, to the tune of adding another check box "Do not load optional index files for parallel solving" or something to that effect.

Although my Pi4 has 8 Gb of RAM plus another 8 Gb of Swap Space on an SSD, and the total size of the index files I have downloaded is 5.6 Gb, the solver consistently determines that there is not enough RAM to load them all into memory. However, just moving the largest sets (those >1Gb) into a subfolder where the solver can't see them, makes the solver happy and I am rewarded with a 5 fold speed increase.

The advantage of having this option of initially loading only the non-optional index files would eliminate the need for moving the optional files into a subfolder, thus keeping them accessible to the solver if parallel solving fails. It could then fall back to sequential solving, now making use of the entire data set.

I thought that was the way the code was structured anyway, but I understand that if that is not the case significant changes might be necessary which do not appear justified at the moment.

But please keep this in mind nevertheless. The difference in speed is indeed enormous and many of us are using small SBCs with limited RAM to run their rigs. That means parallel solving might always necessitate manually removing optional index files - and moving them back in those special cases when the solver fails. That just feels a bit klutzy.

But, by all means, thanks again for building this fantastic solver!

Jo

@ Development Team

Thanks a lot!

I just upgraded and got the latest Kstars - Build: 2020-11-23T10:39:52Z. It won't solve. As the image comes in it prints "Image received", then Kstars goes away after about half a second. Here's everything I could gather on it.

Does anyone know what might be causing this problem?

TIA
Dave

This sounds to me like a library linking issue on your system. I am not 100% certain, but KStars is linked to GSL and GSL is linked to the CBLAS library. Either the link to CBLAS is broken somehow or maybe CBLAS is out of date? That is my best guess.

Thanks for the quick reply.

I thought that was the problem, but when I looked at the GSL library docs, it says they no longer link to gslcblas, leaving it to the application programmer to link the CBLAS library of their choice. I checked the indi cmake files and there is no mention of a CBLAS with any recognizable name. The libgsl has a 106 undefined symbols, all of them beginning with "cblas_".


Perhaps there is an older version that includes cblas.

Thanks,
Dave

For the time being I've gone back to the standard KStars release (Build: 2020-02-24T06:23:39Z). I'm going to rip everything out and reinstall during the week. I have clear cold skies and need to get some observing done.

Thanks for identifying the problem!

Dave