×

INDI Library v1.9.2 Released (14 Sep 2021)

Bimonthly Stable INDI Library release introduces new drivers and fixes for existing ones.

Re:Automatic Offset Calculation

  • Posts: 40
  • Thank you received: 1
OK, I fired up a Windows VM with ASCOM to look at the presets but I'm not sure what I'm seeing.

Offset should vary with gain.  I checked two cameras, each of which has a preset for highest dynamic range, unity gain, and lowest read noise.  They are:
  • ASI183MC-Pro
    • Gain 0, offset 10
    • Gain 111, offset 10
    • Gain 270, offset 10
  • ASI294MM-Pro
    • Gain 0, offset 30
    • Gain 120, offset 30
    • Gain 390, offset 30
These fixed offsets are contrary to what I understand about gain & offset.  Then I started thinking this offset value was based on the maximum hardware gain, a common trick for standardizing your calibration library on a single offset -- but when I tested the ASI183 at Offset 10, it was pretty clearly clipping at gain 270.  20 seems decent, 30 seems safer still.  

So, I don't know how these presets are supposed to work.  Maybe I've got some ASCOM driver issue or something, though I did install the latest ZWO ASCOM drivers and it didn't make a difference.

I switched to the native driver for the ASI294MM-Pro and noticed that ASCOM default offset of 30 works great at gain 120 (unity), but is not enough for gain 390.

Then I thought I would try ZWO's ASIStudio software -- maybe they calculated offset automatically?  That didn't expose offset at all, so initially I thought it must automatic... Gain "low" looked good.  Gain "medium" seemed clipped.  Gain "high" was badly clipped.  So, I don't think it adjusts offset at all, and if you can't adjust it manually then it's effectively broken as an imaging application.

This seems like a missed opportunity for camera makers to deliver gain/offset lookup tables with their drivers (in ASCOM you can add your own presets, apparently).  Software makers could provide a "calibrate offsets" button, but this would be best specified by the mfg in a lookup table.  But absent the lookup table, the button in the astroimaging software could take some pictures to calculate a good offset for a given gain (e.g. at Gain 120 it could take 2-second exposures at offsets in increments of 5 or 10 (starting at 5, never 0) until no pixels clip).  This would be a more automated way to do what the human eye can do with a histogram.  Then it could do this for multiple gains.

Another consideration would be to maybe make the default in KStars larger than 8; 30 seems safe, but I'm sure this varies a lot by camera.  I have no idea what a universally "safe" figure would be for all cameras and gains without needlessly affecting dynamic range.

Hopefully someone with more experience imaging and/or with ASCOM can comment on this further.
The following user(s) said Thank You: Eric
Last edit: 5 months 1 week ago by MountainAir.
5 months 1 week ago #71384

Please Log in or Create an account to join the conversation.

  • Posts: 156
  • Thank you received: 28
Yes, offset varies from model to model, but also from camera specimen to camera. So two ASI533MC could have different offset. A lookup table could be valid only if we put there maximum values. It will be better to have a "sensor analysis tool" as sharpcap has. Also, you can perform this analysis by hand:

1. Put the cap on the camera / telescope.
2. Set the desired gain and enable cooling if your camera has the option.
3. Put offset to 0.
4. Take a 30s dark

You can see that the maximum histogram peak is to the left, touching the zero border. This will mean shadow clipping, so you need to increase the offset. Increment offset by 5 or 10 and repeat. Repeat until you can see a gap between the left histogram border and the beginning of the dark frame maximum peak. This will be the optimal offset for your camera at this gain (maximum range and no shadow clipping).

I've performed this analysis on my ASI533MC and a good offset for me is 30 at unit gain. The ASCOM driver for this camera has a backed in value of 70 that is safe from the point of view of manufacturing tolerances, but very high for my concrete sample :).

So if INDI ends having an offsets table defined, I hope that those values will be only recommendation and we can override them.

Regards
The following user(s) said Thank You: Eric, Doug S
5 months 1 week ago #71394

Please Log in or Create an account to join the conversation.

  • Posts: 40
  • Thank you received: 1
Yep, that's exactly what I normally do to determine offset. Just this one time my images were coming out poorly until I realized that I had forgotten to change the default KStars offset of 8, and thus my calibration library needs to be re-shot. I also run all my sensors though SharpCap's sensor analysis, which I occasionally use with the Smart Histogram feature to suggest good exposure lengths when I change cameras or have different sky quality. Great feature, but I don't think it recommends an offset so that's always a separate step. BTW, SharpCap uses the term "brightness" for offset.
Last edit: 5 months 1 week ago by MountainAir.
5 months 1 week ago #71408

Please Log in or Create an account to join the conversation.

  • Posts: 11
  • Thank you received: 0
I am a bit late to the discussion but I read the thread a few days ago and decided to try the method of finding the optimum offset suggested by rbarberac.I set up on the bench an ASI533mc pro osc camera connected to a powered usb hub plugged into an rpi4 running Stellarmate 1.5.8. Communication between a Macbook air and the rpi was by wifi using external antennas on both the mac and the rpi4.First I connected to the Stellarmate by novnc. Using 30 sec exposure screenshots for a 100 offset and 0 offset.P { margin-bottom: 0.21cm }  are attached.As can be seen there is not a lot of movement along the x-axis of the fits viewer.This would indicate that the change in offset is not being applied to the camera although in the indi control panel for the camera it is shown to have changed. Due to my inexperience I suppose I have not set a parameter to a good value or not pressed a tab somewhere. Can anyone suggest where I am going wrong please?Meanwhile I thought about accessing the camera using Kstars on the Mac. Here are the results of a 100 and a 0 offset. In the screenshots attache the histogram can be clearly seen to move along the x-axis of the fits viewer, as expected, allowing optimisation of the offset.The other curious thing is that the frequency and intensity scales in the 2 cases,one using vnc and the other using Kstars on the Mac, are different. Shouldn't these be close in values? What am I missing?My thanks in advance for any suggestions.P { margin-bottom: 0.21cm }
 
RPI4 with Stellarmate OS
Macbook air
ZwoASI533mc pro
ZwoASI120 mm-s
CEM 26
Telescopes (backorder)
4 months 2 weeks ago #72090
Attachments:

Please Log in or Create an account to join the conversation.

  • Posts: 1941
  • Thank you received: 416
So, why a 30 sec dark and not 20 or 45 or 61.472458 sec?
Wouter van Reeven

ASI6200MM and 7 slot 2" filter wheel with a SkyWatcher Esprit 80 ED on a SkyWatcher HEQ5-Pro
ASI1600MM-Pro Cooled and 5 slot 1.25" filter wheel with an 8" TS Ritchey-Chrétien on a SkyWatcher EQ6-R
4 months 2 weeks ago #72092

Please Log in or Create an account to join the conversation.

  • Posts: 11
  • Thank you received: 0
I was following a suggestion of rbarberac from a previous post.
RPI4 with Stellarmate OS
Macbook air
ZwoASI533mc pro
ZwoASI120 mm-s
CEM 26
Telescopes (backorder)
4 months 2 weeks ago #72102

Please Log in or Create an account to join the conversation.

  • Posts: 1941
  • Thank you received: 416
Yes, the question was for rbarberac :)
Wouter van Reeven

ASI6200MM and 7 slot 2" filter wheel with a SkyWatcher Esprit 80 ED on a SkyWatcher HEQ5-Pro
ASI1600MM-Pro Cooled and 5 slot 1.25" filter wheel with an 8" TS Ritchey-Chrétien on a SkyWatcher EQ6-R
4 months 2 weeks ago #72104

Please Log in or Create an account to join the conversation.

  • Posts: 156
  • Thank you received: 28
Exact exposure time is not relevant once you get some dark signal to look at. I prefer to use 30s but is up to you to use longer or shorter exposures. From your results, I can see clearly two different behaviors. On the first pair of images, I can't see any change, but I don't know if the way the Ekos draws the histogram has some influence. On the second pair you can see clearly how at 0 offset the background is touching the left axis, but at offset 30 there is clearly a gap, so no dark signal is truncate.

My procedure was different: I programmed a sequence increasing the offset on 5 ADU steps from 0 to 70. Take all the pictures and then I opened them on SiriL and zoomed the histogram 10x to look at the profile near 0.

Regards
4 months 2 weeks ago #72119

Please Log in or Create an account to join the conversation.

  • Posts: 778
  • Thank you received: 101
Two comments here:

You do not need a dark signal for that calibration. What you measure is the noise of the ADC. If you have dark signal in that image, it will shift the whole histogram to the right, and you might end up with a too low offset that clips for short exposures. So I'd rather recommend short(er) exposures for this. For histograms, you need to use logarithmic view to properly judge the proper cutoff.

And, as someone who does this calibration for all his cameras, I strongly second the request to make such a table (only) a recommendation, and not an automatic override (the most important feature of any automatics is the button to switch it off ;^>)

For implementation, I could imagine a (camera-specific) gain-offset table in the sqlite database that can be edited by the user (somewhat similar to the focus offset table for filters). Filling that one with recommendations on first connect might indeed be helpful for many users.
openSUSE Tumbleweed KStars git INDI git
GPDX+EQMOD, CEM60EC, ASI2600/1600/290mini+EFW+EAF
4 months 1 week ago #72260

Please Log in or Create an account to join the conversation.

  • Posts: 1009
  • Thank you received: 298
I started a rework of the Observatory module, where I plan to add a tab for sky quality utilities. This calibration tool could go there although it is more related to the sensor than to the sky. I own an Atik Horizon, and while you can choose any combination of gain and offset you want, there are three presets written in the firmware, with increasing gain/offsets paired values. However I observe that Ekos modules are sometimes not consistent in setting the two values at the same time and that some of my frames are clipped when they should not, per exposure batch. For instance, the Align module contains a Gain parameter but no Offset parameter. What happens when we switch from one module to the other needs to be investigated.

-Eric
HEQ5-Pro - Atik 314E - Orion ED80T - DMK21 on Orion 50mm
DIY 3D-printed Moonlite and FWheel RGB/LPR
KStars and indiserver on two Atom 1.6GHz 1GB RAM Linux, VPN remote access
4 months 1 week ago #72267

Please Log in or Create an account to join the conversation.

  • Posts: 40
  • Thank you received: 1
Eric, is there any chance your sky quality utilities could also utilize locally-saved SharpCap Sensor Analysis result files? My apologies if you are already aware: Dr. Robin Glover's SharpCap can run fairly intensive analysis against each of your cameras. This produces a table of results that you can paste into various sub-exposure length spreadsheets, but the magic is when you point your scope and imaging train at your target and then use the Smart Histogram feature to analyze sky quality and automagically recommend the optimal sub-exposure length.

SharpCap is Windows-only, but the files can be copied and accessed anywhere. N.I.N.A. has the ability to use these SharpCap sensor analysis files, though it does not calculate properly (last night I tried it for the first time and it said my EdgeHD 9.25 needed a 3,573-second exposure).

[Edit: This feature may be better in the Camera module, but still would be useful if available in the Observatory tab since I don't think you'd run it for each target -- not practical.]

Also, I love your suggestion for adding offset to EVERY camera-control interface such as guiding, alignment, etc. I *think* what probably happens is that the last-used offset in the camera module is used in any other modules that use that specific camera, but you're right, if you use higher gain for alignment and focusing it might black-clip like crazy if your offset is still set for a gain of 0.
Last edit: 4 months 1 week ago by MountainAir.
4 months 1 week ago #72274

Please Log in or Create an account to join the conversation.

  • Posts: 1009
  • Thank you received: 298
There are a few algorithms available for optimal sub-exposure calculations, I haven't settled for one in particular yet nor actually implementing a single flavour. It all relies on proper frame statistics being available in the FITS data module, so there will be a review there first.

In terms of importing external data, there is also some work to do in various places. Telescopius schedules is one of the imports registered for implementation for instance. Would you point me to the documentation of the format for SharpCap analysis files?

The next question is how to make use of the imported or calculated data. If it is there as a reference, what do we use it for and where do we use it in Ekos. If we simply store the values in a database-backed spreadsheet, there are better tools than Ekos for this. I plan to have for instance an "auto" exposure value which would refer to the result of such utilities, but that only requires a small set of Ekos options, so there are better ideas such as calibrating periodically.

About the capture settings, yes, that exchangeable block could be a very interesting rework. This is in line with the future sequence wizard of the Scheduler that is waiting for the custom placeholder-based storage path generator being developed (yeah so many things in the pipe).

-Eric
HEQ5-Pro - Atik 314E - Orion ED80T - DMK21 on Orion 50mm
DIY 3D-printed Moonlite and FWheel RGB/LPR
KStars and indiserver on two Atom 1.6GHz 1GB RAM Linux, VPN remote access
4 months 1 week ago #72281

Please Log in or Create an account to join the conversation.

Time to create page: 1.332 seconds