I was originally going to title it "... when taking 16bit RAW" but it seems to just manifest itself more with 16 bit.

I started this years astronomy season this weekend and finally wanted to put the 1600MMC to some use after only using it briefly last year.
I did a dist-upgrade on the odroid connected to the scope/cam. (0.1+r3022~201703282004~ubuntu16.04.1 but later tried upgrading to 0.1+r3026~201704011202~ubuntu16.04.1 with the same issue)

It seems the CCD freezes after about 5-20 exposures with 16 bit; I can take more with 8 bit but it eventually freezes as well.

I tried downgrading to the version I used last year (1.3.1+r2763~201612162223~ubuntu16.04.1) and took 100 exposures in 16 bit where it seems the problem does not appear. Are there any known issues? Maybe somehting broken in the ASI binary blob?

I'll be staying on the older version for now but I'm happy to help with debugging this issue.

I was just in contact with ZWO over this very issue, it seems 16bit raw is problematic on some platforms. I added a warning for it, and ZWO just recommended using USB 3.0+

There is an interesting thread about 16bit issues:

zwoug.org/viewtopic.php?f=17&t=6968

My impression from this is that referring to the USB3 cams is just an excuse.

Thanks for the info and link guys.

I did some more digging today and it seems that this issue does indeed only show up on certain platforms; I was not able to reproduce it on x84_64.

I tried swapping out hubs, a variety of cables, different ports (all of the before as USB3 and USB2) and attaching directly.

I cloned HEAD and played around a bit on my armv7l device that I mainly use for capturing.
Out of the box I see freezes of the cam after a few captures.
Moving backwards in time, swapping out libASICamera2.bin freezes still appeared with this commit but the version before seems to work fine and I was able to take a few hundred captures without any issue.

So I think it looks like a bug was introduced in the SDK.

Sorry for the issue, does the frame failing only occur when connect to USB3.0 port? do you try to set USB traffic to 40? you can send email to me directly: yang.zhou(at)zwoptical.com

Yang from ZWO

No, it also happens on USB2. All testing I did was on 40.

To recap, everything works fine on amd64, my testing was mostly done on arm7, an odroid xu4 to be specific where it seems to work with tthe older SDK but not with the current one. I just did a quick test with a raspberry pi3 where I could not get it to work reliably with either version.
I'll do some more testing on the weekend.
Would you be interested in an image for the odroid or the pi where the problem could be reproduced?

I have a Pi3 installed "2016-11-25-raspbian-jessie.img", I want to reproduce it with it. please provide your setting, say exposure time, image type, etc. and what's the error and error frequency. what's your image for Pi3?

Hi Yang,
thanks for looking into this.

I did more tests today.
First a few details of the stuff I used:

Devices:
- ASI1600MM-COOL
- Raspberry Pi 3 (Raspbian, Kernel: 4.1.18-v7+)
- Odroid XU4 (Ubuntu 16.04.2 LTS, Kernel: 3.10.105-137)
- AMD64 (Debian Jessie, Kernel: 4.8.0-1-amd64)

Software:
- Kstars 2.7.6
- libindi and the 3rd party asi driver from git HEAD

Settings:
ZWO CCD ASI1600MM-Cool.CONNECTION.CONNECT=On
ZWO CCD ASI1600MM-Cool.CONNECTION.DISCONNECT=Off
ZWO CCD ASI1600MM-Cool.DRIVER_INFO.DRIVER_NAME=ZWO CCD
ZWO CCD ASI1600MM-Cool.DRIVER_INFO.DRIVER_EXEC=indi_asi_ccd_GIT_VERSION_HERE
ZWO CCD ASI1600MM-Cool.DRIVER_INFO.DRIVER_VERSION=0.5
ZWO CCD ASI1600MM-Cool.DRIVER_INFO.DRIVER_INTERFACE=2
ZWO CCD ASI1600MM-Cool.DEBUG.ENABLE=Off
ZWO CCD ASI1600MM-Cool.DEBUG.DISABLE=On
ZWO CCD ASI1600MM-Cool.SIMULATION.ENABLE=Off
ZWO CCD ASI1600MM-Cool.SIMULATION.DISABLE=On
ZWO CCD ASI1600MM-Cool.CONFIG_PROCESS.CONFIG_LOAD=Off
ZWO CCD ASI1600MM-Cool.CONFIG_PROCESS.CONFIG_SAVE=Off
ZWO CCD ASI1600MM-Cool.CONFIG_PROCESS.CONFIG_DEFAULT=Off
ZWO CCD ASI1600MM-Cool.ACTIVE_DEVICES.ACTIVE_TELESCOPE=Telescope Simulator
ZWO CCD ASI1600MM-Cool.ACTIVE_DEVICES.ACTIVE_FOCUSER=Focuser Simulator
ZWO CCD ASI1600MM-Cool.ACTIVE_DEVICES.ACTIVE_FILTER=CCD Simulator
ZWO CCD ASI1600MM-Cool.ACTIVE_DEVICES.ACTIVE_SKYQUALITY=SQM
ZWO CCD ASI1600MM-Cool.CCD_EXPOSURE.CCD_EXPOSURE_VALUE=0.91799998283386230469
ZWO CCD ASI1600MM-Cool.CCD_ABORT_EXPOSURE.ABORT=Off
ZWO CCD ASI1600MM-Cool.CCD_FRAME.X=0
ZWO CCD ASI1600MM-Cool.CCD_FRAME.Y=0
ZWO CCD ASI1600MM-Cool.CCD_FRAME.WIDTH=4656
ZWO CCD ASI1600MM-Cool.CCD_FRAME.HEIGHT=3520
ZWO CCD ASI1600MM-Cool.CCD_BINNING.HOR_BIN=1
ZWO CCD ASI1600MM-Cool.CCD_BINNING.VER_BIN=1
ZWO CCD ASI1600MM-Cool.FITS_HEADER.FITS_OBSERVER=Unknown
ZWO CCD ASI1600MM-Cool.FITS_HEADER.FITS_OBJECT=Unknown
ZWO CCD ASI1600MM-Cool.CCD_TEMPERATURE.CCD_TEMPERATURE_VALUE=18.100000000000001421
ZWO CCD ASI1600MM-Cool.CCD_INFO.CCD_MAX_X=4656
ZWO CCD ASI1600MM-Cool.CCD_INFO.CCD_MAX_Y=3520
ZWO CCD ASI1600MM-Cool.CCD_INFO.CCD_PIXEL_SIZE=3.7999999523162841797
ZWO CCD ASI1600MM-Cool.CCD_INFO.CCD_PIXEL_SIZE_X=3.7999999523162841797
ZWO CCD ASI1600MM-Cool.CCD_INFO.CCD_PIXEL_SIZE_Y=3.7999999523162841797
ZWO CCD ASI1600MM-Cool.CCD_INFO.CCD_BITSPERPIXEL=8
ZWO CCD ASI1600MM-Cool.CCD_COMPRESSION.CCD_COMPRESS=Off
ZWO CCD ASI1600MM-Cool.CCD_COMPRESSION.CCD_RAW=On
ZWO CCD ASI1600MM-Cool.TELESCOPE_TIMED_GUIDE_NS.TIMED_GUIDE_N=0
ZWO CCD ASI1600MM-Cool.TELESCOPE_TIMED_GUIDE_NS.TIMED_GUIDE_S=0
ZWO CCD ASI1600MM-Cool.TELESCOPE_TIMED_GUIDE_WE.TIMED_GUIDE_W=0
ZWO CCD ASI1600MM-Cool.TELESCOPE_TIMED_GUIDE_WE.TIMED_GUIDE_E=0
ZWO CCD ASI1600MM-Cool.CCD_FRAME_TYPE.FRAME_LIGHT=On
ZWO CCD ASI1600MM-Cool.CCD_FRAME_TYPE.FRAME_BIAS=Off
ZWO CCD ASI1600MM-Cool.CCD_FRAME_TYPE.FRAME_DARK=Off
ZWO CCD ASI1600MM-Cool.CCD_FRAME_TYPE.FRAME_FLAT=Off
ZWO CCD ASI1600MM-Cool.CCD_RAPID_GUIDE.ENABLE=Off
ZWO CCD ASI1600MM-Cool.CCD_RAPID_GUIDE.DISABLE=On
ZWO CCD ASI1600MM-Cool.TELESCOPE_TYPE.TELESCOPE_PRIMARY=On
ZWO CCD ASI1600MM-Cool.TELESCOPE_TYPE.TELESCOPE_GUIDE=Off
ZWO CCD ASI1600MM-Cool.WCS_CONTROL.WCS_ENABLE=Off
ZWO CCD ASI1600MM-Cool.WCS_CONTROL.WCS_DISABLE=On
ZWO CCD ASI1600MM-Cool.UPLOAD_MODE.UPLOAD_CLIENT=On
ZWO CCD ASI1600MM-Cool.UPLOAD_MODE.UPLOAD_LOCAL=Off
ZWO CCD ASI1600MM-Cool.UPLOAD_MODE.UPLOAD_BOTH=Off
ZWO CCD ASI1600MM-Cool.UPLOAD_SETTINGS.UPLOAD_DIR=/home/tliff
ZWO CCD ASI1600MM-Cool.UPLOAD_SETTINGS.UPLOAD_PREFIX=Light_XXX
ZWO CCD ASI1600MM-Cool.CCD_VIDEO_STREAM.STREAM_ON=Off
ZWO CCD ASI1600MM-Cool.CCD_VIDEO_STREAM.STREAM_OFF=On
ZWO CCD ASI1600MM-Cool.STREAM_OPTIONS.STREAM_RATE=0
ZWO CCD ASI1600MM-Cool.FPS.EST_FPS=30
ZWO CCD ASI1600MM-Cool.FPS.AVG_FPS=30
ZWO CCD ASI1600MM-Cool.RECORD_STREAM.RECORD_ON=Off
ZWO CCD ASI1600MM-Cool.RECORD_STREAM.RECORD_DURATION_ON=Off
ZWO CCD ASI1600MM-Cool.RECORD_STREAM.RECORD_FRAME_ON=Off
ZWO CCD ASI1600MM-Cool.RECORD_STREAM.RECORD_OFF=On
ZWO CCD ASI1600MM-Cool.RECORD_FILE.RECORD_FILE_DIR=/tmp/indi__D_
ZWO CCD ASI1600MM-Cool.RECORD_FILE.RECORD_FILE_NAME=indi_record__T_.ser
ZWO CCD ASI1600MM-Cool.RECORD_OPTIONS.RECORD_DURATION=1
ZWO CCD ASI1600MM-Cool.RECORD_OPTIONS.RECORD_FRAME_TOTAL=30
ZWO CCD ASI1600MM-Cool.CCD_STREAM_FRAME.X=0
ZWO CCD ASI1600MM-Cool.CCD_STREAM_FRAME.Y=0
ZWO CCD ASI1600MM-Cool.CCD_STREAM_FRAME.WIDTH=4656
ZWO CCD ASI1600MM-Cool.CCD_STREAM_FRAME.HEIGHT=3520
ZWO CCD ASI1600MM-Cool.CCD_COOLER_POWER.CCD_COOLER_VALUE=0
ZWO CCD ASI1600MM-Cool.CCD_CONTROLS.Gain=200
ZWO CCD ASI1600MM-Cool.CCD_CONTROLS.Gamma=50
ZWO CCD ASI1600MM-Cool.CCD_CONTROLS.Brightness=1
ZWO CCD ASI1600MM-Cool.CCD_CONTROLS.BandWidth=40
ZWO CCD ASI1600MM-Cool.CCD_CONTROLS.Flip=0
ZWO CCD ASI1600MM-Cool.CCD_CONTROLS.AutoExpMaxGain=300
ZWO CCD ASI1600MM-Cool.CCD_CONTROLS.AutoExpMaxExp=30
ZWO CCD ASI1600MM-Cool.CCD_CONTROLS.AutoExpMaxExpMS=30000
ZWO CCD ASI1600MM-Cool.CCD_CONTROLS.AutoExpMaxBrightness=100
ZWO CCD ASI1600MM-Cool.CCD_CONTROLS.HardwareBin=0
ZWO CCD ASI1600MM-Cool.CCD_CONTROLS.HighSpeedMode=0
ZWO CCD ASI1600MM-Cool.CCD_CONTROLS.Pattern adjust=0
ZWO CCD ASI1600MM-Cool.CCD_CONTROLS_MODE.AUTO_Gain=Off
ZWO CCD ASI1600MM-Cool.CCD_CONTROLS_MODE.AUTO_BandWidth=Off
ZWO CCD ASI1600MM-Cool.CCD_VIDEO_FORMAT.ASI_IMG_RAW8=Off
ZWO CCD ASI1600MM-Cool.CCD_VIDEO_FORMAT.ASI_IMG_RAW16=On
ZWO CCD ASI1600MM-Cool.ACTIVE_DEVICES.ACTIVE_TELESCOPE=Telescope Simulator
ZWO CCD ASI1600MM-Cool.ACTIVE_DEVICES.ACTIVE_FOCUSER=Focuser Simulator
ZWO CCD ASI1600MM-Cool.ACTIVE_DEVICES.ACTIVE_FILTER=CCD Simulator
ZWO CCD ASI1600MM-Cool.ACTIVE_DEVICES.ACTIVE_SKYQUALITY=SQM
ZWO CCD ASI1600MM-Cool.CCD_VIDEO_STREAM.STREAM_ON=Off
ZWO CCD ASI1600MM-Cool.CCD_VIDEO_STREAM.STREAM_OFF=On
ZWO CCD ASI1600MM-Cool.STREAM_OPTIONS.STREAM_RATE=0
ZWO CCD ASI1600MM-Cool.FPS.EST_FPS=30
ZWO CCD ASI1600MM-Cool.FPS.AVG_FPS=30
ZWO CCD ASI1600MM-Cool.RECORD_STREAM.RECORD_ON=Off
ZWO CCD ASI1600MM-Cool.RECORD_STREAM.RECORD_DURATION_ON=Off
ZWO CCD ASI1600MM-Cool.RECORD_STREAM.RECORD_FRAME_ON=Off
ZWO CCD ASI1600MM-Cool.RECORD_STREAM.RECORD_OFF=On
ZWO CCD ASI1600MM-Cool.RECORD_FILE.RECORD_FILE_DIR=/tmp/indi__D_
ZWO CCD ASI1600MM-Cool.RECORD_FILE.RECORD_FILE_NAME=indi_record__T_.ser
ZWO CCD ASI1600MM-Cool.RECORD_OPTIONS.RECORD_DURATION=1
ZWO CCD ASI1600MM-Cool.RECORD_OPTIONS.RECORD_FRAME_TOTAL=30
ZWO CCD ASI1600MM-Cool.CCD_STREAM_FRAME.X=0
ZWO CCD ASI1600MM-Cool.CCD_STREAM_FRAME.Y=0
ZWO CCD ASI1600MM-Cool.CCD_STREAM_FRAME.WIDTH=4656
ZWO CCD ASI1600MM-Cool.CCD_STREAM_FRAME.HEIGHT=3520

Everything works fine on the AMD64 machine and as far as i got with my tests also on the raspberry pi3.
The only machine I was able to get the problem was the Odroid. I tested USB2 and USB3 on the Odroid, it made no difference. Since the pi only has USB2, I did not test USB3 there.

I took the current git HEAD and compiled libindi, then I compiled indi_asi_ccd but I went back in commits for the 3rdparty/indi-asi/armv7/libASICamera2.bin file
Here's a list of the commits I took the bin file from and the result of trying to capture 200 frames (1000ms) in 16 bit with it (hashes are commits in git):

- 02030b836576c0f5684d6d562737ad05b1eb1e63 15 frames
- 64a529e31fa9e813f2c01696366323aa2062ef30 15 frames
- 75f72cfb8c372236dfa9eef8b81bed5089e72979 17 frames
- 690506af44c8d02bb28409a1103053942de088d1 18 frames
- 6ffcf202ae4d0355ed2ef6ec52c2336a4da619e0 10 frames
- 6b9fb46cc202b0bcd03a2f5fd1f3a21a43f1d65d works with no problem

Whenever the camera stopped working I hat to disconnect it completely before I could get anymore captures.

From what I can see it looks like there's a change in 6ffcf202ae4d0355ed2ef6ec52c2336a4da619e0 that created this behavior and/or made a bug in the odroid machine apparent.

Here's the built indi for pi3 and odroid: cloud.tliff.de/s/QMRsDaXLD61tJBV (for some libc reason the pi binaries dont work on the droid and vice versa)
Also included is the short shell script I used to build the binaries, they are named indi_asi_ccd_COMMITHASH


I can work around the problem for now by just using an older file, but a fix would of course be nice.
As I said, I'd be happy help. Just ask if you need debug logs or need me to try something.

Thank you very much for your informatin! I'll try to find the problem.

what's the max resolution of ASI1600MMC on Odroid with commit 6b9fb46cc202b0bcd03a2f5fd1f3a21a43f1d65d(work normally) and 6ffcf202ae4d0355ed2ef6ec52c2336a4da619e0 (have problem)? this will help me find the problem.
besides, which platform of the SDK did you use on Odroid, armv7?
Thank you.

Yang

Thanks yang,
Yes armv7, but I looked into it more over easter and it seems to be an odroid issue.
The kernel in there is ancient (3.10) and does not include this patch lkml.org/lkml/2013/5/24/440
Which would explain an error message i noticed in dmesg.
Recompiled the kernel with the patch and tried it out
Next I encountered this:
webcache.googleusercontent.com/search?q=...&client=firefox-b-ab but it seems to work with the value increased to 1M

I let captures run and got about 300 now before failing with:

[  953.639533] [c0] usb 4-1.1.1: Parent hub missing LPM exit latency info.  Power management will be impacted.

[  953.639700] [c0] xhci-hcd xhci-hcd.2.auto: xHCI xhci_drop_endpoint called with disabled ep ddaa6140

[  960.955016] [c5] usb 4-1.1.1: reset SuperSpeed USB device number 4 using xhci-hcd

[  960.969487] [c0] usb 4-1.1.1: Parent hub missing LPM exit latency info.  Power management will be impacted.

[  960.969647] [c0] xhci-hcd xhci-hcd.2.auto: xHCI xhci_drop_endpoint called with disabled ep ddaa6140

[  968.260035] [c0] usb 4-1.1.1: reset SuperSpeed USB device number 4 using xhci-hcd

[  968.274477] [c0] usb 4-1.1.1: Parent hub missing LPM exit latency info.  Power management will be impacted.

[  968.274647] [c0] xhci-hcd xhci-hcd.2.auto: xHCI xhci_drop_endpoint called with disabled ep ddaa6140

[  975.599976] [c0] usb 4-1.1.1: reset SuperSpeed USB device number 4 using xhci-hcd

[  975.614377] [c0] usb 4-1.1.1: Parent hub missing LPM exit latency info.  Power management will be impacted.

[  975.614533] [c0] xhci-hcd xhci-hcd.2.auto: xHCI xhci_drop_endpoint called with disabled ep ddaa6140

[  982.905035] [c0] usb 4-1.1.1: reset SuperSpeed USB device number 4 using xhci-hcd

[  982.919409] [c0] usb 4-1.1.1: Parent hub missing LPM exit latency info.  Power management will be impacted.

[  982.919580] [c0] xhci-hcd xhci-hcd.2.auto: xHCI xhci_drop_endpoint called with disabled ep ddaa6140

[  990.180016] [c5] usb 4-1.1.1: reset SuperSpeed USB device number 4 using xhci-hcd

[  990.194409] [c0] usb 4-1.1.1: Parent hub missing LPM exit latency info.  Power management will be impacted.

[  990.194575] [c0] xhci-hcd xhci-hcd.2.auto: xHCI xhci_drop_endpoint called with disabled ep ddaa6140

[ 1002.378822] [c0] hub 4-1.1:1.0: hub_port_status failed (err = -110)

[ 1007.378798] [c0] hub 4-1.1:1.0: hub_port_status failed (err = -110)

[ 1007.383614] [c0] hub 4-1.1:1.0: cannot disable port 1 (err = -110)

[ 1012.388792] [c0] hub 4-1.1:1.0: hub_port_status failed (err = -110)

....

The lines in the beginning appear through, even when I can successfully capture.
I'd say this is not a problem in the SDK but the odroid-kernel and I'll investigate further in that direction, maybe I can get a 4.x kernel up.


PS: On a sidenote an issue I had where captures shorter than 1000ms seconds would always fail in 16bit seems to have improved as well with the fixed kernel

In case it helps, the 1600 using 16 bit has worked fine on RPI3 using Ubuntu-Mate 16.04.