×

INDI Library v2.0.7 is Released (01 Apr 2024)

Bi-monthly release with minor bug fixes and improvements

Kstars sequence fails after 32 photos. Debugging help needed

  • Posts: 18
  • Thank you received: 3
I've been having a strange issue since I started learning KStars with OAT. For some reason the imaging sequence crashes after a somewhat fixed amount of images. It's not always 32 photos, but today I tried three sequences with 30s, 60s and 120s exposures and every sequence crashed after exactly 32 photos. Sometimes longer exposures have made it up to 60 photos or smth, but 30s exposures rarely much over 30.

When the certain number of photos is reached, the sequence freezes for a bit and then the INDI control panel (the one that shows the mount and DSLR configuration) shows up. Based on the logs the camera connection gets "destroyed" and then found again, which opens the control panel. After that has happened, if I touch the sequence controls, like stop and start it again, KStars crashes totally.

These lines can be found on the log
org.kde.kstars.indi] - Removing device "Canon DSLR EOS 60D"
org.kde.kstars.indi] - INDIListener: Removing device "Canon DSLR EOS 60D"
org.kde.kstars.ekos] - "Canon DSLR EOS 60D is offline."
org.kde.kstars.indi] - Received new device Canon DSLR EOS 60D
org.kde.kstars.indi] - INDIListener: New device "Canon DSLR EOS 60D"

I don't even know if KStars is to blame or if the camera or something inside Astroberry does something weird.
Any idea how to debug this further? I'll add one log file as an example.
1 year 1 month ago #91420
Attachments:

Please Log in or Create an account to join the conversation.

  • Posts: 18
  • Thank you received: 3
I tested this without having Astroberry between the camera and KStars and the issue is gone. So it looks like KStars works correctly and the camera as well. So there's something weird going on with Astroberry.

Any idea how to debug this further? Some logs in Astroberry I should be aware of?
1 year 1 month ago #91465

Please Log in or Create an account to join the conversation.

  • Posts: 18
  • Thank you received: 3
Also saving the subs "locally" on the raspberry didn't make the bug disappear. So it looks like the bug is not related to the file transfer between Astroberry and local Kstars. But since it doesn't work also with KStars on the RPi, it's not about the camera connection either.
Last edit: 1 year 1 month ago by Mikko. Reason: Found the probable reason and made the title better
1 year 1 month ago #91497

Please Log in or Create an account to join the conversation.

  • Posts: 18
  • Thank you received: 3
I found that indi web server on Astroberry logs into `/tmp/indiserver.log`. (Why on earth put logs under /tmp???)

There's this line on the log when the crash happens on the client.
Child process 2670 died

After further investigation, that process belongs to indi_canon_ccd. When I followed the memory usage, it looks like it's reserving more and more memory until the process crashes.
astroberry@astroberry:~ $ while true; do ps -p 2670 -o rss=; sleep 2; done;
19604
19604
19604
19864
19864
28032
...
644088
676428
676704
678112
<no more values because the process is dead>
Last edit: 1 year 1 month ago by Mikko.
1 year 1 month ago #91503

Please Log in or Create an account to join the conversation.

  • Posts: 359
  • Thank you received: 114
Try check dmesg to see if it gets killed by OOM killer.
1 year 1 month ago #91510

Please Log in or Create an account to join the conversation.

yeah looks like some memory leak going on?
1 year 1 month ago #91511

Please Log in or Create an account to join the conversation.

I just tested with Canon 600D and took 50 photos, no issues. I ran the same memory checker and it stayed around 60MB on average with some deviations but nothing alarming. KStars is on a different machine and format is RAW and encoding is FITS.

EDIT: Repeated with NATIVE encoding and same result, no leaks.
Last edit: 1 year 1 month ago by Jasem Mutlaq.
1 year 1 month ago #91518

Please Log in or Create an account to join the conversation.

  • Posts: 18
  • Thank you received: 3
I'm always using NATIVE because FITS frames are all gray and broken for me. Another bug or something weird on my setup(?)

I upgraded INDI on the raspberry from 1.9.2 to 1.9.7 and witnessed the same exact behaviour: indi_canon_ccd memory usage grows until the raspberry memory hits the limit (872M + 100M swap). I could try 2.0.0 but Astroberry is currently using its own repository and I'm not sure if the official INDI repository works for ARM.

I also tried to alter the indiserver startup parameters since there was this -m 1000 parameters I didn't understand, but didn't affect anything. Maybe it has got something to do with Astroberry rather than the driver itself. It's the driver process that's eating around 700M RAM after 30 frames. Cannot understand.

In Astroberry, there's this Python thing commanding the indi server. Any idea if this is how indiserver is supposed to be run? Or is there another way to start it?

edit. of course I just realized that for the configuration to work from remote client, the web server has to be there..
Last edit: 1 year 1 month ago by Mikko.
1 year 1 month ago #91524

Please Log in or Create an account to join the conversation.

  • Posts: 18
  • Thank you received: 3
I've been wrestling this more and there is indeed a memory leak. I don't yet know how or why, but it doesn't happen always but most of the time on my tests.

Happened after the last episode:
1. I installed a fresh Raspbian Buster and manually installed indi-full using the official repository
2. I ran the first tests and found out that it was working properly. I ran a few test runs with 100 frames and it worked just fine for at least three times. Also my FITS frames were now working (at least a few times when I tested). The memory consumption stayed near 60M.
3. Took the whole setup outdoors and controlled it from indoors. Same hardware, same network, same everything. The only thing changing was that I took the tracker outdoors. So a bit worse network and quite a bit colder. This time it always crashed after the Raspberry memory was full. Tried restarting everything a couple of times but no luck.
4. Back indoors a few days later: tested a few runs by restarting things and trying again. A few times everything just worked.
5. Next session outdoors and nothing works. Even tried changing the Raspberry power to a more powerful 3A one. No changes, reserves memory until the driver gets killed by OOM killer.
6. Got annoyed and pulled a huge USB-cable from the carage outdoors and connected everything with wires. No Raspberry in between. This time I managed to get around 400 frames during the night with a single sequence. BUT: when I checked the computer after the sequence, the driver was reserving over 4G of memory. It just didn't crash because my 16G RAM was enough for it. So it was evident that it's leaking memory also on Ubuntu laptop. I posted a screenshot of that.

I still have no idea what is causing the crashes or the randomness with memory leaks. If I can share some logs or something, please tell. I can try another Canon EOS a bit later when I get one on my hands. It looks like it has nothing to do with sequences, because the memoru usage keeps growing even if I just take single preview frames. So I cannot go around this bug by making several shorter sequences.

Also about the FITS frames: for some reason I usually only get some random grayness on the FITS frames I take. It looks like a bug as well and it's really annoying since it prevents me from using all the flashy tools in KStars.
Last edit: 1 year 3 weeks ago by Mikko.
1 year 3 weeks ago #91639
Attachments:

Please Log in or Create an account to join the conversation.

Time to create page: 0.770 seconds